Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-27739

Changes to slave environment variables are ignored by master

      A slave's environment variables seem to be cached upon first connect and then never considered again. This prevents legitimate changes to a slave's environment from having any effect on builds. The only workaround I can find is to delete a slave and recreate it, which in a complex environment is unacceptable.

      This may be a consequence of JENKINS-26755.

          [JENKINS-27739] Changes to slave environment variables are ignored by master

          Milo Hyson created issue -

          Daniel Beck added a comment -

          Issue is missing installed plugins and their versions. Issue is missing specific steps to (easily) reproduce the issue.

          Daniel Beck added a comment - Issue is missing installed plugins and their versions. Issue is missing specific steps to (easily) reproduce the issue.

          Milo Hyson added a comment -

          Installed Plugins:

          Ant Plugin (ant) v1.2
          buildgraph-view (buildgraph-view) v1.1.1
          CloudBees Build Flow plugin (build-flow-plugin) v0.16
          conditional-buildstep (conditional-buildstep) v1.3.3
          Copy Artifact Plugin (copyartifact) v1.32.1
          Credentials Binding Plugin (credentials-binding) v1.3
          Credentials Plugin (credentials) v1.21
          Email Extension Plugin (email-ext) v2.39
          embeddable-build-status (embeddable-build-status) v1.5
          Environment Injector Plugin (envinject) v1.90
          External Monitor Job Type Plugin (external-monitor-job) v1.4
          GitHub API Plugin (github-api) v1.59
          GitHub plugin (github) v1.10
          GitHub Pull Request Builder (ghprb) v1.16-5
          Green Balls (greenballs) v1.14
          Javadoc Plugin (javadoc) v1.3
          Jenkins Active Directory plugin (active-directory) v1.39
          Jenkins build timeout plugin (build-timeout) v1.14
          Jenkins Cobertura Plugin (cobertura) v1.9.6
          Jenkins CVS Plug-in (cvs) v2.12
          Jenkins description setter plugin (description-setter) v1.9
          Jenkins Exclusion Plug-in (Exclusion) v0.10
          Jenkins GIT client plugin (git-client) v1.11.1
          Jenkins GIT plugin (git) v2.3
          Jenkins Gradle plugin (gradle) v1.24
          Jenkins Gravatar plugin (gravatar) v2.1
          Jenkins Job Configuration History Plugin (jobConfigHistory) v2.10
          Jenkins jQuery plugin (jquery) v1.7.2-1
          Jenkins Mailer Plugin (mailer) v1.12
          Jenkins Multijob plugin (jenkins-multijob-plugin) v1.15
          Jenkins Parameterized Trigger plugin (parameterized-trigger) v2.25
          Jenkins Priority Sorter Plugin (PrioritySorter) v2.9
          Jenkins promoted builds plugin (promoted-builds) v2.19
          Jenkins SSH Slaves plugin (ssh-slaves) v1.9
          Jenkins Subversion Plug-in (subversion) v2.4.5
          Jenkins TextFinder plugin (text-finder) v1.10
          Jenkins Throttle Concurrent Builds Plug-in (throttle-concurrents) v1.8.4
          Jenkins Translation Assistance plugin (translation) v1.12
          Job Import Plugin (job-import-plugin) v1.2
          Join plugin (join) v1.15
          JUnit Plugin (junit) v1.2
          LDAP Plugin (ldap) v1.11
          MapDB API Plugin (mapdb-api) v1.0.6.0
          Matrix Authorization Strategy Plugin (matrix-auth) v1.2
          Matrix Project Plugin (matrix-project) v1.4
          Maven Integration plugin (maven-plugin) v2.7.1
          Monitoring (monitoring) v1.53.1
          Next Build Number Plugin (next-build-number) v1.1
          Node and Label parameter plugin (nodelabelparameter) v1.5.1
          NodeJS Plugin (nodejs) v0.2.1
          OWASP Markup Formatter Plugin (antisamy-markup-formatter) v1.3
          PAM Authentication plugin (pam-auth) v1.2
          Performance plugin (performance) v1.11
          Plain Credentials Plugin (plain-credentials) v1.1
          Plot plugin (plot) v1.8
          Publish Over SSH (publish-over-ssh) v1.12
          Run Condition Plugin (run-condition) v1.0
          SCM API Plugin (scm-api) v0.2
          Script Security Plugin (script-security) v1.13
          Semantic Versioning Plugin (semantic-versioning-plugin) v1.7
          Simple Theme Plugin (simple-theme-plugin) v0.3
          SSH Agent Plugin (ssh-agent) v1.5
          SSH Credentials Plugin (ssh-credentials) v1.10
          Token Macro Plugin (token-macro) v1.10
          Windows Slaves Plugin (windows-slaves) v1.0
          Workflow: Step API (workflow-step-api) v1.2
          xUnit plugin (xunit) v1.92
          youtrack-plugin (youtrack-plugin) v0.6.3

          Reproduction:

          1) Add a slave node to a master connecting via JNLP.
          2) Start slave.jar on the slave machine.
          3) On the master, view the new node's system information and note the PATH environment variable.
          4) Kill slave.jar.
          5) Change the PATH on the slave machine.
          6) Restart slave.jar.
          7) Look again at the node's system information on the master and note the PATH variable has not changed.

          Milo Hyson added a comment - Installed Plugins: Ant Plugin (ant) v1.2 buildgraph-view (buildgraph-view) v1.1.1 CloudBees Build Flow plugin (build-flow-plugin) v0.16 conditional-buildstep (conditional-buildstep) v1.3.3 Copy Artifact Plugin (copyartifact) v1.32.1 Credentials Binding Plugin (credentials-binding) v1.3 Credentials Plugin (credentials) v1.21 Email Extension Plugin (email-ext) v2.39 embeddable-build-status (embeddable-build-status) v1.5 Environment Injector Plugin (envinject) v1.90 External Monitor Job Type Plugin (external-monitor-job) v1.4 GitHub API Plugin (github-api) v1.59 GitHub plugin (github) v1.10 GitHub Pull Request Builder (ghprb) v1.16-5 Green Balls (greenballs) v1.14 Javadoc Plugin (javadoc) v1.3 Jenkins Active Directory plugin (active-directory) v1.39 Jenkins build timeout plugin (build-timeout) v1.14 Jenkins Cobertura Plugin (cobertura) v1.9.6 Jenkins CVS Plug-in (cvs) v2.12 Jenkins description setter plugin (description-setter) v1.9 Jenkins Exclusion Plug-in (Exclusion) v0.10 Jenkins GIT client plugin (git-client) v1.11.1 Jenkins GIT plugin (git) v2.3 Jenkins Gradle plugin (gradle) v1.24 Jenkins Gravatar plugin (gravatar) v2.1 Jenkins Job Configuration History Plugin (jobConfigHistory) v2.10 Jenkins jQuery plugin (jquery) v1.7.2-1 Jenkins Mailer Plugin (mailer) v1.12 Jenkins Multijob plugin (jenkins-multijob-plugin) v1.15 Jenkins Parameterized Trigger plugin (parameterized-trigger) v2.25 Jenkins Priority Sorter Plugin (PrioritySorter) v2.9 Jenkins promoted builds plugin (promoted-builds) v2.19 Jenkins SSH Slaves plugin (ssh-slaves) v1.9 Jenkins Subversion Plug-in (subversion) v2.4.5 Jenkins TextFinder plugin (text-finder) v1.10 Jenkins Throttle Concurrent Builds Plug-in (throttle-concurrents) v1.8.4 Jenkins Translation Assistance plugin (translation) v1.12 Job Import Plugin (job-import-plugin) v1.2 Join plugin (join) v1.15 JUnit Plugin (junit) v1.2 LDAP Plugin (ldap) v1.11 MapDB API Plugin (mapdb-api) v1.0.6.0 Matrix Authorization Strategy Plugin (matrix-auth) v1.2 Matrix Project Plugin (matrix-project) v1.4 Maven Integration plugin (maven-plugin) v2.7.1 Monitoring (monitoring) v1.53.1 Next Build Number Plugin (next-build-number) v1.1 Node and Label parameter plugin (nodelabelparameter) v1.5.1 NodeJS Plugin (nodejs) v0.2.1 OWASP Markup Formatter Plugin (antisamy-markup-formatter) v1.3 PAM Authentication plugin (pam-auth) v1.2 Performance plugin (performance) v1.11 Plain Credentials Plugin (plain-credentials) v1.1 Plot plugin (plot) v1.8 Publish Over SSH (publish-over-ssh) v1.12 Run Condition Plugin (run-condition) v1.0 SCM API Plugin (scm-api) v0.2 Script Security Plugin (script-security) v1.13 Semantic Versioning Plugin (semantic-versioning-plugin) v1.7 Simple Theme Plugin (simple-theme-plugin) v0.3 SSH Agent Plugin (ssh-agent) v1.5 SSH Credentials Plugin (ssh-credentials) v1.10 Token Macro Plugin (token-macro) v1.10 Windows Slaves Plugin (windows-slaves) v1.0 Workflow: Step API (workflow-step-api) v1.2 xUnit plugin (xunit) v1.92 youtrack-plugin (youtrack-plugin) v0.6.3 Reproduction: 1) Add a slave node to a master connecting via JNLP. 2) Start slave.jar on the slave machine. 3) On the master, view the new node's system information and note the PATH environment variable. 4) Kill slave.jar. 5) Change the PATH on the slave machine. 6) Restart slave.jar. 7) Look again at the node's system information on the master and note the PATH variable has not changed.

          Daniel Beck added a comment -

          Weird. For me the only affected variable seems to be PATH.

          Daniel Beck added a comment - Weird. For me the only affected variable seems to be PATH .

          Daniel Beck added a comment -

          Could you confirm the above on your environment?

          Is this a regression (i.e. did it ever work for you)? If so, what is the newest version know to not be affected?

          Daniel Beck added a comment - Could you confirm the above on your environment? Is this a regression (i.e. did it ever work for you)? If so, what is the newest version know to not be affected?

          Milo Hyson added a comment -

          I just used PATH as an example. In my actual case I had three custom variables set on the slave machine. When I removed them and restarted slave.jar, those variables remained. I made other changes to test my suspicions (e.g. altering PATH) and they too remained unaltered.

          It's not a regression to my knowledge. We've never needed to change slave variables before.

          Milo Hyson added a comment - I just used PATH as an example. In my actual case I had three custom variables set on the slave machine. When I removed them and restarted slave.jar, those variables remained. I made other changes to test my suspicions (e.g. altering PATH) and they too remained unaltered. It's not a regression to my knowledge. We've never needed to change slave variables before.

          Daniel Beck added a comment -

          My basic 1.607 without any custom plugins had different behavior: Only PATH didn't change (unfortunately I didn't confirm it was set correctly in the shell, so may have been user error), the new variables I defined were correctly updated.

          Could you test whether the same issue occurs in the following configurations:

          • Jenkins 1.599 with Env-Inject 1.90
          • Jenkins 1.599 with Env-Inject not installed
          • Jenkins 1.606 with Env-Inject not installed

          There was a specific change in 1.600 related to slave caching, but it should clear slave environment after restarting.

          As there are, and were, numerous issues with env-inject being weird, knowing whether it affects things either depending on or independent of Jenkins version would be very helpful.

          Daniel Beck added a comment - My basic 1.607 without any custom plugins had different behavior: Only PATH didn't change (unfortunately I didn't confirm it was set correctly in the shell, so may have been user error), the new variables I defined were correctly updated. Could you test whether the same issue occurs in the following configurations: Jenkins 1.599 with Env-Inject 1.90 Jenkins 1.599 with Env-Inject not installed Jenkins 1.606 with Env-Inject not installed There was a specific change in 1.600 related to slave caching, but it should clear slave environment after restarting. As there are, and were, numerous issues with env-inject being weird, knowing whether it affects things either depending on or independent of Jenkins version would be very helpful.

          Milo Hyson added a comment - - edited

          Looks like the problem might have occurred around v1.606. For each of the tests I conducted the following procedure:

          1. Started with the following environment variables on the slave machine
            • PATH = /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
            • SOME_ENVVAR = value
          2. Launched java -jar jenkins.war on the master machine
          3. Added a new node to the master
          4. Started slave.jar on the slave
          5. Ensured PATH and SOME_ENVVAR showed up on master
          6. Killed slave.jar
          7. Appended :/nonexistent to PATH
          8. Removed SOME_ENVVAR
          9. Ensured variables changed on slave machine
          10. Restarted slave.jar
          11. Refreshed node's system-information page on master

          The following were the results:

          • v1.599 without EnvInject
            • PATH updated on master
            • SOME_ENVVAR disappeared from master
          • v1.599 with EnvInject
            • PATH updated on master
            • SOME_ENVVAR disappeared from master
          • v1.606 with EnvInject
            • PATH did not update on master
            • SOME_ENVVAR remained on master

          Milo Hyson added a comment - - edited Looks like the problem might have occurred around v1.606. For each of the tests I conducted the following procedure: Started with the following environment variables on the slave machine PATH = /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin SOME_ENVVAR = value Launched java -jar jenkins.war on the master machine Added a new node to the master Started slave.jar on the slave Ensured PATH and SOME_ENVVAR showed up on master Killed slave.jar Appended :/nonexistent to PATH Removed SOME_ENVVAR Ensured variables changed on slave machine Restarted slave.jar Refreshed node's system-information page on master The following were the results: v1.599 without EnvInject PATH updated on master SOME_ENVVAR disappeared from master v1.599 with EnvInject PATH updated on master SOME_ENVVAR disappeared from master v1.606 with EnvInject PATH did not update on master SOME_ENVVAR remained on master
          Daniel Beck made changes -
          Link New: This issue is duplicated by JENKINS-27928 [ JENKINS-27928 ]

          marlene cote added a comment -

          we are seeing the same behavior, although I did not try to delete the slave and recreate it. We keep having to restart the jenkins server. I also tried to uninstall the envinject plugin, but we have a multijob plugin that depends on it. When I restarted jenkins server, none of my jobs would load due to the missing envinject plugin, so I had to reinstall it.

          marlene cote added a comment - we are seeing the same behavior, although I did not try to delete the slave and recreate it. We keep having to restart the jenkins server. I also tried to uninstall the envinject plugin, but we have a multijob plugin that depends on it. When I restarted jenkins server, none of my jobs would load due to the missing envinject plugin, so I had to reinstall it.

            Unassigned Unassigned
            siggimoo Milo Hyson
            Votes:
            17 Vote for this issue
            Watchers:
            36 Start watching this issue

              Created:
              Updated: