• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core
    • None

      After upgrade from Jenkins version 1.532 to 1.533 "ssh" binary is no longer found. 1.534 is affected too.
      Downgrade back to 1.532 fixed the issue.
      System Informations are same for both versions.

        1. System Information [Jenkins].html
          27 kB
          Pavel Mateja
        2. consoleText
          4 kB
          Pavel Mateja

          [JENKINS-19926] SSH not found in Jeninks version > 1.532

          this is a git issue not an issue with the ssh-slaves launcher

          Stephen Connolly added a comment - this is a git issue not an issue with the ssh-slaves launcher

          Pavel Mateja added a comment -

          Hi, I was just guessing where the bug is.

          Changelog for 1.533 (2013/09/29) says:
          "Offer alternate error message for pattern-based project naming strategy. (pull request 914)"
          How this can cause loss of PATH env variable?

          And I don't think it's git problem. It can't find ssh client binary which seems to be problem of the slave to me. Especially when git plugin did not change during upgrade.

          Pavel Mateja added a comment - Hi, I was just guessing where the bug is. Changelog for 1.533 (2013/09/29) says: "Offer alternate error message for pattern-based project naming strategy. (pull request 914)" How this can cause loss of PATH env variable? And I don't think it's git problem. It can't find ssh client binary which seems to be problem of the slave to me. Especially when git plugin did not change during upgrade.

          from the console text, the issue is in the GIT plugin

          Stephen Connolly added a comment - from the console text, the issue is in the GIT plugin

          or else the issue is on your slave's /etc/profile... but I would suspect the git plugin. It is certainly not an ssh-slaves plugin issue as it has nothing to do with providing an ssh executable

          Stephen Connolly added a comment - or else the issue is on your slave's /etc/profile... but I would suspect the git plugin. It is certainly not an ssh-slaves plugin issue as it has nothing to do with providing an ssh executable

          Pavel Mateja added a comment -

          OK, I'm not Jenkins expert, I have to trust you.

          What about environment variables? Which plugin is responsible for them?

          And how can I find out what has been REALLY changed between 1.532 and 1.533?

          Pavel Mateja added a comment - OK, I'm not Jenkins expert, I have to trust you. What about environment variables? Which plugin is responsible for them? And how can I find out what has been REALLY changed between 1.532 and 1.533?

          Luis Arias added a comment -

          We're experiencing the same issue on upgrading from 1.532 to 1.533 and 1.534. IIRC from my previous round of tests with 1.533 it doesn't only affect git but any external program as if there was a path issue.

          Luis Arias added a comment - We're experiencing the same issue on upgrading from 1.532 to 1.533 and 1.534. IIRC from my previous round of tests with 1.533 it doesn't only affect git but any external program as if there was a path issue.

          Luis Arias added a comment -

          I imagine it has something to do with this commit but I don't know enough about jenkins to help fix it:

          https://github.com/jenkinsci/jenkins/commit/3eea39648943583a530f47f69f93737266537d9d

          Luis Arias added a comment - I imagine it has something to do with this commit but I don't know enough about jenkins to help fix it: https://github.com/jenkinsci/jenkins/commit/3eea39648943583a530f47f69f93737266537d9d

          Luis Arias added a comment -

          Pavel to know what changed between the two releases you can take a look at https://github.com/jenkinsci/jenkins/compare/jenkins-1.532...jenkins-1.533

          Luis Arias added a comment - Pavel to know what changed between the two releases you can take a look at https://github.com/jenkinsci/jenkins/compare/jenkins-1.532...jenkins-1.533

          vjuranek added a comment - - edited

          Hi,
          could you be more specific what kind of change you've observed between 1.532 and 1.533 - e.g. has some env. variable changed on slave/master or completely disappeared? If so, which one, what was the change and how the env. var is setup (taken from system, setup via env. inject plugin etc.)
          Thanks

          PS. 3eea39648943583a530f47f69f93737266537d9d only splits the variable expansion into two steps - adding all variable and then expanding them, as doing it in one step could lead to issue described in JENKINS-19488, so I believe this is not the root cause of this issue, however I'm also not 100% sure

          vjuranek added a comment - - edited Hi, could you be more specific what kind of change you've observed between 1.532 and 1.533 - e.g. has some env. variable changed on slave/master or completely disappeared? If so, which one, what was the change and how the env. var is setup (taken from system, setup via env. inject plugin etc.) Thanks PS. 3eea39648943583a530f47f69f93737266537d9d only splits the variable expansion into two steps - adding all variable and then expanding them, as doing it in one step could lead to issue described in JENKINS-19488 , so I believe this is not the root cause of this issue, however I'm also not 100% sure

          Pavel Mateja added a comment -

          Check the "consoleText" attachement.
          I think PATH env variable disappeared. Binary /usr/bin/git was not found after upgrade. I think /bin and /usr/bin are common PATH everywhere.
          The change was upgrade of debian package from repository. No modification in slave/master configuration or such.

          Pavel Mateja added a comment - Check the "consoleText" attachement. I think PATH env variable disappeared. Binary /usr/bin/git was not found after upgrade. I think /bin and /usr/bin are common PATH everywhere. The change was upgrade of debian package from repository. No modification in slave/master configuration or such.

          Luis Arias added a comment -

          Same here, somehow that commit or maybe something else cleared out the PATH env var. Downgrading to 1.532 restores a properly working jenkins. I have had to do this twice already from 1.533 and 1.534 so now I'm holding the package at 1.532.

          Luis Arias added a comment - Same here, somehow that commit or maybe something else cleared out the PATH env var. Downgrading to 1.532 restores a properly working jenkins. I have had to do this twice already from 1.533 and 1.534 so now I'm holding the package at 1.532.

          vjuranek added a comment -

          @Pavel: I of course check attached files. "ssh: No such file or directory" can be caused by various many things, and what confuses me is attached system information, where PATH is setup: "PATH /usr/local/bin:/usr/bin:/bin:/usr/games". Could you please try to create simple job, which would contain Shell task "which ssh" (eventually "which ssh; echo $PATH") and let me know the result. I need to find some reproducer - running this on Jenkins trunk everything look fine, so it seems to me that the root issue is somewhere else than in Jenkins core (and in my commit mentioned above). Thanks.

          vjuranek added a comment - @Pavel: I of course check attached files. "ssh: No such file or directory" can be caused by various many things, and what confuses me is attached system information, where PATH is setup: "PATH /usr/local/bin:/usr/bin:/bin:/usr/games". Could you please try to create simple job, which would contain Shell task "which ssh" (eventually "which ssh; echo $PATH") and let me know the result. I need to find some reproducer - running this on Jenkins trunk everything look fine, so it seems to me that the root issue is somewhere else than in Jenkins core (and in my commit mentioned above). Thanks.

          Luis Arias added a comment -

          I wonder if this has anything to do with my setup having started out as an old hudson installation ? I'll try the which ssh job for you on a new install but I'll have to wait for the end of the day to do the same on our build machine.

          Luis Arias added a comment - I wonder if this has anything to do with my setup having started out as an old hudson installation ? I'll try the which ssh job for you on a new install but I'll have to wait for the end of the day to do the same on our build machine.

          Luis Arias added a comment -

          which ssh works fine on a new install of 1.534.

          Luis Arias added a comment - which ssh works fine on a new install of 1.534.

          vjuranek added a comment -

          @Luis: there can be plenty of reasons why it fails, some backward compatibility issue can be one them. It could be also something very specific in your env. setup which didn't appear before, but now, due to some change, causes problems etc. Therefore I'm trying to narrow down the area where to look, in the best case find the reproducer.

          vjuranek added a comment - @Luis: there can be plenty of reasons why it fails, some backward compatibility issue can be one them. It could be also something very specific in your env. setup which didn't appear before, but now, due to some change, causes problems etc. Therefore I'm trying to narrow down the area where to look, in the best case find the reproducer.

          Luis Arias added a comment -

          Ok guys, I found the root cause, we have a PATH variable defined in the jenkins config which in our case the value is $PATH:/var/lib/jenkins/... . I tried defining a PATH variable in this way on my clean install and the which ssh job gives the following messages in the console:

          Started by user anonymous
          Building in workspace /var/lib/jenkins/jobs/which ssh/workspace
          [workspace] $ /bin/sh -xe /tmp/hudson6953420740987669272.sh
          + which ssh
          /tmp/hudson6953420740987669272.sh: 2: /tmp/hudson6953420740987669272.sh: which: not found
          Build step 'Execute shell' marked build as failure
          Finished: FAILURE
          

          Luis Arias added a comment - Ok guys, I found the root cause, we have a PATH variable defined in the jenkins config which in our case the value is $PATH:/var/lib/jenkins/... . I tried defining a PATH variable in this way on my clean install and the which ssh job gives the following messages in the console: Started by user anonymous Building in workspace / var /lib/jenkins/jobs/which ssh/workspace [workspace] $ /bin/sh -xe /tmp/hudson6953420740987669272.sh + which ssh /tmp/hudson6953420740987669272.sh: 2: /tmp/hudson6953420740987669272.sh: which: not found Build step 'Execute shell' marked build as failure Finished: FAILURE

          Luis Arias added a comment -

          So I believe now the cause is indeed

          https://github.com/jenkinsci/jenkins/commit/3eea39648943583a530f47f69f93737266537d9d

          the reason is that on the first (and new pass) the overrides overwrite any existing values in the environment, in our case the original PATH of the jenkins process itself.

          Luis Arias added a comment - So I believe now the cause is indeed https://github.com/jenkinsci/jenkins/commit/3eea39648943583a530f47f69f93737266537d9d the reason is that on the first (and new pass) the overrides overwrite any existing values in the environment, in our case the original PATH of the jenkins process itself.

          Seems you made the offending commit!

          Stephen Connolly added a comment - Seems you made the offending commit!

          do we have an workaround or an ETA for this to be fixed?
          Thank you!

          Voiculescu Bogdan Alexandru added a comment - do we have an workaround or an ETA for this to be fixed? Thank you!

          vjuranek added a comment -

          yup, looks so, it's really 3eea39648943583a530f47f69f93737266537d9d. Thanks Luis for investigation! As for ETA, I already have a fix, later today I'll try to figure out some tests and will send PR.

          vjuranek added a comment - yup, looks so, it's really 3eea39648943583a530f47f69f93737266537d9d. Thanks Luis for investigation! As for ETA, I already have a fix, later today I'll try to figure out some tests and will send PR.

          vjuranek added a comment -

          https://github.com/jenkinsci/jenkins/pull/970

          please review and sorry for offending 3eea39648943583a530f47f69f93737266537d9d

          vjuranek added a comment - https://github.com/jenkinsci/jenkins/pull/970 please review and sorry for offending 3eea39648943583a530f47f69f93737266537d9d

          Code changed in jenkins
          User: Vojtech Juranek
          Path:
          core/src/main/java/hudson/Launcher.java
          test/src/test/java/hudson/LauncherTest.java
          http://jenkins-ci.org/commit/jenkins/b95f2694529d22523701e8fe20f4f20b2f4d9fc4
          Log:
          [FIXED JENKINS-19926] Add only new env. variables

          Previous patch (3eea39648943583a530f47f69f93737266537d9d) broke usescases when evn. variable overwrites already existing one
          like PATH = $PATH:/my/path

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Vojtech Juranek Path: core/src/main/java/hudson/Launcher.java test/src/test/java/hudson/LauncherTest.java http://jenkins-ci.org/commit/jenkins/b95f2694529d22523701e8fe20f4f20b2f4d9fc4 Log: [FIXED JENKINS-19926] Add only new env. variables Previous patch (3eea39648943583a530f47f69f93737266537d9d) broke usescases when evn. variable overwrites already existing one like PATH = $PATH:/my/path

          Code changed in jenkins
          User: Nicolas De loof
          Path:
          core/src/main/java/hudson/Launcher.java
          test/src/test/java/hudson/LauncherTest.java
          http://jenkins-ci.org/commit/jenkins/86fe60eee7101a3416108c72fcb68fdc6f8c041b
          Log:
          Merge pull request #970 from vjuranek/JENKINS-19926

          [FIXED JENKINS-19926] Add only new env. variables

          Compare: https://github.com/jenkinsci/jenkins/compare/71afe8f0a994...86fe60eee710

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nicolas De loof Path: core/src/main/java/hudson/Launcher.java test/src/test/java/hudson/LauncherTest.java http://jenkins-ci.org/commit/jenkins/86fe60eee7101a3416108c72fcb68fdc6f8c041b Log: Merge pull request #970 from vjuranek/ JENKINS-19926 [FIXED JENKINS-19926] Add only new env. variables Compare: https://github.com/jenkinsci/jenkins/compare/71afe8f0a994...86fe60eee710

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2951
          [FIXED JENKINS-19926] Add only new env. variables (Revision b95f2694529d22523701e8fe20f4f20b2f4d9fc4)

          Result = UNSTABLE
          Vojtech Juranek : b95f2694529d22523701e8fe20f4f20b2f4d9fc4
          Files :

          • core/src/main/java/hudson/Launcher.java
          • test/src/test/java/hudson/LauncherTest.java

          dogfood added a comment - Integrated in jenkins_main_trunk #2951 [FIXED JENKINS-19926] Add only new env. variables (Revision b95f2694529d22523701e8fe20f4f20b2f4d9fc4) Result = UNSTABLE Vojtech Juranek : b95f2694529d22523701e8fe20f4f20b2f4d9fc4 Files : core/src/main/java/hudson/Launcher.java test/src/test/java/hudson/LauncherTest.java

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          http://jenkins-ci.org/commit/jenkins/304b2c071daf0ce0e604d01e37cbbf5f01d89a8e
          Log:
          JENKINS-19926 Noting retroactively.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html http://jenkins-ci.org/commit/jenkins/304b2c071daf0ce0e604d01e37cbbf5f01d89a8e Log: JENKINS-19926 Noting retroactively.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2976
          JENKINS-19926 Noting retroactively. (Revision 304b2c071daf0ce0e604d01e37cbbf5f01d89a8e)

          Result = SUCCESS
          Jesse Glick : 304b2c071daf0ce0e604d01e37cbbf5f01d89a8e
          Files :

          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #2976 JENKINS-19926 Noting retroactively. (Revision 304b2c071daf0ce0e604d01e37cbbf5f01d89a8e) Result = SUCCESS Jesse Glick : 304b2c071daf0ce0e604d01e37cbbf5f01d89a8e Files : changelog.html

          Code changed in jenkins
          User: ikedam
          Path:
          core/src/main/java/hudson/EnvVars.java
          core/src/main/java/hudson/Launcher.java
          core/src/test/java/hudson/EnvVarsTest.java
          http://jenkins-ci.org/commit/jenkins/4204d22496a77c2ccc0bb9f959592a01e27fb54d
          Log:
          [FIX JENKINS-20280] Calculates the order to override variables. This is an improved fix for JENKINS-19488 and JENKINS-19926.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: ikedam Path: core/src/main/java/hudson/EnvVars.java core/src/main/java/hudson/Launcher.java core/src/test/java/hudson/EnvVarsTest.java http://jenkins-ci.org/commit/jenkins/4204d22496a77c2ccc0bb9f959592a01e27fb54d Log: [FIX JENKINS-20280] Calculates the order to override variables. This is an improved fix for JENKINS-19488 and JENKINS-19926 .

          dogfood added a comment -

          Integrated in jenkins_main_trunk #2984
          [FIX JENKINS-20280] Calculates the order to override variables. This is an improved fix for JENKINS-19488 and JENKINS-19926. (Revision 4204d22496a77c2ccc0bb9f959592a01e27fb54d)

          Result = SUCCESS
          kohsuke : 4204d22496a77c2ccc0bb9f959592a01e27fb54d
          Files :

          • core/src/main/java/hudson/EnvVars.java
          • core/src/test/java/hudson/EnvVarsTest.java
          • core/src/main/java/hudson/Launcher.java

          dogfood added a comment - Integrated in jenkins_main_trunk #2984 [FIX JENKINS-20280] Calculates the order to override variables. This is an improved fix for JENKINS-19488 and JENKINS-19926 . (Revision 4204d22496a77c2ccc0bb9f959592a01e27fb54d) Result = SUCCESS kohsuke : 4204d22496a77c2ccc0bb9f959592a01e27fb54d Files : core/src/main/java/hudson/EnvVars.java core/src/test/java/hudson/EnvVarsTest.java core/src/main/java/hudson/Launcher.java

            vjuranek vjuranek
            lemrouch Pavel Mateja
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: