Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19900

Mercurial polling stopped working; never triggers a build

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Recently, builds that poll Bitbucket using mercurial stopped working. The polling never triggers a build. Triggering these builds manually works no problem, so mercurial is working.

      The poll line looks like:

      H/5 * * * *

      The poll log says the following:

      Started on Oct 4, 2013 12:06:35 PM
      Polling SCM changes on master
      [staging] $ hg pull --rev staging
      remote: /bin/sh: ssh: command not found
      abort: no suitable response from remote hg!
      [staging] $ hg log --rev staging --template

      {node}

      [staging] $ hg log --rev staging --template

      {rev}

      Done. Took 0.48 sec
      No changes

      This machine absolutely has SSH and Mercurial installed (manual builds work).

      Jenkins 1.533
      Jenkins Mailer Plugin 1.47

        Attachments

          Activity

          Hide
          jdufresne Jon Dufresne added a comment -

          Does the Mercurial polling run in some very limited isolated environment? If so, is there a way I could debug this environment to test what my path is and if ssh is recognized?

          As the Jenkins user, I see:

          -bash-4.1$ echo $PATH
          /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin
          -bash-4.1$ which ssh
          /usr/bin/ssh

          Show
          jdufresne Jon Dufresne added a comment - Does the Mercurial polling run in some very limited isolated environment? If so, is there a way I could debug this environment to test what my path is and if ssh is recognized? As the Jenkins user, I see: -bash-4.1$ echo $PATH /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin -bash-4.1$ which ssh /usr/bin/ssh
          Hide
          jglick Jesse Glick added a comment -

          Does the Mercurial polling run in some very limited isolated environment?

          Not really, it should just be the same as the Jenkins user running the command printed in the polling log.

          The location of SSH on your machine is probably irrelevant, since the error message is coming from the BitBucket side. The precise form of the remote URL you specify in the job might matter (and check paths.default in the workspace .hg/hgrc).

          Just checked using 1.533 and 1.47 on Ubuntu 13.04 with my private BitBucket test repository (SSH clone URL). Created a job with SCM polling trigger, using an existing branch, ran a build, did another commit on that branch, waited, and the new build appeared. The build’s polling log looks normal:

          Started on Oct 8, 2013 8:39:46 AM
          Polling SCM changes on master
          [workspace] $ hg pull --rev development
          pulling from ssh://hg@bitbucket.org/jglick/proprietary
          searching for changes
          adding changesets
          adding manifests
          adding file changes
          added 1 changesets with 1 changes to 1 files
          (run 'hg update' to get a working copy)
          [workspace] $ hg log --rev development --template {node}
          [workspace] $ hg log --rev development --template {rev}
          [workspace] $ hg status --rev 7695fd1e132a5abe9cb3450ea31d0d7722a35ba6 --rev b2691731d705de01bd5a4833a6b63de5b7af17d9
          Dependent changes detected
          Done. Took 2.2 sec
          Changes found
          
          Show
          jglick Jesse Glick added a comment - Does the Mercurial polling run in some very limited isolated environment? Not really, it should just be the same as the Jenkins user running the command printed in the polling log. The location of SSH on your machine is probably irrelevant, since the error message is coming from the BitBucket side. The precise form of the remote URL you specify in the job might matter (and check paths.default in the workspace .hg/hgrc ). Just checked using 1.533 and 1.47 on Ubuntu 13.04 with my private BitBucket test repository (SSH clone URL). Created a job with SCM polling trigger, using an existing branch, ran a build, did another commit on that branch, waited, and the new build appeared. The build’s polling log looks normal: Started on Oct 8, 2013 8:39:46 AM Polling SCM changes on master [workspace] $ hg pull --rev development pulling from ssh://hg@bitbucket.org/jglick/proprietary searching for changes adding changesets adding manifests adding file changes added 1 changesets with 1 changes to 1 files (run 'hg update' to get a working copy) [workspace] $ hg log --rev development --template {node} [workspace] $ hg log --rev development --template {rev} [workspace] $ hg status --rev 7695fd1e132a5abe9cb3450ea31d0d7722a35ba6 --rev b2691731d705de01bd5a4833a6b63de5b7af17d9 Dependent changes detected Done. Took 2.2 sec Changes found
          Hide
          jglick Jesse Glick added a comment -

          Closing since there is nothing to be done on the Jenkins side unless you can figure out why BitBucket is sending an error message under some, but not other, conditions, and demonstrate that the Mercurial plugin is calling hg with the wrong arguments or environment. I would suggest contacting Atlassian support.

          Show
          jglick Jesse Glick added a comment - Closing since there is nothing to be done on the Jenkins side unless you can figure out why BitBucket is sending an error message under some, but not other, conditions, and demonstrate that the Mercurial plugin is calling hg with the wrong arguments or environment. I would suggest contacting Atlassian support.
          Hide
          jdufresne Jon Dufresne added a comment -

          In case you're interested, I was able to resolve the issue by creating the following `~/.hgrc`

          $ cat ~/.hgrc
          [ui]
          ssh=/usr/bin/ssh
          

          The poll log now reads as follows:

          Started on Oct 8, 2013 12:48:10 PM
          Polling SCM changes on master
          [development] $ hg pull --rev development
          pulling from ssh://hg@bitbucket.org/<USER>/<REPO>
          searching for changes
          no changes found
          [development] $ hg log --rev development --template {node}
          [development] $ hg log --rev development --template {rev}
          Done. Took 2.7 sec
          No changes
          

          Why Jenkins was working previously, from the command line, and from manual builds, but not from SCM polling I do not know. I still think this indicates a bug in the polling plugin, but if you think otherwise, at least I have a working installation. If you are interested, I am willing to conduct additional tests if you wish to further diagnose this issue. If you have no interest, I'll stop bugging you .

          Thank you for your patience and your help.

          Show
          jdufresne Jon Dufresne added a comment - In case you're interested, I was able to resolve the issue by creating the following `~/.hgrc` $ cat ~/.hgrc [ui] ssh=/usr/bin/ssh The poll log now reads as follows: Started on Oct 8, 2013 12:48:10 PM Polling SCM changes on master [development] $ hg pull --rev development pulling from ssh: //hg@bitbucket.org/<USER>/<REPO> searching for changes no changes found [development] $ hg log --rev development --template {node} [development] $ hg log --rev development --template {rev} Done. Took 2.7 sec No changes Why Jenkins was working previously, from the command line, and from manual builds, but not from SCM polling I do not know. I still think this indicates a bug in the polling plugin, but if you think otherwise, at least I have a working installation. If you are interested, I am willing to conduct additional tests if you wish to further diagnose this issue. If you have no interest, I'll stop bugging you . Thank you for your patience and your help.
          Hide
          jglick Jesse Glick added a comment -

          Interesting. If you want to dig into Mercurial sources, ui.ssh seems to be read in mercurial/sshpeer.py. The fact that setting it to /usr/bin/ssh makes things work implies that the default value (just ssh) is somehow defective in your environment, though I cannot guess how exactly (based on the output of which ssh). You should turn on the debug flag on your Mercurial installation in Jenkins configuration, and if necessary get out a Java debugger.

          It occurs to me that since you only seem to be testing with 1.533, you might be suffering from JENKINS-19307; try 1.526, or (currently in release candidate status) 1.535. The known symptoms are different from yours, but this bug does affect SCM polling based on external processes generally, and you might have found an obscure variant of it. Again I do not have a specific proposal for how that bug would cause Mercurial, and BitBucket, to behave in the odd way you describe; it is just a guess.

          Show
          jglick Jesse Glick added a comment - Interesting. If you want to dig into Mercurial sources, ui.ssh seems to be read in mercurial/sshpeer.py . The fact that setting it to /usr/bin/ssh makes things work implies that the default value (just ssh ) is somehow defective in your environment, though I cannot guess how exactly (based on the output of which ssh ). You should turn on the debug flag on your Mercurial installation in Jenkins configuration, and if necessary get out a Java debugger. It occurs to me that since you only seem to be testing with 1.533, you might be suffering from JENKINS-19307 ; try 1.526, or (currently in release candidate status) 1.535. The known symptoms are different from yours, but this bug does affect SCM polling based on external processes generally, and you might have found an obscure variant of it. Again I do not have a specific proposal for how that bug would cause Mercurial, and BitBucket, to behave in the odd way you describe; it is just a guess.

            People

            Assignee:
            jglick Jesse Glick
            Reporter:
            jdufresne Jon Dufresne
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: