Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48867

Mercurial multibranch pipeline deletes all branches when repository is down

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • mercurial-plugin
    • None
    • Jenkins ver. 2.101

    Description

      A multibranch job that pulls from mercurial deletes all branches when that repository goes down. This was discovered during Bitbucket's outage going on today, and I lost all the branch build history.

      Steps to reproduce:

      1. Create a local mercurial repo with below basic Jenkinsfile and publish using hg serve
      2. Start vanilla weekly docker image
      3. Install multibranch, mercurial
      4. Create multibranch job pointing at that repo, let it scan and build default branch
      5. Kill hg serve
      6. Run "Scan Multibranch Pipeline Now", all branches get deleted

      Basic jenkinsfile

      pipeline {
          agent any
      	stages {
      		stage("Build Data") {
      			steps {
      				sh 'echo yay'
      			}
      		}
          }
      }
      

      Logs

      [Tue Jan 09 17:19:31 UTC 2018] Starting branch indexing...
      Acquired master cache lock.
      [D69ADC747155FE53F1452F758ACEC24C520106F9-lblakey-ws1] $ hg pull
      pulling from http://myWorkstation:8000/
      abort: error: Connection refused
      ERROR: Failed to update /var/jenkins_home/hgcache/D69ADC747155FE53F1452F758ACEC24C520106F9-myWorkstation
      Master cache lock released.
      ERROR: Could not use caches, not fetching branch heads
      [Tue Jan 09 17:19:31 UTC 2018] Finished branch indexing. Indexing took 66 ms
      Evaluating orphaned items in test
      Will remove default
      Finished: SUCCESS
      

      Attachments

        Issue Links

          Activity

            thelq Leon Blakey created issue -
            thelq Leon Blakey made changes -
            Field Original Value New Value
            Description A multibranch job that pulls from mercurial deletes all branches when that repository goes down. This was discovered during Bitbucket's outage going on today, and I lost all the branch build history.

            Steps to reproduce:

            # Create a local mercurial repo with below basic Jenkinsfile and publish using hg serve
            # Start vanilla weekly docker image
            # Install multibranch, mercurial
            # Create multibranch job pointing at that repo, let it scan and build default branch
            # Kill hg serve
            # Run "Scan Multibranch Pipeline Now", all branches get deleted

            Basic jenkinsfile

            {noformat}
            pipeline {
                agent any
            stages {
            stage("Build Data") {
            steps {
            sh 'echo yay'
            }
            }
                }
            }
            {noformat}
            A multibranch job that pulls from mercurial deletes all branches when that repository goes down. This was discovered during Bitbucket's outage going on today, and I lost all the branch build history.

            Steps to reproduce:

            # Create a local mercurial repo with below basic Jenkinsfile and publish using hg serve
            # Start vanilla weekly docker image
            # Install multibranch, mercurial
            # Create multibranch job pointing at that repo, let it scan and build default branch
            # Kill hg serve
            # Run "Scan Multibranch Pipeline Now", all branches get deleted

            Basic jenkinsfile

            {noformat}
            pipeline {
                agent any
            stages {
            stage("Build Data") {
            steps {
            sh 'echo yay'
            }
            }
                }
            }
            {noformat}

            Logs

            {noformat}
            [Tue Jan 09 17:19:31 UTC 2018] Starting branch indexing...
            Acquired master cache lock.
            [D69ADC747155FE53F1452F758ACEC24C520106F9-lblakey-ws1] $ hg pull
            pulling from http://myWorkstation:8000/
            abort: error: Connection refused
            ERROR: Failed to update /var/jenkins_home/hgcache/D69ADC747155FE53F1452F758ACEC24C520106F9-myWorkstation
            Master cache lock released.
            ERROR: Could not use caches, not fetching branch heads
            [Tue Jan 09 17:19:31 UTC 2018] Finished branch indexing. Indexing took 66 ms
            Evaluating orphaned items in test
            Will remove default
            Finished: SUCCESS
            {noformat}
            abayer Andrew Bayer made changes -
            Component/s workflow-multibranch-plugin [ 21465 ]
            wynx WynX Alucard added a comment - - edited

            Recently I've also encountered this with a local mercurial server (mercurial-scm) not being available. Wiped all builds and branches, very problematic! Note that in my case the account used to access mercurial was (temporarily) blocked, resulting in the same behaviour.

             

            Jenkins v2.107.3 (current LTS)

            Mercurial plugin v2.3

            SCM API plugin 2.2.7

            wynx WynX Alucard added a comment - - edited Recently I've also encountered this with a local mercurial server (mercurial-scm) not being available. Wiped all builds and branches, very problematic! Note that in my case the account used to access mercurial was (temporarily) blocked, resulting in the same behaviour.   Jenkins v2.107.3 (current LTS) Mercurial plugin v2.3 SCM API plugin 2.2.7
            brewmook Dave E added a comment - - edited

            I got this too.

            Short-term workaround, either of the following under "Orphaned Item Strategy":

            • Switch off "Discard old items" entirely, and take care of that by hand
            • Set positive number in "Max # of old items to keep"/"Days to keep old items".

            Your mileage may vary depending on how often your Mercurial server goes down.

            Having dug into this a bit, I believe the orphaned item stuff is behaving as expected.

            The curious bit is the "Could not use caches, not fetching branch heads" error. Looking in MercurialSCMSource.java, the retrieve methods simply returns if the cache is null. Should this not be throwing an exception instead? The knock-on effect seems to be that the orphaned item strategy kicks in and all the branches are marked as closed. I would expect the polling/indexing to abort entirely if the cache is not available/working.

            brewmook Dave E added a comment - - edited I got this too. Short-term workaround, either of the following under "Orphaned Item Strategy": Switch off "Discard old items" entirely, and take care of that by hand Set positive number in "Max # of old items to keep"/"Days to keep old items". Your mileage may vary depending on how often your Mercurial server goes down. Having dug into this a bit, I believe the orphaned item stuff is behaving as expected. The curious bit is the "Could not use caches, not fetching branch heads" error. Looking in MercurialSCMSource.java , the retrieve methods simply returns if the cache is null. Should this not be throwing an exception instead? The knock-on effect seems to be that the orphaned item strategy kicks in and all the branches are marked as closed. I would expect the polling/indexing to abort entirely if the cache is not available/working.
            brewmook Dave E added a comment -

            It doesn't help that the next consequence of this is a build storm when the mercurial server comes back up. All branches are marked as reopened and everything rebuilds. We currently have some issues with our hg server, and it reboots several times a week, so the build storms are proving to be the most frustrating consequence.

            brewmook Dave E added a comment - It doesn't help that the next consequence of this is a build storm when the mercurial server comes back up. All branches are marked as reopened and everything rebuilds. We currently have some issues with our hg server, and it reboots several times a week, so the build storms are proving to be the most frustrating consequence.
            mmchr Mads Mohr Christensen added a comment - - edited

            This issue also deleted my build history because of a temporary outage at a bad time.

            I have opened a PR https://github.com/jenkinsci/mercurial-plugin/pull/144 implementing the suggestion by brewmook

            mmchr Mads Mohr Christensen added a comment - - edited This issue also deleted my build history because of a temporary outage at a bad time. I have opened a PR  https://github.com/jenkinsci/mercurial-plugin/pull/144  implementing the suggestion by brewmook
            jglick Jesse Glick made changes -
            Assignee Mads Mohr Christensen [ mmchr ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-62486 [ JENKINS-62486 ]
            jglick Jesse Glick made changes -
            Assignee Mads Mohr Christensen [ mmchr ] Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "mercurial-plugin #149 (Web Link)" [ 26106 ]
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Fixed but Unreleased [ 10203 ]
            jglick Jesse Glick made changes -
            Released As https://github.com/jenkinsci/mercurial-plugin/releases/tag/mercurial-2.11
            Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]

            People

              jglick Jesse Glick
              thelq Leon Blakey
              Votes:
              5 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: