Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48867

Mercurial multibranch pipeline deletes all branches when repository is down

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: mercurial-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.101
    • Similar Issues:

      Description

      A multibranch job that pulls from mercurial deletes all branches when that repository goes down. This was discovered during Bitbucket's outage going on today, and I lost all the branch build history.

      Steps to reproduce:

      1. Create a local mercurial repo with below basic Jenkinsfile and publish using hg serve
      2. Start vanilla weekly docker image
      3. Install multibranch, mercurial
      4. Create multibranch job pointing at that repo, let it scan and build default branch
      5. Kill hg serve
      6. Run "Scan Multibranch Pipeline Now", all branches get deleted

      Basic jenkinsfile

      pipeline {
          agent any
      	stages {
      		stage("Build Data") {
      			steps {
      				sh 'echo yay'
      			}
      		}
          }
      }
      

      Logs

      [Tue Jan 09 17:19:31 UTC 2018] Starting branch indexing...
      Acquired master cache lock.
      [D69ADC747155FE53F1452F758ACEC24C520106F9-lblakey-ws1] $ hg pull
      pulling from http://myWorkstation:8000/
      abort: error: Connection refused
      ERROR: Failed to update /var/jenkins_home/hgcache/D69ADC747155FE53F1452F758ACEC24C520106F9-myWorkstation
      Master cache lock released.
      ERROR: Could not use caches, not fetching branch heads
      [Tue Jan 09 17:19:31 UTC 2018] Finished branch indexing. Indexing took 66 ms
      Evaluating orphaned items in test
      Will remove default
      Finished: SUCCESS
      

        Attachments

          Issue Links

            Activity

            Hide
            wynx WynX Alucard added a comment - - edited

            Recently I've also encountered this with a local mercurial server (mercurial-scm) not being available. Wiped all builds and branches, very problematic! Note that in my case the account used to access mercurial was (temporarily) blocked, resulting in the same behaviour.

             

            Jenkins v2.107.3 (current LTS)

            Mercurial plugin v2.3

            SCM API plugin 2.2.7

            Show
            wynx WynX Alucard added a comment - - edited Recently I've also encountered this with a local mercurial server (mercurial-scm) not being available. Wiped all builds and branches, very problematic! Note that in my case the account used to access mercurial was (temporarily) blocked, resulting in the same behaviour.   Jenkins v2.107.3 (current LTS) Mercurial plugin v2.3 SCM API plugin 2.2.7
            Hide
            brewmook Dave E added a comment - - edited

            I got this too.

            Short-term workaround, either of the following under "Orphaned Item Strategy":

            • Switch off "Discard old items" entirely, and take care of that by hand
            • Set positive number in "Max # of old items to keep"/"Days to keep old items".

            Your mileage may vary depending on how often your Mercurial server goes down.

            Having dug into this a bit, I believe the orphaned item stuff is behaving as expected.

            The curious bit is the "Could not use caches, not fetching branch heads" error. Looking in MercurialSCMSource.java, the retrieve methods simply returns if the cache is null. Should this not be throwing an exception instead? The knock-on effect seems to be that the orphaned item strategy kicks in and all the branches are marked as closed. I would expect the polling/indexing to abort entirely if the cache is not available/working.

            Show
            brewmook Dave E added a comment - - edited I got this too. Short-term workaround, either of the following under "Orphaned Item Strategy": Switch off "Discard old items" entirely, and take care of that by hand Set positive number in "Max # of old items to keep"/"Days to keep old items". Your mileage may vary depending on how often your Mercurial server goes down. Having dug into this a bit, I believe the orphaned item stuff is behaving as expected. The curious bit is the "Could not use caches, not fetching branch heads" error. Looking in MercurialSCMSource.java , the retrieve methods simply returns if the cache is null. Should this not be throwing an exception instead? The knock-on effect seems to be that the orphaned item strategy kicks in and all the branches are marked as closed. I would expect the polling/indexing to abort entirely if the cache is not available/working.
            Hide
            brewmook Dave E added a comment -

            It doesn't help that the next consequence of this is a build storm when the mercurial server comes back up. All branches are marked as reopened and everything rebuilds. We currently have some issues with our hg server, and it reboots several times a week, so the build storms are proving to be the most frustrating consequence.

            Show
            brewmook Dave E added a comment - It doesn't help that the next consequence of this is a build storm when the mercurial server comes back up. All branches are marked as reopened and everything rebuilds. We currently have some issues with our hg server, and it reboots several times a week, so the build storms are proving to be the most frustrating consequence.
            Hide
            mmchr Mads Mohr Christensen added a comment - - edited

            This issue also deleted my build history because of a temporary outage at a bad time.

            I have opened a PR https://github.com/jenkinsci/mercurial-plugin/pull/144 implementing the suggestion by Dave E

            Show
            mmchr Mads Mohr Christensen added a comment - - edited This issue also deleted my build history because of a temporary outage at a bad time. I have opened a PR  https://github.com/jenkinsci/mercurial-plugin/pull/144  implementing the suggestion by Dave E

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              thelq Leon Blakey
              Votes:
              5 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: