Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
Jenkins ver. 2.101
Description
A multibranch job that pulls from mercurial deletes all branches when that repository goes down. This was discovered during Bitbucket's outage going on today, and I lost all the branch build history.
Steps to reproduce:
- Create a local mercurial repo with below basic Jenkinsfile and publish using hg serve
- Start vanilla weekly docker image
- Install multibranch, mercurial
- Create multibranch job pointing at that repo, let it scan and build default branch
- Kill hg serve
- Run "Scan Multibranch Pipeline Now", all branches get deleted
Basic jenkinsfile
pipeline {
agent any
stages {
stage("Build Data") {
steps {
sh 'echo yay'
}
}
}
}
Logs
[Tue Jan 09 17:19:31 UTC 2018] Starting branch indexing... Acquired master cache lock. [D69ADC747155FE53F1452F758ACEC24C520106F9-lblakey-ws1] $ hg pull pulling from http://myWorkstation:8000/ abort: error: Connection refused ERROR: Failed to update /var/jenkins_home/hgcache/D69ADC747155FE53F1452F758ACEC24C520106F9-myWorkstation Master cache lock released. ERROR: Could not use caches, not fetching branch heads [Tue Jan 09 17:19:31 UTC 2018] Finished branch indexing. Indexing took 66 ms Evaluating orphaned items in test Will remove default Finished: SUCCESS
Attachments
Issue Links
- is duplicated by
-
-
- Resolved
-
- links to
Activity
I got this too.
Short-term workaround, either of the following under "Orphaned Item Strategy":
- Switch off "Discard old items" entirely, and take care of that by hand
- Set positive number in "Max # of old items to keep"/"Days to keep old items".
Your mileage may vary depending on how often your Mercurial server goes down.
Having dug into this a bit, I believe the orphaned item stuff is behaving as expected.
The curious bit is the "Could not use caches, not fetching branch heads" error. Looking in MercurialSCMSource.java, the retrieve methods simply returns if the cache is null. Should this not be throwing an exception instead? The knock-on effect seems to be that the orphaned item strategy kicks in and all the branches are marked as closed. I would expect the polling/indexing to abort entirely if the cache is not available/working.
Dave E
added a comment - - edited I got this too.
Short-term workaround, either of the following under "Orphaned Item Strategy":
Switch off "Discard old items" entirely, and take care of that by hand
Set positive number in "Max # of old items to keep"/"Days to keep old items".
Your mileage may vary depending on how often your Mercurial server goes down.
Having dug into this a bit, I believe the orphaned item stuff is behaving as expected.
The curious bit is the "Could not use caches, not fetching branch heads" error. Looking in MercurialSCMSource.java , the retrieve methods simply returns if the cache is null. Should this not be throwing an exception instead? The knock-on effect seems to be that the orphaned item strategy kicks in and all the branches are marked as closed. I would expect the polling/indexing to abort entirely if the cache is not available/working. Dave E
added a comment - It doesn't help that the next consequence of this is a build storm when the mercurial server comes back up. All branches are marked as reopened and everything rebuilds. We currently have some issues with our hg server, and it reboots several times a week, so the build storms are proving to be the most frustrating consequence.
Dave E
added a comment - It doesn't help that the next consequence of this is a build storm when the mercurial server comes back up. All branches are marked as reopened and everything rebuilds. We currently have some issues with our hg server, and it reboots several times a week, so the build storms are proving to be the most frustrating consequence. This issue also deleted my build history because of a temporary outage at a bad time.
I have opened a PR https://github.com/jenkinsci/mercurial-plugin/pull/144 implementing the suggestion by brewmook
Mads Mohr Christensen
added a comment - - edited This issue also deleted my build history because of a temporary outage at a bad time.
I have opened a PR https://github.com/jenkinsci/mercurial-plugin/pull/144 implementing the suggestion by brewmook 
Recently I've also encountered this with a local mercurial server (mercurial-scm) not being available. Wiped all builds and branches, very problematic! Note that in my case the account used to access mercurial was (temporarily) blocked, resulting in the same behaviour.
Jenkins v2.107.3 (current LTS)
Mercurial plugin v2.3
SCM API plugin 2.2.7