Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62486

Multibranch jobs rebuild and lose build history when source control is down

    XMLWordPrintable

Details

    Description

      We used Mercurial (hg) for source control and most of our Jenkins jobs are multi-branch jobs.  They all have skipDefaultCheckout set to false and have a build step that calls `deleteDir` and then `checkout scm` (because it doesn't appear as though default checkout can do a clean checkout.)

      When our version control server is goes down and then comes back up (like a reboot), all the multibranch builds are triggered even though nothing in the code has changed.  Additionally, the lose all their build history

       

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-48867 Mercurial multibranch pipeline deletes all branches when repository is down

          • Major - Major loss of function.
          • Resolved

          Activity

            jessebs jessebs added a comment -

            Some more details.  We're no longer deleting the directory but instead doing a clean checkout.  The problem still exists. 

            It appears as though Jenkins will delete all known details of branches in the time our source control server is down.

            Here's a relevant log:

            Started by timer
[Wed May 27 17:50:14 UTC 2020] Starting branch indexing...
Acquired master cache lock.
[85701BDF9C7C8AEC476538CEBD36486C4DA433CD-repo] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" pull
pulling from https://example.com/scm/hg/repo
abort: HTTP Error 502: Bad Gateway
ERROR: Failed to update /var/lib/jenkins/hgcache/85701BDF9C7C8AEC476538CEBD36486C4DA433CD-repo
Master cache lock released.
ERROR: Could not use caches, not fetching branch heads
[Wed May 27 17:50:14 UTC 2020] Finished branch indexing. Indexing took 0.28 sec
Evaluating orphaned items in repo
Will remove default
Finished: SUCCESS

            I would think that the unsuccessful response would cause the job to fail, thereby avoiding branch indexing and removal of branches

            jessebs jessebs added a comment - Some more details.  We're no longer deleting the directory but instead doing a clean checkout.  The problem still exists.  It appears as though Jenkins will delete all known details of branches in the time our source control server is down. Here's a relevant log: Started by timer [Wed May 27 17:50:14 UTC 2020] Starting branch indexing... Acquired master cache lock. [85701BDF9C7C8AEC476538CEBD36486C4DA433CD-repo] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" pull pulling from https://example.com/scm/hg/repo abort: HTTP Error 502: Bad Gateway ERROR: Failed to update /var/lib/jenkins/hgcache/85701BDF9C7C8AEC476538CEBD36486C4DA433CD-repo Master cache lock released. ERROR: Could not use caches, not fetching branch heads [Wed May 27 17:50:14 UTC 2020] Finished branch indexing. Indexing took 0.28 sec Evaluating orphaned items in repo Will remove default Finished: SUCCESS I would think that the unsuccessful response would cause the job to fail, thereby avoiding branch indexing and removal of branches

            People

              vaimr Denis Saponenko
              jessebs jessebs
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: