Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62486

Multibranch jobs rebuild and lose build history when source control is down

    XMLWordPrintable

Details

    Description

      We used Mercurial (hg) for source control and most of our Jenkins jobs are multi-branch jobs.  They all have skipDefaultCheckout set to false and have a build step that calls `deleteDir` and then `checkout scm` (because it doesn't appear as though default checkout can do a clean checkout.)

      When our version control server is goes down and then comes back up (like a reboot), all the multibranch builds are triggered even though nothing in the code has changed.  Additionally, the lose all their build history

       

      Attachments

        Issue Links

          Activity

            jessebs jessebs added a comment -

            Some more details.  We're no longer deleting the directory but instead doing a clean checkout.  The problem still exists. 

            It appears as though Jenkins will delete all known details of branches in the time our source control server is down.

            Here's a relevant log:

            Started by timer
            [Wed May 27 17:50:14 UTC 2020] Starting branch indexing...
            Acquired master cache lock.
            [85701BDF9C7C8AEC476538CEBD36486C4DA433CD-repo] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" pull
            pulling from https://example.com/scm/hg/repo
            abort: HTTP Error 502: Bad Gateway
            ERROR: Failed to update /var/lib/jenkins/hgcache/85701BDF9C7C8AEC476538CEBD36486C4DA433CD-repo
            Master cache lock released.
            ERROR: Could not use caches, not fetching branch heads
            [Wed May 27 17:50:14 UTC 2020] Finished branch indexing. Indexing took 0.28 sec
            Evaluating orphaned items in repo
            Will remove default
            Finished: SUCCESS

            I would think that the unsuccessful response would cause the job to fail, thereby avoiding branch indexing and removal of branches

            jessebs jessebs added a comment - Some more details.  We're no longer deleting the directory but instead doing a clean checkout.  The problem still exists.  It appears as though Jenkins will delete all known details of branches in the time our source control server is down. Here's a relevant log: Started by timer [Wed May 27 17:50:14 UTC 2020] Starting branch indexing... Acquired master cache lock. [85701BDF9C7C8AEC476538CEBD36486C4DA433CD-repo] $ hg --config auth.jenkins.prefix=* --config ******** --config ******** --config "auth.jenkins.schemes=http https" pull pulling from https://example.com/scm/hg/repo abort: HTTP Error 502: Bad Gateway ERROR: Failed to update /var/lib/jenkins/hgcache/85701BDF9C7C8AEC476538CEBD36486C4DA433CD-repo Master cache lock released. ERROR: Could not use caches, not fetching branch heads [Wed May 27 17:50:14 UTC 2020] Finished branch indexing. Indexing took 0.28 sec Evaluating orphaned items in repo Will remove default Finished: SUCCESS I would think that the unsuccessful response would cause the job to fail, thereby avoiding branch indexing and removal of branches

            People

              vaimr Denis Saponenko
              jessebs jessebs
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: