Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55388

multibranch pipeline deleting history and building unexpectedly

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major

      I have a multibranch pipeline setup with a Jenkinsfile. I have a release branch (current-release) that isn't updated very often. Every once in a while, around a month or so, Jenkins decides to delete all of the previous builds of the branch and build the branch again. The build number starts at 1 again.

      I checked my repository scan log and it had already turned over since the build, so I don't know what was in there. The most recent build said it started because of branch indexing.

      I have "Discard old items" set to 1 days. My understanding this is just for deleting old branches.

      I'm using a github repository as my source with push notifications turned on.

      In my jenkins stdout/stderr log I see the following. The build occurred at 12:51pm. Could this cleanup possibly have deleted information for the wrong branch?

      INFO: ALPN callback dropped: SPDY and HTTP/2 are disabled. Is alpn-boot on the boot class path?
      Jan 02, 2019 12:33:29 PM jenkins.branch.WorkspaceLocatorImpl getWorkspaceRoot
      WARNING: JENKINS-2111 path sanitization ineffective when using legacy Workspace Root Directory ?${ITEM_ROOTDIR}/workspace?; switch to ?${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}? as in JENKINS-8446 / JENKINS-21942
      Jan 02, 2019 12:33:32 PM jenkins.branch.WorkspaceLocatorImpl getWorkspaceRoot
      WARNING: JENKINS-2111 path sanitization ineffective when using legacy Workspace Root Directory ?${ITEM_ROOTDIR}/workspace?; switch to ?${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}? as in JENKINS-8446 / JENKINS-21942
      Jan 02, 2019 12:33:32 PM jenkins.branch.WorkspaceLocatorImpl$Deleter$CleanupTask run
      INFO: deleting obsolete workspace C:\Users\fll\jenkins\workspace\SW_issue_675_group-name-location on fll-build
      Jan 02, 2019 12:33:32 PM jenkins.branch.MultiBranchProject$BranchIndexing run
      INFO: FLL-SW #20190102.123300 branch indexing action completed: SUCCESS in 32 sec
      Jan 02, 2019 12:34:09 PM jenkins.branch.WorkspaceLocatorImpl$Deleter$CleanupTask run
      INFO: deleting obsolete workspace C:\Users\fll\jenkins\workspace\SW_issue_675_group-name-location@tmp on fll-build
      Jan 02, 2019 12:51:02 PM com.squareup.okhttp.internal.Platform$JdkWithJettyBootPlatform getSelectedProtocol
      

      The question is why is my build history getting deleted?

       

          [JENKINS-55388] multibranch pipeline deleting history and building unexpectedly

          jpschewe added a comment -

          I found a clue. I've been capturing the git index log and found a message in the log just before the the branch was rebuilt.

          [Mon Jun 03 10:39:07 EDT 2019] Finished branch indexing. Indexing took 7.5 sec
          Evaluating orphaned items in FLL-SW
          Will remove current-release as it is too old
          Finished: SUCCESS
           

           

          What is this message about the branch being too old? I want the branch jobs to stay around forever, unless the branch is deleted.

           

          jpschewe added a comment - I found a clue. I've been capturing the git index log and found a message in the log just before the the branch was rebuilt. [Mon Jun 03 10:39:07 EDT 2019] Finished branch indexing. Indexing took 7.5 sec Evaluating orphaned items in FLL-SW Will remove current-release as it is too old Finished: SUCCESS   What is this message about the branch being too old? I want the branch jobs to stay around forever, unless the branch is deleted.  

          jpschewe added a comment -

          I'm using the github plugin for the source. For the others seeing this issue are you also using the github plugin for the source?

          jpschewe added a comment - I'm using the github plugin for the source. For the others seeing this issue are you also using the github plugin for the source?

          jpschewe added a comment -

          More information. Found something else that is odd.

          On scan shows

              Checking branch current-release
                ‘Jenkinsfile’ found
              Met criteria
          No changes detected: current-release (still at d79cc62c7dc3e036229e8bd1adc5a8aab7f9aa57)
          

          The next scan shows

              Checking branch current-release
                ‘Jenkinsfile’ not found
              Does not meet criteria
          

          and this is the one that also says at the bottom that current-release is being removed.

           

          I haven't made any changes to the current release branch and it does in fact have a Jenkinsfile as can be seen by the next scan that says

              Checking branch current-release
                ‘Jenkinsfile’ found
              Met criteria
          Scheduled build for branch: current-release 

          The repository for this Jenkins job is at https://github.com/jpschewe/fll-sw/

          The last commit was February 4, 2019. This particular reset of the history was on June 3, 2019. I think I've had a reset between those two dates, but not since I started monitoring the scan log in May.

           

          jpschewe added a comment - More information. Found something else that is odd. On scan shows Checking branch current-release ‘Jenkinsfile’ found Met criteria No changes detected: current-release (still at d79cc62c7dc3e036229e8bd1adc5a8aab7f9aa57) The next scan shows Checking branch current-release ‘Jenkinsfile’ not found Does not meet criteria and this is the one that also says at the bottom that current-release is being removed.   I haven't made any changes to the current release branch and it does in fact have a Jenkinsfile as can be seen by the next scan that says Checking branch current-release ‘Jenkinsfile’ found Met criteria Scheduled build for branch: current-release The repository for this Jenkins job is at https://github.com/jpschewe/fll-sw/ The last commit was February 4, 2019. This particular reset of the history was on June 3, 2019. I think I've had a reset between those two dates, but not since I started monitoring the scan log in May.  

          Sudheer Gundanna added a comment - - edited

          This happened again today [July-18-2019]for me. Master branch all previous builds were removed and started building from #1.  Is there any resolution for this issue?  this is so frustrating. It has happened 5 times before.

          Jenkins version we are using is 2.164.2

          GitHub plugin: 1.29.4

          Sudheer Gundanna added a comment - - edited This happened again today [July-18-2019] for me. Master branch all previous builds were removed and started building from #1.  Is there any resolution for this issue?  this is so frustrating. It has happened 5 times before. Jenkins version we are using is 2.164.2 GitHub plugin: 1.29.4

          Is this related to JENKINS-54126?

          Suganya Ravikumar added a comment - Is this related to  JENKINS-54126 ?

          Roman Pickl added a comment -

          I see this (or something related) for all branches. once in a while jenkins throws away everything (maybe after a problem with scanning our local github) and restarts all builds.

          Roman Pickl added a comment - I see this (or something related) for all branches. once in a while jenkins throws away everything (maybe after a problem with scanning our local github) and restarts all builds.

          Morgan Robertson added a comment - This seems very similar to  https://issues.jenkins-ci.org/browse/JENKINS-55220

          Do we all have the 'Discard old items' item enabled in Jenkins?  I have a suspicion that what's happening is that GitHub or Bitbucket or whatever git service we use occasionally fails to properly respond to the Jenkins poll request and then Jenkins removes the branch due to the 'Discard old items' option.  I'm going to try disabling this.

          If so, two potential fixes:

          • Allow certain branches to be excepted from the 'Discard old items' job.
          • Make every Bitbucket / GitHub / etc. reliably respond to requests (hah).

          Morgan Robertson added a comment - Do we all have the 'Discard old items' item enabled in Jenkins?  I have a suspicion that what's happening is that GitHub or Bitbucket or whatever git service we use occasionally fails to properly respond to the Jenkins poll request and then Jenkins removes the branch due to the 'Discard old items' option.  I'm going to try disabling this. If so, two potential fixes: Allow certain branches to be excepted from the 'Discard old items' job. Make every Bitbucket / GitHub / etc. reliably respond to requests (hah).

          For us, this was happening because the branch was also picked up as a pull request and was found in the pull request tab. The message in the logs read `Will remove xxx as it is too old`, but when we closed the PR and rescanned, it remained in the branches tab. Not sure why the log messages were so unrelated to the actual issue. 

          Jesse Anderson added a comment - For us, this was happening because the branch was also picked up as a pull request and was found in the pull request tab. The message in the logs read `Will remove xxx as it is too old`, but when we closed the PR and rescanned, it remained in the branches tab. Not sure why the log messages were so unrelated to the actual issue. 

          Stian Jørgensrud added a comment - - edited

          We are also experiencing this! I didn't notice this happening when we had below 50 pipelines in our GitHub organization folder. Noticing it now that we have 61 repositories on GitHub (with a certain label) and after a scan we end up with between 50 and 59 pipelines (which is too few).

          This probably happens for Bitbucket as well https://issues.jenkins.io/browse/JENKINS-55220

          Combined these issues have 45 votes, more than any other issue for GitHub Branch Source.

          Stian Jørgensrud added a comment - - edited We are also experiencing this! I didn't notice this happening when we had below 50 pipelines in our GitHub organization folder. Noticing it now that we have 61 repositories on GitHub (with a certain label) and after a scan we end up with between 50 and 59 pipelines (which is too few). This probably happens for Bitbucket as well https://issues.jenkins.io/browse/JENKINS-55220 Combined these issues have 45 votes, more than any other issue for GitHub Branch Source.

            Unassigned Unassigned
            jpschewe jpschewe
            Votes:
            22 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated: