Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33045

Branch Project/Jobs get erased

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Duplicate
    • pipeline
    • Jenkins ver. 1.644
      All Workflow/Pipeline plugins at 1.12
      ThinBackup 1.7.4
      See attachment for Jenkins SystemInfo

    Description

      I have a multibranch workflow project that has erased all of it's branch projects/jobs twice in the past week.
      The symptoms are:

      • MBW project erases all of its branches and their job history
      • it immediately rescans the github repo
      • then starts jobs for each branch it finds (which is exactly the same number of branch job/projects it deleted).
      • the build numbers are all reset to 1
      • all build directories for the branches have lost the previous builds (/var/lib/jenkins/jobs/THE-PROJECT/branches/MYBRANCH/builds/ only has 1/)

      Here is a snap of the logs for the relevant time period:

      Feb 19, 2016 1:38:52 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      ntsvc-nsapi-build/cpd #11 completed: FAILURE
      Feb 19, 2016 4:00:24 AM INFO org.jvnet.hudson.plugins.thinbackup.backup.HudsonBackup backupJobsDirectory
      Found 15 jobs to back up.
      Feb 19, 2016 4:01:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 7:48:07 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
      overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
      Feb 19, 2016 7:59:48 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      ntsvc-nsapi-build/bunyan_syslog #1 completed: FAILURE
      Feb 19, 2016 7:59:49 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
      overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
      Feb 19, 2016 8:01:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:02:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:03:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:04:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:05:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:06:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:07:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:08:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:09:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:10:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:10:24 AM INFO org.jvnet.hudson.plugins.thinbackup.utils.Utils waitUntilIdleAndSwitchToQuietMode
      Force quiet mode for jenkins now and wait unilt all executors are idle.
      Feb 19, 2016 8:11:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:12:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      ThinBackup Worker Thread thread is still running. Execution aborted.
      Feb 19, 2016 8:12:57 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      ntsvc-nsapi-build/cpd #1 completed: SUCCESS
      Feb 19, 2016 8:12:58 AM INFO org.jvnet.hudson.plugins.thinbackup.backup.HudsonBackup backupJobsDirectory
      Found 15 jobs to back up.
      Feb 19, 2016 8:12:59 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
      overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
      Feb 19, 2016 8:25:49 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      ntsvc-nsapi-build/master #1 completed: SUCCESS
      Feb 19, 2016 8:25:50 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
      overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
      Feb 19, 2016 8:33:22 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      ntsvc-nsapi-build/ns-340 #1 completed: SUCCESS
      

      At 7:48am the MBW project started the erase sequence listed above.... as one can see the first newly reindexed branch project/job (ntsvc-nsapi-build/bunyan_syslog) started at the same time and failed 11mins later at 7:59:48am.

      Below is another occurrence of the problem:

      Feb 22, 2016 8:20:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
      Started Fingerprint cleanup
      Feb 22, 2016 8:20:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
      Finished Fingerprint cleanup. 1 ms
      Feb 22, 2016 1:36:09 PM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
      overriding old revision state hudson.scm.SCMRevisionState$None@76212d09 from git git@github.secureserver.net:NS/NS.git
      Feb 22, 2016 1:54:36 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      ntsvc-nsapi-build/bunyan_syslog #1 completed: FAILURE
      Feb 22, 2016 1:54:37 PM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
      overriding old revision state hudson.scm.SCMRevisionState$None@76212d09 from git git@github.secureserver.net:NS/NS.git
      Feb 22, 2016 1:56:19 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
      ntsvc-nsapi-build/ns-351 #1 completed: ABORTED
      

      At 1:36pm the MBW project again started the erase sequence listed above.... as one can see the first newly reindexed branch project/job (ntsvc-nsapi-build/bunyan_syslog) started at the same time and failed 18mins later at 1:54:36pm.

      Originally, I thought the ThinBackup plugin was related to the issue, as it showed up in the first log near the time of the job erasures, but the 2nd log shows that the issue can happen w/o the backup running.

      Attachments

        Issue Links

          Activity

            potatopankakes Jeff Burke created issue -
            potatopankakes Jeff Burke made changes -
            Field Original Value New Value
            Description I have a multibranch workflow project that has erased all of it's branch projects/jobs twice in the past week.
            The symptoms are:
            - MBW project erases all of its branches and their job history
            - it immediately rescans the github repo
            - then starts jobs for each branch it finds (which is exactly the same number of branch job/projects it deleted).
            - the build numbers are all reset to 1
            - all build directories for the branches have lost the previous builds (/var/lib/jenkins/jobs/THE-PROJECT/branches/MYBRANCH/builds/ only has 1/)

            It is possible that this is being caused by an interaction w/the ThinBackup plugin that I am running. It kicks off at various times throughout the day, and on this last incident, there was a time overlap. Here is a snap of the logs for the relevant time period:
            {noformat}
            Feb 19, 2016 1:38:52 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/cpd #11 completed: FAILURE
            Feb 19, 2016 4:00:24 AM INFO org.jvnet.hudson.plugins.thinbackup.backup.HudsonBackup backupJobsDirectory
            Found 15 jobs to back up.
            Feb 19, 2016 4:01:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 7:48:07 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
            overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
            Feb 19, 2016 7:59:48 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/bunyan_syslog #1 completed: FAILURE
            Feb 19, 2016 7:59:49 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
            overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
            Feb 19, 2016 8:01:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:02:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:03:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:04:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:05:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:06:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:07:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:08:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:09:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:10:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:10:24 AM INFO org.jvnet.hudson.plugins.thinbackup.utils.Utils waitUntilIdleAndSwitchToQuietMode
            Force quiet mode for jenkins now and wait unilt all executors are idle.
            Feb 19, 2016 8:11:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:12:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:12:57 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/cpd #1 completed: SUCCESS
            Feb 19, 2016 8:12:58 AM INFO org.jvnet.hudson.plugins.thinbackup.backup.HudsonBackup backupJobsDirectory
            Found 15 jobs to back up.
            Feb 19, 2016 8:12:59 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
            overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
            Feb 19, 2016 8:25:49 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/master #1 completed: SUCCESS
            Feb 19, 2016 8:25:50 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
            overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
            Feb 19, 2016 8:33:22 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/ns-340 #1 completed: SUCCESS
            {noformat}

            The log shows a full backup started at/near 4am, and then a differential was started at/near 8am. At 7:48am the MBW project started the erase sequence listed above.... as one can see the first newly reindexed branch project/job (ntsvc-nsapi-build/bunyan_syslog) started at the same time and failed 11mins later at 7:59:48am.

            The ThinBackup cron under the Manage Jenkins menus is:
            Full: H 4 * * *
            Differential: H 8,10,12,14,16,18,20 * * 1-5

            I have currently changed the cron to avoid backups during 'active hours'.
            I have a multibranch workflow project that has erased all of it's branch projects/jobs twice in the past week.
            The symptoms are:
            - MBW project erases all of its branches and their job history
            - it immediately rescans the github repo
            - then starts jobs for each branch it finds (which is exactly the same number of branch job/projects it deleted).
            - the build numbers are all reset to 1
            - all build directories for the branches have lost the previous builds (/var/lib/jenkins/jobs/THE-PROJECT/branches/MYBRANCH/builds/ only has 1/)

            Here is a snap of the logs for the relevant time period:
            {noformat}
            Feb 19, 2016 1:38:52 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/cpd #11 completed: FAILURE
            Feb 19, 2016 4:00:24 AM INFO org.jvnet.hudson.plugins.thinbackup.backup.HudsonBackup backupJobsDirectory
            Found 15 jobs to back up.
            Feb 19, 2016 4:01:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 7:48:07 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
            overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
            Feb 19, 2016 7:59:48 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/bunyan_syslog #1 completed: FAILURE
            Feb 19, 2016 7:59:49 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
            overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
            Feb 19, 2016 8:01:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:02:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:03:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:04:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:05:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:06:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:07:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:08:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:09:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:10:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:10:24 AM INFO org.jvnet.hudson.plugins.thinbackup.utils.Utils waitUntilIdleAndSwitchToQuietMode
            Force quiet mode for jenkins now and wait unilt all executors are idle.
            Feb 19, 2016 8:11:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:12:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
            ThinBackup Worker Thread thread is still running. Execution aborted.
            Feb 19, 2016 8:12:57 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/cpd #1 completed: SUCCESS
            Feb 19, 2016 8:12:58 AM INFO org.jvnet.hudson.plugins.thinbackup.backup.HudsonBackup backupJobsDirectory
            Found 15 jobs to back up.
            Feb 19, 2016 8:12:59 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
            overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
            Feb 19, 2016 8:25:49 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/master #1 completed: SUCCESS
            Feb 19, 2016 8:25:50 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
            overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git
            Feb 19, 2016 8:33:22 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/ns-340 #1 completed: SUCCESS
            {noformat}

            At 7:48am the MBW project started the erase sequence listed above.... as one can see the first newly reindexed branch project/job (ntsvc-nsapi-build/bunyan_syslog) started at the same time and failed 11mins later at 7:59:48am.

            Below is another occurrence of the problem:
            {noformat}
            Feb 22, 2016 8:20:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
            Started Fingerprint cleanup
            Feb 22, 2016 8:20:55 AM INFO hudson.model.AsyncPeriodicWork$1 run
            Finished Fingerprint cleanup. 1 ms
            Feb 22, 2016 1:36:09 PM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
            overriding old revision state hudson.scm.SCMRevisionState$None@76212d09 from git git@github.secureserver.net:NS/NS.git
            Feb 22, 2016 1:54:36 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/bunyan_syslog #1 completed: FAILURE
            Feb 22, 2016 1:54:37 PM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add
            overriding old revision state hudson.scm.SCMRevisionState$None@76212d09 from git git@github.secureserver.net:NS/NS.git
            Feb 22, 2016 1:56:19 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish
            ntsvc-nsapi-build/ns-351 #1 completed: ABORTED
            {noformat}

            At 1:36pm the MBW project again started the erase sequence listed above.... as one can see the first newly reindexed branch project/job (ntsvc-nsapi-build/bunyan_syslog) started at the same time and failed 18mins later at 1:54:36pm.

            Originally, I thought the ThinBackup plugin was related to the issue, as it showed up in the first log near the time of the job erasures, but the 2nd log shows that the issue can happen w/o the backup running.

            jglick Jesse Glick made changes -
            Link This issue duplicates JENKINS-34776 [ JENKINS-34776 ]
            jglick Jesse Glick made changes -
            Resolution Duplicate [ 3 ]
            Status Open [ 1 ] Resolved [ 5 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 168874 ] JNJira + In-Review [ 198488 ]
            abayer Andrew Bayer made changes -
            Component/s pipeline-general [ 21692 ]
            abayer Andrew Bayer made changes -
            Component/s workflow-plugin [ 18820 ]
            abayer Andrew Bayer made changes -
            Labels workflow pipeline workflow
            abayer Andrew Bayer made changes -
            Labels pipeline workflow pipeline

            People

              jglick Jesse Glick
              potatopankakes Jeff Burke
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: