Major I have a multibranch workflow project that has erased all of it's branch projects/jobs twice in the past week.
The symptoms are:
- MBW project erases all of its branches and their job history
- it immediately rescans the github repo
- then starts jobs for each branch it finds (which is exactly the same number of branch job/projects it deleted).
- the build numbers are all reset to 1
- all build directories for the branches have lost the previous builds (/var/lib/jenkins/jobs/THE-PROJECT/branches/MYBRANCH/builds/ only has 1/)
Here is a snap of the logs for the relevant time period:
Feb 19, 2016 1:38:52 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish ntsvc-nsapi-build/cpd #11 completed: FAILURE Feb 19, 2016 4:00:24 AM INFO org.jvnet.hudson.plugins.thinbackup.backup.HudsonBackup backupJobsDirectory Found 15 jobs to back up. Feb 19, 2016 4:01:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 7:48:07 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git Feb 19, 2016 7:59:48 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish ntsvc-nsapi-build/bunyan_syslog #1 completed: FAILURE Feb 19, 2016 7:59:49 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git Feb 19, 2016 8:01:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:02:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:03:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:04:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:05:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:06:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:07:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:08:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:09:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:10:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:10:24 AM INFO org.jvnet.hudson.plugins.thinbackup.utils.Utils waitUntilIdleAndSwitchToQuietMode Force quiet mode for jenkins now and wait unilt all executors are idle. Feb 19, 2016 8:11:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:12:23 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 19, 2016 8:12:57 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish ntsvc-nsapi-build/cpd #1 completed: SUCCESS Feb 19, 2016 8:12:58 AM INFO org.jvnet.hudson.plugins.thinbackup.backup.HudsonBackup backupJobsDirectory Found 15 jobs to back up. Feb 19, 2016 8:12:59 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git Feb 19, 2016 8:25:49 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish ntsvc-nsapi-build/master #1 completed: SUCCESS Feb 19, 2016 8:25:50 AM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add overriding old revision state hudson.scm.SCMRevisionState$None@5de21ef7 from git git@github.secureserver.net:NS/NS.git Feb 19, 2016 8:33:22 AM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish ntsvc-nsapi-build/ns-340 #1 completed: SUCCESS
At 7:48am the MBW project started the erase sequence listed above.... as one can see the first newly reindexed branch project/job (ntsvc-nsapi-build/bunyan_syslog) started at the same time and failed 11mins later at 7:59:48am.
Below is another occurrence of the problem:
Feb 22, 2016 8:20:55 AM INFO hudson.model.AsyncPeriodicWork$1 run Started Fingerprint cleanup Feb 22, 2016 8:20:55 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished Fingerprint cleanup. 1 ms Feb 22, 2016 1:36:09 PM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add overriding old revision state hudson.scm.SCMRevisionState$None@76212d09 from git git@github.secureserver.net:NS/NS.git Feb 22, 2016 1:54:36 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish ntsvc-nsapi-build/bunyan_syslog #1 completed: FAILURE Feb 22, 2016 1:54:37 PM WARNING org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState add overriding old revision state hudson.scm.SCMRevisionState$None@76212d09 from git git@github.secureserver.net:NS/NS.git Feb 22, 2016 1:56:19 PM INFO org.jenkinsci.plugins.workflow.job.WorkflowRun finish ntsvc-nsapi-build/ns-351 #1 completed: ABORTED
At 1:36pm the MBW project again started the erase sequence listed above.... as one can see the first newly reindexed branch project/job (ntsvc-nsapi-build/bunyan_syslog) started at the same time and failed 18mins later at 1:54:36pm.
Originally, I thought the ThinBackup plugin was related to the issue, as it showed up in the first log near the time of the job erasures, but the 2nd log shows that the issue can happen w/o the backup running.
- duplicates
-
JENKINS-34776 Jobs are removed if the remote is unavailable
-
- Resolved
-