Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12096

ThinBackup Worker Thread thread is still running. Execution aborted.

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • thinbackup-plugin
    • None
    • UNIX Solaris 10, Jenkins 1.415, ThinBackup 1.4

      We have ThinBackup set to run at 1am on Sunday mornings (i.e. Backup schedule for full backups = "0 1 * * 0"). The past run failed immediately with messages every 1 minute like the following:

      Dec 11, 2011 1:01:30 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.
      Dec 11, 2011 1:02:30 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.
      Dec 11, 2011 1:03:30 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.
      Dec 11, 2011 1:04:30 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.

      The problem continued until someone noticed later that day that regular build jobs were being blocked from running due to a "prepare for shutdown" block that ThinBackup triggers before running that never got canceled. The shutdown block was manually canceled to clear the job backlog.

      Our logs show no other entries or errors to identify the cause of the problem, or why a "ThinBackup Worker Thread thread" would be running, as if multiple ThinBackup workers were in progress and blocking each other.

      The first previous ThinBackup log entries were from a week before:

      Dec 4, 2011 11:41:10 PM org.jvnet.hudson.plugins.thinbackup.ThinBackupMgmtLink doBackupManual
      INFO: Starting manual backup.
      Dec 4, 2011 11:42:25 PM org.jvnet.hudson.plugins.thinbackup.backup.HudsonBackup backupJobs
      INFO: Found 79 jobs to back up.

      I had kicked off a manual backup, and I had seen it successfully complete in the UI, yet I see no log entry that indicates it did actually finished, either because the logging level for such messages is now FINEST, or because it never actually terminated the thread, resulting in the issue I'm reporting here.

      Either this is a problem with manual "Backup Now" runs, or it is a 1.4 issue. Note: I upgraded on Dec 4 from 1.2 to 1.4 in order to pickup a fix (suggested by JENKINS-9117) for backup failures never recovering and canceling the build block.

          [JENKINS-12096] ThinBackup Worker Thread thread is still running. Execution aborted.

          After writing this defect I discovered more about the issue and feel I need to write a different defect or update JENKINS-9117 because I have a scenario that appears to indicate that ThinBackup hit an issue during backup and didn't cancel shutdown mode correctly.

          Daniel Kirkdorffer added a comment - After writing this defect I discovered more about the issue and feel I need to write a different defect or update JENKINS-9117 because I have a scenario that appears to indicate that ThinBackup hit an issue during backup and didn't cancel shutdown mode correctly.

          Reopening because there does still appear something that blocked this from running as it should have. Other things I have been able to see should not be related to the problem I've written about.

          Daniel Kirkdorffer added a comment - Reopening because there does still appear something that blocked this from running as it should have. Other things I have been able to see should not be related to the problem I've written about.

          Thomas Fürer added a comment -

          cannot reproduce this issue.
          Only if the backup is already/still running and a new backup will be trigger this happens. But I don't think that some have such a tight schedule. The other thing is that something on the filesystem could block the backup. In this case you should watch out for permission settings or parallel file operations.

          in all this cases I cannot handle the issue in the backup plugin, so I decided to close this one.

          Thomas Fürer added a comment - cannot reproduce this issue. Only if the backup is already/still running and a new backup will be trigger this happens. But I don't think that some have such a tight schedule. The other thing is that something on the filesystem could block the backup. In this case you should watch out for permission settings or parallel file operations. in all this cases I cannot handle the issue in the backup plugin, so I decided to close this one.

          Karthic Bala added a comment -

          Now i am getting same warning message in our jenkins logs

          Jenkins - Upgraded from v1.565 to 1.606
          Thinbackup : 1.7.4

          Logs:
          Jul 06, 2015 2:31:48 PM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.
          Jul 06, 2015 2:32:00 PM hudson.plugins.accurev.FindAccurevClientExe invoke
          INFO: Path to the AccuRev client: /usr/bin/accurev
          Jul 06, 2015 2:32:48 PM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.
          Jul 06, 2015 2:33:48 PM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.

          Karthic Bala added a comment - Now i am getting same warning message in our jenkins logs Jenkins - Upgraded from v1.565 to 1.606 Thinbackup : 1.7.4 Logs: Jul 06, 2015 2:31:48 PM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun WARNING: ThinBackup Worker Thread thread is still running. Execution aborted. Jul 06, 2015 2:32:00 PM hudson.plugins.accurev.FindAccurevClientExe invoke INFO: Path to the AccuRev client: /usr/bin/accurev Jul 06, 2015 2:32:48 PM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun WARNING: ThinBackup Worker Thread thread is still running. Execution aborted. Jul 06, 2015 2:33:48 PM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.

          Karthic Bala added a comment -

          Seeing this issue again..

          Karthic Bala added a comment - Seeing this issue again..

          Swapnil Kotwal added a comment - - edited

          I have restarted Jenkins but still "Creating backup..." thread not yet stopped.

          Swapnil Kotwal added a comment - - edited I have restarted Jenkins but still "Creating backup..." thread not yet stopped.

          Liya Katz added a comment -

          Experience this issue as well.
          Jenkins 1.643
          ThinBackup 1.7.4

          Liya Katz added a comment - Experience this issue as well. Jenkins 1.643 ThinBackup 1.7.4

          x86 nick added a comment -

          Same issue with
          Jenkins ver. 2.7.2

          ameter names, even though it represents a security breach
          Sep 16, 2016 9:10:15 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.
          Sep 16, 2016 9:11:15 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.

          x86 nick added a comment - Same issue with Jenkins ver. 2.7.2 ameter names, even though it represents a security breach Sep 16, 2016 9:10:15 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun WARNING: ThinBackup Worker Thread thread is still running. Execution aborted. Sep 16, 2016 9:11:15 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.

          Dear all,

          I'm using Jenkins LTS 2.32.2 / Latest JDK V8 / Windows platform 2008 R2 64 bits
          ThinBackup plugin V1.9

          I have thoses entries in my logs during backup twice a day (diferencial backup)
          I have huge problem with master queue locking, I wonder if this could not be the root cause?

          Diff = H 12,22 * * 1-6
          Full = H 23 * * 7

          Tks,

          ./Frederic

          Frédéric Meyrou added a comment - Dear all, I'm using Jenkins LTS 2.32.2 / Latest JDK V8 / Windows platform 2008 R2 64 bits ThinBackup plugin V1.9 I have thoses entries in my logs during backup twice a day (diferencial backup) I have huge problem with master queue locking, I wonder if this could not be the root cause? Diff = H 12,22 * * 1-6 Full = H 23 * * 7 Tks, ./Frederic

          David Haws added a comment -

          I'm running into this issue as well.  It's causing a red "Jenkins is shutting down" banner at the top of the page.  Here's the error from the log file:
          WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.

          ThinBackup version 1.9.  How do I get the red banner to go away?

           

           

          David Haws added a comment - I'm running into this issue as well.  It's causing a red "Jenkins is shutting down" banner at the top of the page.  Here's the error from the log file: WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. ThinBackup version 1.9.  How do I get the red banner to go away?    

          Emil Wypych added a comment -

          Have the same issue. Backups are scheduled on every night and every backup generates the following logs:

          Feb 26, 2018 1:33:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:34:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:35:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:36:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:37:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:38:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:39:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          

          Jenkins version: 2.107
          ThinBackup version: 1.9

          Scheduler:

          FULL: H(0-15) 1 * * 1

          DIFF: H(0-15) 1 * * 2-7

          Emil Wypych added a comment - Have the same issue. Backups are scheduled on every night and every backup generates the following logs: Feb 26, 2018 1:33:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:34:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:35:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:36:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:37:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:38:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:39:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Jenkins version: 2.107 ThinBackup version: 1.9 Scheduler: FULL: H(0-15) 1 * * 1 DIFF: H(0-15) 1 * * 2-7

          I'm watching this too on thinBackup 1.9, Jenkins 2.138.2

          The backup itself looks fine, but the warning messages are irritating.

          There is even no SUCCESS message for a finished backup.

          Torsten Werner added a comment - I'm watching this too on thinBackup 1.9, Jenkins 2.138.2 The backup itself looks fine, but the warning messages are irritating. There is even no SUCCESS message for a finished backup.

          Nicola Kahale added a comment -

          I'm running into this same issue and it's really causing a headache since it's causing Jenkin's to indefinitely be in a Quiet Down until manually cancelled.

           

          Jenkins Version: 2.164.3

          ThinBackup Version: 1.8

          Nicola Kahale added a comment - I'm running into this same issue and it's really causing a headache since it's causing Jenkin's to indefinitely be in a Quiet Down until manually cancelled.   Jenkins Version: 2.164.3 ThinBackup Version: 1.8

          Mark Slipets added a comment -

          Same issue bothering our setup.

          Jenkins Version: 2.222.3

          ThinBackup Version: 1.9

          Mark Slipets added a comment - Same issue bothering our setup. Jenkins Version: 2.222.3 ThinBackup Version: 1.9

          Roman Zwi added a comment -

          Same issue here - lots of entries like the following in /var/log/jenkins/*.log:

          2021-04-09 04:41:00.629+0000 [id=28] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted.
          2021-04-09 04:42:00.629+0000 [id=28] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted.
          2021-04-09 04:43:00.629+0000 [id=42] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted.

          I noticed that there was another job running that never finished. Can this be a problem?
          What will happen if ThinBackup triggers the Quiet Down but there is a running job that never finishes?

          Jenkins 2.284
          ThinBackup 1.10

          Roman Zwi added a comment - Same issue here - lots of entries like the following in /var/log/jenkins/*.log: 2021-04-09 04:41:00.629+0000 [id=28] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted. 2021-04-09 04:42:00.629+0000 [id=28] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted. 2021-04-09 04:43:00.629+0000 [id=42] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted. I noticed that there was another job running that never finished. Can this be a problem? What will happen if ThinBackup triggers the Quiet Down but there is a running job that never finishes? Jenkins 2.284 ThinBackup 1.10

          jo added a comment -

          I had the same issue today. It also seemed like a job was stuck (according to the log, it was finished yesterday, but it showed up as running in some UI parts). After I cancelled it, everything went back to normal, the backup finished, stuck jobs executed. (Jenkins.io/Jira threw away everything I typed twice already, so this time I'll make it rather short.)

          jo added a comment - I had the same issue today. It also seemed like a job was stuck (according to the log, it was finished yesterday, but it showed up as running in some UI parts). After I cancelled it, everything went back to normal, the backup finished, stuck jobs executed. (Jenkins.io/Jira threw away everything I typed twice already, so this time I'll make it rather short.)

          jo added a comment - - edited

          Some chronological events:

          First message of waiting backup:

          2021-10-05 20:01:26.495+0000 [id=37] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted.

          Jobs are still executing:

          2021-10-05 20:10:22.301+0000 [id=6571]  INFO    o.j.p.workflow.job.WorkflowRun#finish: Job-Alpha #84 completed: SUCCESS

          Waiting for idle:

          2021-10-05 22:00:26.598+0000 [id=6440]  INFO    o.j.h.p.thinbackup.utils.Utils#waitUntilIdleAndSwitchToQuietMode: Force quiet mode for jenkins now and wait until all executors are idle.

          No indication it ever happens.
          I kill the job, backup finishes, jobs get executed:

          2021-10-06 06:45:26.495+0000 [id=38]    WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted.
          2021-10-06 06:46:09.446+0000 [id=6440]  INFO    o.j.h.p.t.backup.HudsonBackup#backupJobsDirectory: Found 51 jobs to back up.
          2021-10-06 06:46:12.257+0000 [id=12293] INFO    o.j.h.p.thinbackup.utils.Utils#moveOldBackupsToZipFile: DONE moving 1 backup set(s) to ZIP files.
          2021-10-06 06:46:27.128+0000 [id=12279] INFO    o.j.p.workflow.job.WorkflowRun#finish: Otherjob-Alpha #75 completed: SUCCESS
          

          This might be interesting; it could have been the job that hung (it was definitely the Job name, I'm not sure if it was the run #36, but it is the only one showing something peculiar):

          2021-10-05 14:30:18.101+0000 [id=2804]  INFO    o.j.p.workflow.job.WorkflowRun#finish: Job-that-hung #36 completed: FAILURE
          2021-10-05 14:30:18.221+0000 [id=2804]  WARNING o.j.p.w.flow.FlowExecutionList#unregister: Owner[Job-that-hung/36:Job-that-hung #36] was not in the list to begin with: []

          The job log itself isn't timestamped, alas. It is somewhat peculiar:

          Lightweight checkout support not available, falling back to full checkout.
          Checking out hudson.scm.NullSCM into /daten/jenkins/workspace/Job-that-hung@script to read Jenkinsfile.gitlab
          ERROR: /daten/jenkins/workspace/Job-that-hung@script/Jenkinsfile.gitlab not found
          Finished: FAILURE
          

          The peculiar thing is, I don't see usually @script - Workspaces. Sometimes I have @tmp, @2, @2@tmp. Though it could well be I just don't usually see that part if the Jenkinsfile checks out OK.

          This is Jenkins 2.303.1 under Debian 10.

          jo added a comment - - edited Some chronological events: First message of waiting backup: 2021-10-05 20:01:26.495+0000 [id=37] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted. Jobs are still executing: 2021-10-05 20:10:22.301+0000 [id=6571] INFO o.j.p.workflow.job.WorkflowRun#finish: Job-Alpha #84 completed: SUCCESS Waiting for idle: 2021-10-05 22:00:26.598+0000 [id=6440] INFO o.j.h.p.thinbackup.utils.Utils#waitUntilIdleAndSwitchToQuietMode: Force quiet mode for jenkins now and wait until all executors are idle. No indication it ever happens. I kill the job, backup finishes, jobs get executed: 2021-10-06 06:45:26.495+0000 [id=38] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted. 2021-10-06 06:46:09.446+0000 [id=6440] INFO o.j.h.p.t.backup.HudsonBackup#backupJobsDirectory: Found 51 jobs to back up. 2021-10-06 06:46:12.257+0000 [id=12293] INFO o.j.h.p.thinbackup.utils.Utils#moveOldBackupsToZipFile: DONE moving 1 backup set(s) to ZIP files. 2021-10-06 06:46:27.128+0000 [id=12279] INFO o.j.p.workflow.job.WorkflowRun#finish: Otherjob-Alpha #75 completed: SUCCESS This might be interesting; it could have been the job that hung (it was definitely the Job name, I'm not sure if it was the run #36, but it is the only one showing something peculiar): 2021-10-05 14:30:18.101+0000 [id=2804] INFO o.j.p.workflow.job.WorkflowRun#finish: Job-that-hung #36 completed: FAILURE 2021-10-05 14:30:18.221+0000 [id=2804] WARNING o.j.p.w.flow.FlowExecutionList#unregister: Owner[Job-that-hung/36:Job-that-hung #36] was not in the list to begin with: [] The job log itself isn't timestamped, alas. It is somewhat peculiar: Lightweight checkout support not available, falling back to full checkout. Checking out hudson.scm.NullSCM into /daten/jenkins/workspace/Job-that-hung@script to read Jenkinsfile.gitlab ERROR: /daten/jenkins/workspace/Job-that-hung@script/Jenkinsfile.gitlab not found Finished: FAILURE The peculiar thing is, I don't see usually @script - Workspaces. Sometimes I have @tmp, @2, @2@tmp. Though it could well be I just don't usually see that part if the Jenkinsfile checks out OK. This is Jenkins 2.303.1 under Debian 10.

            tofuatjava Thomas Fürer
            dankirkd Daniel Kirkdorffer
            Votes:
            19 Vote for this issue
            Watchers:
            29 Start watching this issue

              Created:
              Updated: