Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12096

ThinBackup Worker Thread thread is still running. Execution aborted.

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • thinbackup-plugin
    • None
    • UNIX Solaris 10, Jenkins 1.415, ThinBackup 1.4

      We have ThinBackup set to run at 1am on Sunday mornings (i.e. Backup schedule for full backups = "0 1 * * 0"). The past run failed immediately with messages every 1 minute like the following:

      Dec 11, 2011 1:01:30 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.
      Dec 11, 2011 1:02:30 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.
      Dec 11, 2011 1:03:30 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.
      Dec 11, 2011 1:04:30 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
      WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.

      The problem continued until someone noticed later that day that regular build jobs were being blocked from running due to a "prepare for shutdown" block that ThinBackup triggers before running that never got canceled. The shutdown block was manually canceled to clear the job backlog.

      Our logs show no other entries or errors to identify the cause of the problem, or why a "ThinBackup Worker Thread thread" would be running, as if multiple ThinBackup workers were in progress and blocking each other.

      The first previous ThinBackup log entries were from a week before:

      Dec 4, 2011 11:41:10 PM org.jvnet.hudson.plugins.thinbackup.ThinBackupMgmtLink doBackupManual
      INFO: Starting manual backup.
      Dec 4, 2011 11:42:25 PM org.jvnet.hudson.plugins.thinbackup.backup.HudsonBackup backupJobs
      INFO: Found 79 jobs to back up.

      I had kicked off a manual backup, and I had seen it successfully complete in the UI, yet I see no log entry that indicates it did actually finished, either because the logging level for such messages is now FINEST, or because it never actually terminated the thread, resulting in the issue I'm reporting here.

      Either this is a problem with manual "Backup Now" runs, or it is a 1.4 issue. Note: I upgraded on Dec 4 from 1.2 to 1.4 in order to pickup a fix (suggested by JENKINS-9117) for backup failures never recovering and canceling the build block.

          [JENKINS-12096] ThinBackup Worker Thread thread is still running. Execution aborted.

          x86 nick added a comment -

          Same issue with
          Jenkins ver. 2.7.2

          ameter names, even though it represents a security breach
          Sep 16, 2016 9:10:15 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.
          Sep 16, 2016 9:11:15 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.

          x86 nick added a comment - Same issue with Jenkins ver. 2.7.2 ameter names, even though it represents a security breach Sep 16, 2016 9:10:15 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun WARNING: ThinBackup Worker Thread thread is still running. Execution aborted. Sep 16, 2016 9:11:15 AM org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun WARNING: ThinBackup Worker Thread thread is still running. Execution aborted.

          Dear all,

          I'm using Jenkins LTS 2.32.2 / Latest JDK V8 / Windows platform 2008 R2 64 bits
          ThinBackup plugin V1.9

          I have thoses entries in my logs during backup twice a day (diferencial backup)
          I have huge problem with master queue locking, I wonder if this could not be the root cause?

          Diff = H 12,22 * * 1-6
          Full = H 23 * * 7

          Tks,

          ./Frederic

          Frédéric Meyrou added a comment - Dear all, I'm using Jenkins LTS 2.32.2 / Latest JDK V8 / Windows platform 2008 R2 64 bits ThinBackup plugin V1.9 I have thoses entries in my logs during backup twice a day (diferencial backup) I have huge problem with master queue locking, I wonder if this could not be the root cause? Diff = H 12,22 * * 1-6 Full = H 23 * * 7 Tks, ./Frederic

          David Haws added a comment -

          I'm running into this issue as well.  It's causing a red "Jenkins is shutting down" banner at the top of the page.  Here's the error from the log file:
          WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.

          ThinBackup version 1.9.  How do I get the red banner to go away?

           

           

          David Haws added a comment - I'm running into this issue as well.  It's causing a red "Jenkins is shutting down" banner at the top of the page.  Here's the error from the log file: WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. ThinBackup version 1.9.  How do I get the red banner to go away?    

          Emil Wypych added a comment -

          Have the same issue. Backups are scheduled on every night and every backup generates the following logs:

          Feb 26, 2018 1:33:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:34:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:35:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:36:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:37:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:38:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          Feb 26, 2018 1:39:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun
          ThinBackup Worker Thread thread is still running. Execution aborted.
          

          Jenkins version: 2.107
          ThinBackup version: 1.9

          Scheduler:

          FULL: H(0-15) 1 * * 1

          DIFF: H(0-15) 1 * * 2-7

          Emil Wypych added a comment - Have the same issue. Backups are scheduled on every night and every backup generates the following logs: Feb 26, 2018 1:33:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:34:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:35:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:36:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:37:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:38:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Feb 26, 2018 1:39:13 AM WARNING org.jvnet.hudson.plugins.thinbackup.hudson.model.AsyncPeriodicWork doRun ThinBackup Worker Thread thread is still running. Execution aborted. Jenkins version: 2.107 ThinBackup version: 1.9 Scheduler: FULL: H(0-15) 1 * * 1 DIFF: H(0-15) 1 * * 2-7

          I'm watching this too on thinBackup 1.9, Jenkins 2.138.2

          The backup itself looks fine, but the warning messages are irritating.

          There is even no SUCCESS message for a finished backup.

          Torsten Werner added a comment - I'm watching this too on thinBackup 1.9, Jenkins 2.138.2 The backup itself looks fine, but the warning messages are irritating. There is even no SUCCESS message for a finished backup.

          Nicola Kahale added a comment -

          I'm running into this same issue and it's really causing a headache since it's causing Jenkin's to indefinitely be in a Quiet Down until manually cancelled.

           

          Jenkins Version: 2.164.3

          ThinBackup Version: 1.8

          Nicola Kahale added a comment - I'm running into this same issue and it's really causing a headache since it's causing Jenkin's to indefinitely be in a Quiet Down until manually cancelled.   Jenkins Version: 2.164.3 ThinBackup Version: 1.8

          Mark Slipets added a comment -

          Same issue bothering our setup.

          Jenkins Version: 2.222.3

          ThinBackup Version: 1.9

          Mark Slipets added a comment - Same issue bothering our setup. Jenkins Version: 2.222.3 ThinBackup Version: 1.9

          Roman Zwi added a comment -

          Same issue here - lots of entries like the following in /var/log/jenkins/*.log:

          2021-04-09 04:41:00.629+0000 [id=28] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted.
          2021-04-09 04:42:00.629+0000 [id=28] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted.
          2021-04-09 04:43:00.629+0000 [id=42] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted.

          I noticed that there was another job running that never finished. Can this be a problem?
          What will happen if ThinBackup triggers the Quiet Down but there is a running job that never finishes?

          Jenkins 2.284
          ThinBackup 1.10

          Roman Zwi added a comment - Same issue here - lots of entries like the following in /var/log/jenkins/*.log: 2021-04-09 04:41:00.629+0000 [id=28] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted. 2021-04-09 04:42:00.629+0000 [id=28] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted. 2021-04-09 04:43:00.629+0000 [id=42] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted. I noticed that there was another job running that never finished. Can this be a problem? What will happen if ThinBackup triggers the Quiet Down but there is a running job that never finishes? Jenkins 2.284 ThinBackup 1.10

          jo added a comment -

          I had the same issue today. It also seemed like a job was stuck (according to the log, it was finished yesterday, but it showed up as running in some UI parts). After I cancelled it, everything went back to normal, the backup finished, stuck jobs executed. (Jenkins.io/Jira threw away everything I typed twice already, so this time I'll make it rather short.)

          jo added a comment - I had the same issue today. It also seemed like a job was stuck (according to the log, it was finished yesterday, but it showed up as running in some UI parts). After I cancelled it, everything went back to normal, the backup finished, stuck jobs executed. (Jenkins.io/Jira threw away everything I typed twice already, so this time I'll make it rather short.)

          jo added a comment - - edited

          Some chronological events:

          First message of waiting backup:

          2021-10-05 20:01:26.495+0000 [id=37] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted.

          Jobs are still executing:

          2021-10-05 20:10:22.301+0000 [id=6571]  INFO    o.j.p.workflow.job.WorkflowRun#finish: Job-Alpha #84 completed: SUCCESS

          Waiting for idle:

          2021-10-05 22:00:26.598+0000 [id=6440]  INFO    o.j.h.p.thinbackup.utils.Utils#waitUntilIdleAndSwitchToQuietMode: Force quiet mode for jenkins now and wait until all executors are idle.

          No indication it ever happens.
          I kill the job, backup finishes, jobs get executed:

          2021-10-06 06:45:26.495+0000 [id=38]    WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted.
          2021-10-06 06:46:09.446+0000 [id=6440]  INFO    o.j.h.p.t.backup.HudsonBackup#backupJobsDirectory: Found 51 jobs to back up.
          2021-10-06 06:46:12.257+0000 [id=12293] INFO    o.j.h.p.thinbackup.utils.Utils#moveOldBackupsToZipFile: DONE moving 1 backup set(s) to ZIP files.
          2021-10-06 06:46:27.128+0000 [id=12279] INFO    o.j.p.workflow.job.WorkflowRun#finish: Otherjob-Alpha #75 completed: SUCCESS
          

          This might be interesting; it could have been the job that hung (it was definitely the Job name, I'm not sure if it was the run #36, but it is the only one showing something peculiar):

          2021-10-05 14:30:18.101+0000 [id=2804]  INFO    o.j.p.workflow.job.WorkflowRun#finish: Job-that-hung #36 completed: FAILURE
          2021-10-05 14:30:18.221+0000 [id=2804]  WARNING o.j.p.w.flow.FlowExecutionList#unregister: Owner[Job-that-hung/36:Job-that-hung #36] was not in the list to begin with: []

          The job log itself isn't timestamped, alas. It is somewhat peculiar:

          Lightweight checkout support not available, falling back to full checkout.
          Checking out hudson.scm.NullSCM into /daten/jenkins/workspace/Job-that-hung@script to read Jenkinsfile.gitlab
          ERROR: /daten/jenkins/workspace/Job-that-hung@script/Jenkinsfile.gitlab not found
          Finished: FAILURE
          

          The peculiar thing is, I don't see usually @script - Workspaces. Sometimes I have @tmp, @2, @2@tmp. Though it could well be I just don't usually see that part if the Jenkinsfile checks out OK.

          This is Jenkins 2.303.1 under Debian 10.

          jo added a comment - - edited Some chronological events: First message of waiting backup: 2021-10-05 20:01:26.495+0000 [id=37] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted. Jobs are still executing: 2021-10-05 20:10:22.301+0000 [id=6571] INFO o.j.p.workflow.job.WorkflowRun#finish: Job-Alpha #84 completed: SUCCESS Waiting for idle: 2021-10-05 22:00:26.598+0000 [id=6440] INFO o.j.h.p.thinbackup.utils.Utils#waitUntilIdleAndSwitchToQuietMode: Force quiet mode for jenkins now and wait until all executors are idle. No indication it ever happens. I kill the job, backup finishes, jobs get executed: 2021-10-06 06:45:26.495+0000 [id=38] WARNING o.j.h.p.t.h.m.AsyncPeriodicWork#doRun: ThinBackup Worker Thread thread is still running. Execution aborted. 2021-10-06 06:46:09.446+0000 [id=6440] INFO o.j.h.p.t.backup.HudsonBackup#backupJobsDirectory: Found 51 jobs to back up. 2021-10-06 06:46:12.257+0000 [id=12293] INFO o.j.h.p.thinbackup.utils.Utils#moveOldBackupsToZipFile: DONE moving 1 backup set(s) to ZIP files. 2021-10-06 06:46:27.128+0000 [id=12279] INFO o.j.p.workflow.job.WorkflowRun#finish: Otherjob-Alpha #75 completed: SUCCESS This might be interesting; it could have been the job that hung (it was definitely the Job name, I'm not sure if it was the run #36, but it is the only one showing something peculiar): 2021-10-05 14:30:18.101+0000 [id=2804] INFO o.j.p.workflow.job.WorkflowRun#finish: Job-that-hung #36 completed: FAILURE 2021-10-05 14:30:18.221+0000 [id=2804] WARNING o.j.p.w.flow.FlowExecutionList#unregister: Owner[Job-that-hung/36:Job-that-hung #36] was not in the list to begin with: [] The job log itself isn't timestamped, alas. It is somewhat peculiar: Lightweight checkout support not available, falling back to full checkout. Checking out hudson.scm.NullSCM into /daten/jenkins/workspace/Job-that-hung@script to read Jenkinsfile.gitlab ERROR: /daten/jenkins/workspace/Job-that-hung@script/Jenkinsfile.gitlab not found Finished: FAILURE The peculiar thing is, I don't see usually @script - Workspaces. Sometimes I have @tmp, @2, @2@tmp. Though it could well be I just don't usually see that part if the Jenkinsfile checks out OK. This is Jenkins 2.303.1 under Debian 10.

            tofuatjava Thomas Fürer
            dankirkd Daniel Kirkdorffer
            Votes:
            19 Vote for this issue
            Watchers:
            29 Start watching this issue

              Created:
              Updated: