Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61687

Exceptions due to race condition during build deletion

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      jenkins-2.253

      Description

      Messages like the following are being logged on ci.jenkins.io after updating to 2.222.1.

      2020-03-25 19:18:34.958+0000 [id=21774]	WARNING	j.m.BackgroundGlobalBuildDiscarder#lambda$processJob$0: An exception occurred when executing Project Build Discarder
      Also:   java.nio.file.NoSuchFileException: /var/jenkins_home/jobs/Infra/jobs/plugin-site-api/branches/generate-data/builds/93295 -> /var/jenkins_home/jobs/Infra/jobs/plugin-site-api/branches/generate-data/builds/.93295
      		at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
      		at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
      		at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:417)
      		at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:267)
      		at java.base/java.nio.file.Files.move(Files.java:1421)
      		at hudson.model.Run.delete(Run.java:1621)
      		at hudson.tasks.LogRotator.perform(LogRotator.java:166)
      jenkins.util.io.CompositeIOException: Failed to rotate logs for [Infra/plugin-site-api/generate-data #93295]
      	at hudson.tasks.LogRotator.perform(LogRotator.java:223)
      	at hudson.model.Job.logRotate(Job.java:469)
      	at jenkins.model.JobGlobalBuildDiscarderStrategy.apply(JobGlobalBuildDiscarderStrategy.java:54)
      	at jenkins.model.BackgroundGlobalBuildDiscarder.lambda$processJob$0(BackgroundGlobalBuildDiscarder.java:67)
      	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
      	at jenkins.model.BackgroundGlobalBuildDiscarder.processJob(BackgroundGlobalBuildDiscarder.java:61)
      	at jenkins.model.GlobalBuildDiscarderListener.onFinalized(GlobalBuildDiscarderListener.java:49)
      	at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255)
      	at hudson.model.Run.onEndBuilding(Run.java:2018)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:617)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1018)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:488)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:834) 

      or

      2020-03-25 19:28:42.053+0000 [id=56]	WARNING	o.j.p.workflow.job.WorkflowRun#lambda$finish$2: failed to perform log rotation after Infra/plugin-site-api/generate-data #93303
      Also:   java.nio.file.NoSuchFileException: /var/jenkins_home/jobs/Infra/jobs/plugin-site-api/branches/generate-data/builds/93296 -> /var/jenkins_home/jobs/Infra/jobs/plugin-site-api/branches/generate-data/builds/.93296
      		at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
      		at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
      		at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:417)
      		at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:267)
      		at java.base/java.nio.file.Files.move(Files.java:1421)
      		at hudson.model.Run.delete(Run.java:1621)
      		at hudson.tasks.LogRotator.perform(LogRotator.java:166)
      jenkins.util.io.CompositeIOException: Failed to rotate logs for [Infra/plugin-site-api/generate-data #93296]
      	at hudson.tasks.LogRotator.perform(LogRotator.java:223)
      	at hudson.model.Job.logRotate(Job.java:469)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.lambda$finish$2(WorkflowRun.java:612)
      	at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:834) 

      None of them occurred while background deletion was ongoing.

        Attachments

          Issue Links

            Activity

            Hide
            gmshake Zhenlei Huang added a comment -

            Local test looks good with the fix

            Show
            gmshake Zhenlei Huang added a comment - Local test looks good with the fix
            Hide
            allan_burdajewicz Allan BURDAJEWICZ added a comment -

            Daniel Beck By the way, isn't the "JobGlobalBuildDiscarderStrategy" supposed to run periodically ? According to the documentation:

            Build discarders configured for a job are only run after a build finishes. This option runs jobs' configured build discarders periodically, applying configuration changes even when no new builds are run. This option has no effect if there is no build discarder configured for a job.

            Show
            allan_burdajewicz Allan BURDAJEWICZ added a comment - Daniel Beck By the way, isn't the "JobGlobalBuildDiscarderStrategy" supposed to run periodically ? According to the documentation: Build discarders configured for a job are only run after a build finishes. This option runs jobs' configured build discarders periodically , applying configuration changes even when no new builds are run. This option has no effect if there is no build discarder configured for a job.
            Hide
            danielbeck Daniel Beck added a comment -

            Allan BURDAJEWICZ

            It does; and additionally it runs global build discarders once on a project when a build finishes. I thought it makes no sense to wait up to an hour (IIRC) to delete builds. I think the result is much nicer this way (except for exposing this bug in many situations, of course). The documentation probably just didn't keep up with the evolution of the feature and could be improved.

            Show
            danielbeck Daniel Beck added a comment - Allan BURDAJEWICZ It does; and additionally it runs global build discarders once on a project when a build finishes. I thought it makes no sense to wait up to an hour (IIRC) to delete builds. I think the result is much nicer this way (except for exposing this bug in many situations, of course). The documentation probably just didn't keep up with the evolution of the feature and could be improved.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Created JENKINS-63275 as a follow-up discussed in the PR

            Show
            oleg_nenashev Oleg Nenashev added a comment - Created  JENKINS-63275 as a follow-up discussed in the PR
            Hide
            antonio_petricca Antonio Petricca added a comment -

            Same problem for me:

            2020-08-26 07:26:05.412+0000 [id=511] WARNING j.m.BackgroundGlobalBuildDiscarder#lambda$processJob$0: An exception occurred when executing Project Build Discarder
            
            2020-08-26 07:26:05.412+0000 [id=511] WARNING j.m.BackgroundGlobalBuildDiscarder#lambda$processJob$0: An exception occurred when executing Project Build DiscarderAlso:   java.nio.file.NoSuchFileException: /var/jenkins_home/jobs/simp/branches/tasks-BET-31993.du2529/builds/8 -> /var/jenkins_home/jobs/simp/branches/tasks-BET-31993.du2529/builds/.8 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at hudson.model.Run.delete(Run.java:1645) at hudson.tasks.LogRotator.perform(LogRotator.java:166)jenkins.util.io.CompositeIOException: Failed to rotate logs for [simp/tasks%2FBET-31993 #8] at hudson.tasks.LogRotator.perform(LogRotator.java:223) at hudson.model.Job.logRotate(Job.java:469) at jenkins.model.JobGlobalBuildDiscarderStrategy.apply(JobGlobalBuildDiscarderStrategy.java:54) at jenkins.model.BackgroundGlobalBuildDiscarder.lambda$processJob$0(BackgroundGlobalBuildDiscarder.java:67) at java.lang.Iterable.forEach(Iterable.java:75) at jenkins.model.BackgroundGlobalBuildDiscarder.processJob(BackgroundGlobalBuildDiscarder.java:61) at jenkins.model.GlobalBuildDiscarderListener.onFinalized(GlobalBuildDiscarderListener.java:49) at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255) at hudson.model.Run.onEndBuilding(Run.java:2042) at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:625) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137) at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1026) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
            

             

            Show
            antonio_petricca Antonio Petricca added a comment - Same problem for me: 2020-08-26 07:26:05.412+0000 [id=511] WARNING j.m.BackgroundGlobalBuildDiscarder#lambda$processJob$0: An exception occurred when executing Project Build Discarder 2020-08-26 07:26:05.412+0000 [id=511] WARNING j.m.BackgroundGlobalBuildDiscarder#lambda$processJob$0: An exception occurred when executing Project Build DiscarderAlso:   java.nio.file.NoSuchFileException: / var /jenkins_home/jobs/simp/branches/tasks-BET-31993.du2529/builds/8 -> / var /jenkins_home/jobs/simp/branches/tasks-BET-31993.du2529/builds/.8 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at hudson.model.Run.delete(Run.java:1645) at hudson.tasks.LogRotator.perform(LogRotator.java:166)jenkins.util.io.CompositeIOException: Failed to rotate logs for [simp/tasks%2FBET-31993 #8] at hudson.tasks.LogRotator.perform(LogRotator.java:223) at hudson.model.Job.logRotate(Job.java:469) at jenkins.model.JobGlobalBuildDiscarderStrategy.apply(JobGlobalBuildDiscarderStrategy.java:54) at jenkins.model.BackgroundGlobalBuildDiscarder.lambda$processJob$0(BackgroundGlobalBuildDiscarder.java:67) at java.lang.Iterable.forEach(Iterable.java:75) at jenkins.model.BackgroundGlobalBuildDiscarder.processJob(BackgroundGlobalBuildDiscarder.java:61) at jenkins.model.GlobalBuildDiscarderListener.onFinalized(GlobalBuildDiscarderListener.java:49) at hudson.model.listeners.RunListener.fireFinalized(RunListener.java:255) at hudson.model.Run.onEndBuilding(Run.java:2042) at org.jenkinsci.plugins.workflow.job.WorkflowRun.finish(WorkflowRun.java:625) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$800(WorkflowRun.java:137) at org.jenkinsci.plugins.workflow.job.WorkflowRun$GraphL.onNewHead(WorkflowRun.java:1026) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:489) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)  

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              danielbeck Daniel Beck
              Votes:
              20 Vote for this issue
              Watchers:
              32 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: