Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22395

Run.delete (from LogRotator) failing with "...looks to have already been deleted"

    XMLWordPrintable

Details

    Description

      Sometimes "log rotation" fails with an exception like

      SEVERE  hudson.model.Run#execute: Failed to rotate log
      java.io.IOException: .../jobs/.../modules/...$.../builds/2013-... looks to have already been deleted
          at hudson.model.Run.delete(Run.java:1432)
          at hudson.maven.MavenModuleSetBuild.delete(MavenModuleSetBuild.java:420)
          at hudson.tasks.LogRotator.perform(LogRotator.java:136)
          at hudson.model.Job.logRotate(Job.java:437)
          at hudson.maven.MavenModuleSet.logRotate(MavenModuleSet.java:851)
          at hudson.model.Run.execute(Run.java:1728)
          at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:509)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:246)
      

      Usually MavenModuleSetBuild is involved, perhaps suggesting a problem with deleted or skipped module builds (I cannot reproduce in either scenario), though I think I have also seen this happen for freestyle builds. Unclear whether the directory actually exists but File.isDirectory thinks it does not (the code originally tried to delete the dir without checking this and failed with "... is in use"); or whether the directory was actually deleted earlier but this Run was not cleaned up properly for some reason.

      Hypothesis: AbstractLazyLoadRunMap.idOnDisk is not removed by removeValue, called from AbstractProject.removeRun (from Run.delete). Perhaps something is later resurrecting the AbstractBuild from idOnDisk, making it again available for another round of log rotation? But if the actual directory was deleted, it is hard to see how: load would just fail.

      Or perhaps Run.delete is the race condition: the run is removed from the parent after its directory has been deleted. The method is synchronized, but that does not help if two copies of the Run exist, which might happen due to other lazy-loading bugs.

      Diagnostics added to date:

      Attachments

        Issue Links

          Activity

            jglick Jesse Glick created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-19377 [ JENKINS-19377 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-17508 [ JENKINS-17508 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-17553 [ JENKINS-17553 ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Labels exception log-rotator exception lazy-loading log-rotator
            jglick Jesse Glick made changes -
            Attachment JENKINS-22395.diff [ 25692 ]
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            teilo James Nord made changes -
            Labels exception lazy-loading log-rotator exception lazy-loading log-rotator lts-candidate
            jglick Jesse Glick made changes -
            Labels exception lazy-loading log-rotator lts-candidate exception lazy-loading log-rotator
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            scm_issue_link SCM/JIRA link daemon made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]
            jglick Jesse Glick made changes -
            Labels exception lazy-loading log-rotator exception lazy-loading log-rotator lts-candidate
            olivergondza Oliver Gond┼ża made changes -
            Labels exception lazy-loading log-rotator lts-candidate 1.565.2-fixed exception lazy-loading log-rotator
            jglick Jesse Glick made changes -
            Link This issue depends on JENKINS-25788 [ JENKINS-25788 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 154496 ] JNJira + In-Review [ 194923 ]
            svanoort Sam Van Oort made changes -
            Assignee Jesse Glick [ jglick ]
            Resolution Fixed [ 1 ]
            Status Resolved [ 5 ] Reopened [ 4 ]
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status Reopened [ 4 ] Resolved [ 5 ]

            People

              jglick Jesse Glick
              jglick Jesse Glick
              Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: