Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73824

Deletion of Pipeline jobs does not wait for ongoing builds to fully complete

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core

      See https://github.com/jenkinsci/jenkins/pull/9790. Description reproduced here:

      While investigating an issue with branch indexing for multibranch projects leaving build directories behind when deleting projects, I think I found a rather severe issue with job deletion in general for Pipelines. Since its introduction in https://github.com/jenkinsci/jenkins/pull/2789, the logic for cancelling ongoing builds and waiting for them to complete when deleting their parent job has checked Thread.isAlive rather than Executor.isActive, which is not correct for asynchronous tasks such as the main Pipeline execution. See the Javadoc here. Oleg actually suggested changing to isActive in the original PR here for other reasons.

      The result is that although ongoing Pipeline builds are interrupted, ItemDeletion.cancelBuildsInProgress does not wait for those builds to fully complete, which can at least lead to files being written back into the job directory that was just deleted. There are probably other more exotic issues possible for Pipelines that do not shut down quickly when interrupted.

      If there are any other build types that use AsynchronousExecution, they would also be affected.

          [JENKINS-73824] Deletion of Pipeline jobs does not wait for ongoing builds to fully complete

          Devin Nusbaum created issue -
          Devin Nusbaum made changes -
          Description Original: See https://github.com/jenkinsci/jenkins/pull/9790. Description reproduced here:

          While investigating an issue with branch indexing for multibranch projects leaving build directories behind when deleting projects, I think I found a rather severe issue with job deletion in general for Pipelines. Since its introduction in https://github.com/jenkinsci/jenkins/pull/2789, the logic for cancelling ongoing builds and waiting for them to complete when deleting their parent job has checked `Thread.isAlive` rather than {{Executor.isActive}}, which is not correct for asynchronous tasks such as the main Pipeline execution. See the Javadoc here|https://github.com/jenkinsci/jenkins/blob/435bb7989126d6ef3e3d78ddc5597354b71bff74/core/src/main/java/hudson/model/Executor.java#L641-L649]. Oleg actually suggested changing to {{isActive}} in the original PR [here|https://github.com/jenkinsci/jenkins/pull/2789#discussion_r105224731] for other reasons.

          The result is that although ongoing Pipeline builds are interrupted, {{ItemDeletion.cancelBuildsInProgress}} does not wait for those builds to fully complete, which can at least lead to files being written back into the job directory that was just deleted. There are probably other more exotic issues possible for Pipelines that do not shut down quickly when interrupted.
          New: See https://github.com/jenkinsci/jenkins/pull/9790. Description reproduced here:

          While investigating an issue with branch indexing for multibranch projects leaving build directories behind when deleting projects, I think I found a rather severe issue with job deletion in general for Pipelines. Since its introduction in https://github.com/jenkinsci/jenkins/pull/2789, the logic for cancelling ongoing builds and waiting for them to complete when deleting their parent job has checked {{Thread.isAlive}} rather than {{Executor.isActive}}, which is not correct for asynchronous tasks such as the main Pipeline execution. See the Javadoc [here|https://github.com/jenkinsci/jenkins/blob/435bb7989126d6ef3e3d78ddc5597354b71bff74/core/src/main/java/hudson/model/Executor.java#L641-L649]. Oleg actually suggested changing to {{isActive}} in the original PR [here|https://github.com/jenkinsci/jenkins/pull/2789#discussion_r105224731] for other reasons.

          The result is that although ongoing Pipeline builds are interrupted, {{ItemDeletion.cancelBuildsInProgress}} does not wait for those builds to fully complete, which can at least lead to files being written back into the job directory that was just deleted. There are probably other more exotic issues possible for Pipelines that do not shut down quickly when interrupted.
          Devin Nusbaum made changes -
          Summary Original: Pipeline jobs can be deleted while their builds are not yet fully completed New: Pipeline job deletion does not wait for ongoing builds to fully complete
          Devin Nusbaum made changes -
          Summary Original: Pipeline job deletion does not wait for ongoing builds to fully complete New: Deletion of Pipeline jobs does not wait for ongoing builds to fully complete
          Devin Nusbaum made changes -
          Description Original: See https://github.com/jenkinsci/jenkins/pull/9790. Description reproduced here:

          While investigating an issue with branch indexing for multibranch projects leaving build directories behind when deleting projects, I think I found a rather severe issue with job deletion in general for Pipelines. Since its introduction in https://github.com/jenkinsci/jenkins/pull/2789, the logic for cancelling ongoing builds and waiting for them to complete when deleting their parent job has checked {{Thread.isAlive}} rather than {{Executor.isActive}}, which is not correct for asynchronous tasks such as the main Pipeline execution. See the Javadoc [here|https://github.com/jenkinsci/jenkins/blob/435bb7989126d6ef3e3d78ddc5597354b71bff74/core/src/main/java/hudson/model/Executor.java#L641-L649]. Oleg actually suggested changing to {{isActive}} in the original PR [here|https://github.com/jenkinsci/jenkins/pull/2789#discussion_r105224731] for other reasons.

          The result is that although ongoing Pipeline builds are interrupted, {{ItemDeletion.cancelBuildsInProgress}} does not wait for those builds to fully complete, which can at least lead to files being written back into the job directory that was just deleted. There are probably other more exotic issues possible for Pipelines that do not shut down quickly when interrupted.
          New: See https://github.com/jenkinsci/jenkins/pull/9790. Description reproduced here:

          While investigating an issue with branch indexing for multibranch projects leaving build directories behind when deleting projects, I think I found a rather severe issue with job deletion in general for Pipelines. Since its introduction in https://github.com/jenkinsci/jenkins/pull/2789, the logic for cancelling ongoing builds and waiting for them to complete when deleting their parent job has checked {{Thread.isAlive}} rather than {{Executor.isActive}}, which is not correct for asynchronous tasks such as the main Pipeline execution. See the Javadoc [here|https://github.com/jenkinsci/jenkins/blob/435bb7989126d6ef3e3d78ddc5597354b71bff74/core/src/main/java/hudson/model/Executor.java#L641-L649]. Oleg actually suggested changing to {{isActive}} in the original PR [here|https://github.com/jenkinsci/jenkins/pull/2789#discussion_r105224731] for other reasons.

          The result is that although ongoing Pipeline builds are interrupted, {{ItemDeletion.cancelBuildsInProgress}} does not wait for those builds to fully complete, which can at least lead to files being written back into the job directory that was just deleted. There are probably other more exotic issues possible for Pipelines that do not shut down quickly when interrupted.

          If there are any other build types that use {{AsynchronousExecution}}, they would also be affected.
          Devin Nusbaum made changes -
          Remote Link New: This issue links to "jenkinsci/jenkins#9790 (Web Link)" [ 30014 ]
          Devin Nusbaum made changes -
          Remote Link New: This issue links to "jenkinsci/workflow-job-plugin#468 (Web Link)" [ 30015 ]
          Devin Nusbaum made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Devin Nusbaum made changes -
          Status Original: In Progress [ 3 ] New: In Review [ 10005 ]
          Devin Nusbaum made changes -
          Labels New: lts-candidate
          Mark Waite made changes -
          Released As New: 2.480
          Resolution New: Fixed [ 1 ]
          Status Original: In Review [ 10005 ] New: Resolved [ 5 ]

            dnusbaum Devin Nusbaum
            dnusbaum Devin Nusbaum
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: