Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37790

abort on pipeline with parallel not succeeding / providing feedback on nodes

      Jenkinsfile is creating ~24 sub-jobs and running 5 of them at a time in parallel.
      Each sub-job is running up to 5 VMs that need to be cleaned up on abort to make room for the next jobs.

      The only time I have seen abort work on the first click is when all of the jobs are in the queue and none of them are running. Every other time, the console output has given me "Click here to forcibly terminate running steps" and later "Click here to forcibly kill entire build", and if i don't click those links, the job will stick around for days.
      In this instance, i've created two support bundles, at the times indicated.

      When I went to manually clean up the VMs, i found that 2 out of 5 of the workspaces had been successfully cleaned up before I got there, but all 5 workspaces look the same from the POV of the console output...they all look like they failed to abort.

      =============================================================
      Aborted by LeahKlearman

      ...............................................................................................................................................................................................................................................................................................................................................Sending interrupt signal to process
      ..........................................................Sending interrupt signal to process
      Sending interrupt signal to process
      Sending interrupt signal to process
      Sending interrupt signal to process
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [Pipeline] // node
      [looks like all the queued sub-jobs have been successfully canceled here]

      Click here to forcibly terminate running steps
      [clicked "forcibly terminate running steps" now and created bundle support_2016-08-29_20.44.15]
      Terminating sh
      Terminating sh
      Terminating sh
      Terminating sh
      Terminating sh

      [output from one of the jobs]

      Click here to forcibly kill entire build
      [clicked "forcibly kill entire build" now and created support_2016-08-29_20.45.14]
      Hard kill!
      =============================================================

          [JENKINS-37790] abort on pipeline with parallel not succeeding / providing feedback on nodes

          Jesse Glick added a comment -

          I think I saw something similar in parallel-test-executor-plugin/demo but did not check if it was reproducible.

          Jesse Glick added a comment - I think I saw something similar in parallel-test-executor-plugin/demo but did not check if it was reproducible.

          I have been seeing this as well. It is incredibly obnoxious to have to click abort, and then go into the console and click another half to dozen times to get the pipeline to stop.

          [osx-10.11] Sending interrupt signal to process
          [Pipeline] [osx-10.11] }
          [Pipeline] [osx-10.11] // wrap
          [Pipeline] [osx-10.11] wrap
          [Pipeline] [osx-10.11] {
          Click here to forcibly terminate running steps
          Click here to forcibly terminate running steps
          Aborted by Joshua Hoblitt
          Aborted by Joshua Hoblitt
          Aborted by Joshua Hoblitt
          [Pipeline] [osx-10.11] sh
          Terminating sh
          Terminating parallel
          [osx-10.11] [miniconda3-4.2.12-7c8e67] Running shell script
          [osx-10.11] Sending interrupt signal to process
          [osx-10.11] Sending interrupt signal to process
          [osx-10.11] Sending interrupt signal to process
          [osx-10.11] + rm -rf ./build/.lockDir
          [osx-10.11] Sending interrupt signal to process
          [osx-10.11] Sending interrupt signal to process
          [osx-10.11] Sending interrupt signal to process
          [Pipeline] // parallel
          [Pipeline] }
          [Pipeline] [osx-10.11] }
          [Pipeline] // stage
          [Pipeline] echo
          result: FAILURE
          [Pipeline] slackSend
          Slack Send Pipeline step configured values from global config - teamDomain: true, token: true, channel: true, color: false
          ERROR: Slack notification failed. See Jenkins logs for details.
          Click here to forcibly terminate running steps
          Click here to forcibly terminate running steps
          Click here to forcibly terminate running steps
          Click here to forcibly terminate running steps
          Click here to forcibly terminate running steps
          Terminating wrap
          Click here to forcibly terminate running steps
          Click here to forcibly terminate running steps
          Click here to forcibly terminate running steps
          Terminating wrap
          Terminating wrap
          Terminating wrap
          Terminating wrap
          Terminating wrap
          Terminating wrap
          Terminating wrap
          Terminating wrap
          Click here to forcibly kill entire build
          Hard kill!
          

          Joshua Hoblitt added a comment - I have been seeing this as well. It is incredibly obnoxious to have to click abort, and then go into the console and click another half to dozen times to get the pipeline to stop. [osx-10.11] Sending interrupt signal to process [Pipeline] [osx-10.11] } [Pipeline] [osx-10.11] // wrap [Pipeline] [osx-10.11] wrap [Pipeline] [osx-10.11] { Click here to forcibly terminate running steps Click here to forcibly terminate running steps Aborted by Joshua Hoblitt Aborted by Joshua Hoblitt Aborted by Joshua Hoblitt [Pipeline] [osx-10.11] sh Terminating sh Terminating parallel [osx-10.11] [miniconda3-4.2.12-7c8e67] Running shell script [osx-10.11] Sending interrupt signal to process [osx-10.11] Sending interrupt signal to process [osx-10.11] Sending interrupt signal to process [osx-10.11] + rm -rf ./build/.lockDir [osx-10.11] Sending interrupt signal to process [osx-10.11] Sending interrupt signal to process [osx-10.11] Sending interrupt signal to process [Pipeline] // parallel [Pipeline] } [Pipeline] [osx-10.11] } [Pipeline] // stage [Pipeline] echo result: FAILURE [Pipeline] slackSend Slack Send Pipeline step configured values from global config - teamDomain: true , token: true , channel: true , color: false ERROR: Slack notification failed. See Jenkins logs for details. Click here to forcibly terminate running steps Click here to forcibly terminate running steps Click here to forcibly terminate running steps Click here to forcibly terminate running steps Click here to forcibly terminate running steps Terminating wrap Click here to forcibly terminate running steps Click here to forcibly terminate running steps Click here to forcibly terminate running steps Terminating wrap Terminating wrap Terminating wrap Terminating wrap Terminating wrap Terminating wrap Terminating wrap Terminating wrap Click here to forcibly kill entire build Hard kill!

            Unassigned Unassigned
            klrmn Leah Klearman
            Votes:
            6 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: