Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50960

Pipeline jobs hangs after restart with build trigger step

    XMLWordPrintable

Details

    Description

      Hello,

      I finally reproduced the issue with pipeline stuck on our instance.

      how to reproduce it:

      1. create pipeline with build trigger step
      2. run pipeline
      3. kill/restart jenkins during execution of job which was triggered by the pipeline job (execution has to be started). The goal is to interrupt it without calling listener about stopping/failing execution (which usually can happen by restart or falling jenkins instance)
      4. after start of jenkins, there is pipeline job endlessly waiting for finishing the execution of triggered job and calling from listener about completion of the job execution.

       

      If I understand correctly the issue happens due this code:

      when execution of the step is called the resume value of execution thread is set on null

      https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java#L173

      the thread is not runnable until resume value is non null.

      https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java#L138

      until isRunnable returns true, thread group do not continue execution of this thread (and pipeline script) and expects that thread is waiting for completion of something (which in aforementioned case will never happen).

      https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThreadGroup.java#L331

       

      In case of build step trigger, the listener should fire that everything is completed and set it in context which causes that resume value is no longer null and new run of thread group is scheduled.

      https://github.com/jenkinsci/pipeline-build-step-plugin/blob/pipeline-build-step-2.7/src/main/java/org/jenkinsci/plugins/workflow/support/steps/build/BuildTriggerListener.java#L47

      https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsStepContext.java#L333

      https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsStepContext.java#L340

      https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsStepContext.java#L443

      https://github.com/jenkinsci/workflow-cps-plugin/blob/workflow-cps-2.49/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsThread.java#L261

       

      If I understand it correctly, currently everything is dependable on calling listener which does not have to be called by build in some unexpected situation or bug. It is not mistake in pipeline but I think that pipeline (or at least build trigger step) should count with such situation and handle it better. Because when the pipeline is in such state, this state is saved and no matter how many times the Jenkins is restarted it hangs forever until someone interrupt it form UI. 

      I thought about two solutions:

      build trigger solution - periodically check if job is still executing (no very nice solution)

      pipeline solution - extends api from something like - stillNotCompleted method, where it will periodically ask the build step if we are still waiting for some completion. 

       

      What do you think about it? Do you have any better idea how to solve it?

       

       

      Attachments

        Activity

          I added jobs directory where the jobs (with name "sleep") is in that state.

          lvotypkova Lucie Votypkova added a comment - I added jobs directory where the jobs (with name "sleep") is in that state.

          svanoort can you look at it?

          lvotypkova Lucie Votypkova added a comment - svanoort can you look at it?
          svanoort Sam Van Oort added a comment -

          lvotypkova Is this still reproducible?

          svanoort Sam Van Oort added a comment - lvotypkova Is this still reproducible?

          People

            Unassigned Unassigned
            lvotypkova Lucie Votypkova
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: