Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71388

pipeline job stuck with "Still waiting to schedule task" but node is idling and then whole queue stay in deadlock state

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • core, pipeline

      Update 2023-10-27 for LTS v2.414.3

      with Jenkins v2.414.3 this bug is now so massive:

      after each restart service goes into this corrupt state just in a few seconds
      -> Jenkins no longer usable for us
      => after downgrade back to 2.414.2 it is usable again
      => but still worse

      hint: we use extensively the Jenkins API to update labels and config of nodes (>150 nodes in parallel) to balance jobs, I think we found here a race condition

      some minutes after restart Jenkins Server service some pipeline job stuck in state "Still waiting to schedule task" and whole queue deadlock forever

      pipeline example:

      pipeline {
    agent { node { label params.nodename } }
    options { buildDiscarder(logRotator(numToKeepStr: '5000', artifactNumToKeepStr: '5000')) }
    parameters
    {
        string (name: 'nodename', defaultValue: '')
        string (name: 'commandLine', defaultValue: '')
        string (name: 'sourceBranch', defaultValue: 'master', description: '')
    }
    stages {
        stage('Execute') {
            when {
                beforeAgent true
                expression { params.nodename != '' }
            }
            steps {
                bat 'echo hello'
            }
        }
    }
}

      parameter example:
      nodename: VM-514A56DC0979

      node is connected and idling

      finally the whole jenkins queue deadlocks
      no progress until I restart the Jenkins Server service
      but after service restart and some minutes it stuck again in that state

      have also an observation what can be related to this:
      if I want to change configuration of that node it lead to an serious error in Jenkins

      see
      pipeline:

      pipeline job console log:

      parameter values:

      node state:

      queue deadlock:

      thread dump:

      Thread #0
	at DSL.node(java.util.concurrent.ExecutionException: java.lang.ClassCastException)
	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(LabelScript.groovy:42)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(ModelInterpreter.groovy:594)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(ModelInterpreter.groovy:76)
	at WorkflowScript.run(WorkflowScript:1)

      plugin version:

        1. image-2023-06-02-09-40-59-468.png
          363 kB
          Thomas Patzig
        2. image-2023-06-02-09-45-40-806.png
          294 kB
          Thomas Patzig
        3. image-2023-06-02-09-48-56-261.png
          91 kB
          Thomas Patzig
        4. image-2023-06-02-09-52-34-345.png
          111 kB
          Thomas Patzig
        5. image-2023-06-02-09-55-24-851.png
          98 kB
          Thomas Patzig
        6. image-2023-06-02-09-56-51-926.png
          189 kB
          Thomas Patzig
        7. image-2023-06-02-10-01-37-389.png
          62 kB
          Thomas Patzig
        8. image-2023-06-02-10-11-27-173.png
          151 kB
          Thomas Patzig
        9. image-2023-06-02-10-12-25-538.png
          211 kB
          Thomas Patzig
        10. image-2023-06-02-10-19-55-364.png
          401 kB
          Thomas Patzig
        11. image-2023-08-01-11-52-20-646.png
          379 kB
          Thomas Patzig
        12. image-2023-08-01-12-04-29-412.png
          280 kB
          Thomas Patzig

            Unassigned Unassigned
            thomaspatzig Thomas Patzig
            Votes:
            14 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: