pipeline job stuck with "Still waiting to schedule task" but node is idling and then whole queue stay in deadlock state

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Critical
    • Component/s: core, pipeline
    • Environment:

      Update 2023-10-27 for LTS v2.414.3

      with Jenkins v2.414.3 this bug is now so massive:

      after each restart service goes into this corrupt state just in a few seconds
      -> Jenkins no longer usable for us
      => after downgrade back to 2.414.2 it is usable again
      => but still worse

      hint: we use extensively the Jenkins API to update labels and config of nodes (>150 nodes in parallel) to balance jobs, I think we found here a race condition

      some minutes after restart Jenkins Server service some pipeline job stuck in state "Still waiting to schedule task" and whole queue deadlock forever

      pipeline example:

      pipeline {
    agent { node { label params.nodename } }
    options { buildDiscarder(logRotator(numToKeepStr: '5000', artifactNumToKeepStr: '5000')) }
    parameters
    {
        string (name: 'nodename', defaultValue: '')
        string (name: 'commandLine', defaultValue: '')
        string (name: 'sourceBranch', defaultValue: 'master', description: '')
    }
    stages {
        stage('Execute') {
            when {
                beforeAgent true
                expression { params.nodename != '' }
            }
            steps {
                bat 'echo hello'
            }
        }
    }
}

      parameter example:
      nodename: VM-514A56DC0979

      node is connected and idling

      finally the whole jenkins queue deadlocks
      no progress until I restart the Jenkins Server service
      but after service restart and some minutes it stuck again in that state

      have also an observation what can be related to this:
      if I want to change configuration of that node it lead to an serious error in Jenkins

      see
      pipeline:

      pipeline job console log:

      parameter values:

      node state:

      queue deadlock:

      thread dump:

      Thread #0
	at DSL.node(java.util.concurrent.ExecutionException: java.lang.ClassCastException)
	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.LabelScript.run(LabelScript.groovy:42)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(ModelInterpreter.groovy:594)
	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(ModelInterpreter.groovy:76)
	at WorkflowScript.run(WorkflowScript:1)

      plugin version:

        1. image-2023-06-02-09-40-59-468.png
          363 kB
          Thomas Patzig
        2. image-2023-06-02-09-45-40-806.png
          294 kB
          Thomas Patzig
        3. image-2023-06-02-09-48-56-261.png
          91 kB
          Thomas Patzig
        4. image-2023-06-02-09-52-34-345.png
          111 kB
          Thomas Patzig
        5. image-2023-06-02-09-55-24-851.png
          98 kB
          Thomas Patzig
        6. image-2023-06-02-09-56-51-926.png
          189 kB
          Thomas Patzig
        7. image-2023-06-02-10-01-37-389.png
          62 kB
          Thomas Patzig
        8. image-2023-06-02-10-11-27-173.png
          151 kB
          Thomas Patzig
        9. image-2023-06-02-10-12-25-538.png
          211 kB
          Thomas Patzig
        10. image-2023-06-02-10-19-55-364.png
          401 kB
          Thomas Patzig
        11. image-2023-08-01-11-52-20-646.png
          379 kB
          Thomas Patzig
        12. image-2023-08-01-12-04-29-412.png
          280 kB
          Thomas Patzig

            Assignee:
            Harsh
            Reporter:
            Thomas Patzig
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Archived: