Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-75339

Pipeline plugins upgrade from 2.2221 to 2.2234 breaks pipeline workflow

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major

      Reporting this as I expect failure like this could start affecting more users.

      And we are not sure yet how to solve this for our workflows.

      We do regular upgrades of jenkins plugins and versions and deploy, deploying with this version upgrade looks like it could break workflow in jenkins containers and we will have failures on every site, not sure if in addition every site would have a broken jenkins.

       

      Workflow run after Pipeline upgrades from 2.2221 to 2.2234 failed with java stack-dump in field groovy.lang.Closure.delegate, .. yadda ..,  in object of type org.jenkinsci.plugins.workflow.cps.CpsThreadGroup
      Caused: java.io.InvalidClassException: org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AnyScript; Class does not extend stream superclass .. 

      The workflow failed after a 1 minute sleep  at end of a pipeline stage or at the start of a new pipeline stage. Apologies I am unfamiliar with digging in more to the java stack-dump, full dump below. 

      We run jenkins in a docker container and upgrades are done by stopping docker container and starting new with new image with updated scripts, jenkins plugins, etc ... The XXX-ku-upgrade.yml task does this (by spawning an upgrade container to manage switch-over).

       

      stage('ku upgrade') {
               steps {
                  script {
                      def inventory = readYaml file: 'inventory.yml'

                      if (inventory.all.children.ku.hosts.ku_container.latest_version == inventory.all.children.ku.hosts.ku_container.current_version)

      {                     echo 'No new version downloaded. Skip upgrade.'                     currentBuild.result = 'SUCCESS'                     return                 }

                      ansiblePlaybook inventory: 'inventory.d/', playbook: '/ansible_playbooks/kuku/XXX-ku-upgrade.yml'
                  }
                  sleep 60
               }
            }
            stage('Send deploy event') {
                steps {
                    sleep 60
                    script {
                          def inventory = readYaml file: 'inventory.yml'

                          if (inventory.all.children.ku.hosts.ku_container.latest_version == inventory.all.children.ku.hosts.ku_container.current_version)

      {                         echo 'No upgrade was done. Skip event.'                         currentBuild.result = 'SUCCESS'                         return                     }

                          timeout(time: 60, unit: 'MINUTES')

      {                         build job: "Event-job", propagate: false,                         parameters: [                               string(name: 'EVENT', value: "DeployedOK"),                               string(name: 'VERSION', value: inventory.all.children.ku.hosts.ku_container.latest_version),                               string(name: 'PARENT_JOB_NAME', value: "KU-workflow")                         ]                     }

                      }
                  }
              }

         }

       

      We can see warning about docker-workflow plugin (in release 2.2221) but we do not use that.

      "Warning: this is a breaking change for docker-workflow. Please update to https://github.com/jenkinsci/docker-workflow-plugin/releases/tag/599.v76126c79a_a_2d or newer.

      Also, Declarative builds running across the upgrade may fail. New builds should work."

       

      Workflow failure on upgrade, from workflow job/console:

      No debug.log produced

      [Pipeline] }
      [Pipeline] // script
      [Pipeline] sleep
      Sleeping for 1 min 0 sec
      an exception which occurred:
          in field groovy.lang.Closure.delegate
          in object org.jenkinsci.plugins.workflow.cps.CpsClosure2@1c5abcac
          in object of type org.jenkinsci.plugins.workflow.cps.CpsClosure2
          in field groovy.lang.Closure.delegate
          in object org.jenkinsci.plugins.workflow.cps.CpsClosure2@3ed08b09
          in object of type org.jenkinsci.plugins.workflow.cps.CpsClosure2
          in object of type java.util.HashMap
          in field org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.closures
          in object org.jenkinsci.plugins.workflow.cps.CpsThreadGroup@7726ffa3
          in object of type org.jenkinsci.plugins.workflow.cps.CpsThreadGroup
      Caused: java.io.InvalidClassException: org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AnyScript; Class does not extend stream superclass
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:1122)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1431)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:312)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:259)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1978)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1891)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1837)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1837)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1472)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:312)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:259)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1978)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1891)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1837)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1837)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1472)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:312)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:149)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:135)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.MarshallerObjectInputStream.readObjectOverride(MarshallerObjectInputStream.java:53)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverObjectInputStream.readObjectOverride(RiverObjectInputStream.java:307)
          at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:500)
          at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:467)
          at java.base/java.util.HashMap.readObject(HashMap.java:1552)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.reflect.SerMethods.callReadObject(SerMethods.java:83)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:252)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1868)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1472)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:312)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:259)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1978)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1891)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1472)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:312)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:244)
          at PluginClassLoader for workflow-support//org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
          at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:331)
          at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader$SandboxedUnmarshaller.sandbox(RiverReader.java:250)
          at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader$SandboxedUnmarshaller.readObject(RiverReader.java:263)
          at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:879)
          at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$2.onSuccess(CpsFlowExecution.java:872)
          at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures$1.run(Futures.java:147)
          at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.DirectExecutor.execute(DirectExecutor.java:33)
          at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1307)
          at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:772)
          at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:157)
          at PluginClassLoader for workflow-support//org.jenkinsci.plugins.workflow.support.concurrent.Futures.addCallback(Futures.java:97)
          at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramAsync(CpsFlowExecution.java:869)
          at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:836)
          at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:733)
          at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:597)
          at hudson.model.RunMap.retrieve(RunMap.java:273)
          at hudson.model.RunMap.retrieve(RunMap.java:65)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:703)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:685)
          at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:579)
          at hudson.model.RunMap.getById(RunMap.java:237)
          at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.run(WorkflowRun.java:979)
          at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.get(WorkflowRun.java:991)
          at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.flow.FlowExecutionList$DefaultStorage.resume(FlowExecutionList.java:252)
          at PluginClassLoader for workflow-api//org.jenkinsci.plugins.workflow.flow.FlowExecutionList.resume(FlowExecutionList.java:127)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:67)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base/java.lang.Thread.run(Thread.java:840)
      ERROR: Failed to load program
      Finished: FAILURE

          [JENKINS-75339] Pipeline plugins upgrade from 2.2221 to 2.2234 breaks pipeline workflow

          James Coleman created issue -
          James Coleman made changes -
          Environment Original: Version 2.492.1

          Recently older versions without issue:
            Pipeline Version 600.vb_57cdd26fdd7
            Pipeline: Model API Version 2.2221.vc657003fb_d93
            Pipeline: Stage Tags Metadata Version 2.2221.vc657003fb_d93
            Pipeline: Declarative Version 2.2221.vc657003fb_d93
            Pipeline: Declarative Extension Points API Version 2.2221.vc657003fb_d93

          Upgraded and re-deployed jenkins in docker container, same jenkins version, new plugin versions for these:
            Pipeline Version 600.vb_57cdd26fdd7
            Pipeline: Model API Version 2.2234.v4a_b_13b_8cd590
            Pipeline: Stage Tags Metadata Version 2.2234.v4a_b_13b_8cd590
            Pipeline: Declarative Version 2.2234.v4a_b_13b_8cd590
            Pipeline: Declarative Extension Points API Version 2.2234.v4a_b_13b_8cd590


          New: Version 2.492.1

          Recently older versions without issue:
            Pipeline Version 600.vb_57cdd26fdd7
            Pipeline: Model API Version 2.2221.vc657003fb_d93
            Pipeline: Stage Tags Metadata Version 2.2221.vc657003fb_d93
            Pipeline: Declarative Version 2.2221.vc657003fb_d93
            Pipeline: Declarative Extension Points API Version 2.2221.vc657003fb_d93

          Upgraded and re-deployed jenkins in docker container, same jenkins version, new plugin versions for these:
            Pipeline Version 600.vb_57cdd26fdd7
            Pipeline: Model API Version 2.2234.v4a_b_13b_8cd590
            Pipeline: Stage Tags Metadata Version 2.2234.v4a_b_13b_8cd590
            Pipeline: Declarative Version 2.2234.v4a_b_13b_8cd590
            Pipeline: Declarative Extension Points API Version 2.2234.v4a_b_13b_8cd590

          bash-4.4$ java --version
          openjdk 17.0.14 2025-01-21 LTS
          OpenJDK Runtime Environment (Red_Hat-17.0.14.0.7-1) (build 17.0.14+7-LTS)
          OpenJDK 64-Bit Server VM (Red_Hat-17.0.14.0.7-1) (build 17.0.14+7-LTS, mixed mode, sharing)
          James Coleman made changes -
          Component/s New: workflow-cps-plugin [ 21713 ]

          Mark Waite added a comment -

          The release notes for 2.2234 indicate that it is a breaking change for the Docker Pipeline plugin and needs release 599.v76126c79a_a_2d, though you indicate that you aren't using the Docker Pipeline plugin.

          jglick any recommendations for this issue?

          jamesc0 could you provide the complete list of plugins installed and their versions as reported by the system groovy script? That will increase the chances that others can duplicate the issue.

          Mark Waite added a comment - The release notes for 2.2234 indicate that it is a breaking change for the Docker Pipeline plugin and needs release 599.v76126c79a_a_2d , though you indicate that you aren't using the Docker Pipeline plugin. jglick any recommendations for this issue? jamesc0 could you provide the complete list of plugins installed and their versions as reported by the system groovy script ? That will increase the chances that others can duplicate the issue.

          James Coleman added a comment -

          Thanks markewaite , yes I don't think we are running Docker pipeline (unless it has a name not including pipeline or docker).

          Here is the list of plugins and versions, hopefully useful to check.

          Perhaps it is also a breaking change for workflow-cps plugin ?

            Breaks the current workflow with error, I expect subsequent workflows pass ok however more testing needed on that.

           

          Jenkins: 2.492.1
          OS: Linux - 3.10.0-1160.15.2.el7.x86_64
          Java: 17.0.14 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)

          ansible:500.v7564a_db_8feec
          ansicolor:1.0.6
          antisamy-markup-formatter:173.v680e3a_b_69ff3
          apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
          asm-api:9.7.1-97.v4cc844130d97
          badge:2.7
          bootstrap5-api:5.3.3-2
          bouncycastle-api:2.30.1.80-256.vf98926042a_9b_
          branch-api:2.1214.v3f652804588d
          build-monitor-plugin:1.14-961.v676e38a_7a_248
          build-pipeline-plugin:2.0.2
          buildresult-trigger:1.01
          buildtriggerbadge:251.vdf6ef853f3f5
          caffeine-api:3.2.0-161.v691ef352cee1
          checks-api:367.v18b_7f530e54a_
          cloudbees-disk-usage-simple:232.v713eeed2e1f4
          cloudbees-folder:6.982.vf165a_16c9507
          command-launcher:118.v72741845c17a_
          commons-compress-api:1.27.1-2
          commons-lang3-api:3.17.0-84.vb_b_938040b_078
          commons-text-api:1.13.0-153.v91dcd89e2a_22
          conditional-buildstep:1.5.0
          credentials:1408.va_622a_b_f5b_1b_1
          credentials-binding:687.v619cb_15e923f
          cucumber-reports:5.8.5
          cvs:471.v28a_41b_03ee4f
          dashboard-view:2.528.v3470c02b_d7c9
          data-tables-api:2.2.2-1
          disable-job-button:1.v9db_352414f90
          display-url-api:2.209.v582ed814ff2f
          durable-task:581.v299a_5609d767
          echarts-api:5.6.0-1
          eddsa-api:0.3.0-13.v7cb_69ed68f00
          emoji-symbols-api:16.0-26.v9818ff7423f0
          envinject-api:1.235.va_14c74f8f487
          flatpickr-api:4.6.13-15.vf6e534338831
          font-awesome-api:6.7.2-1
          fstrigger:1.02
          groovy:497.v7b_061a_a_de65d
          groovy-postbuild:272.v52a_03efb_8653
          gson-api:2.12.1-113.v347686d6729f
          handlebars:3.0.8
          instance-identity:201.vd2a_b_5a_468a_a_6
          ionicons-api:82.v0597178874e1
          jackson2-api:2.17.0-389.va_5c7e45cd806
          jakarta-activation-api:2.1.3-2
          jakarta-mail-api:2.1.3-2
          javadoc:310.v032f3f16b_0f8
          javax-activation-api:1.2.0-8
          javax-mail-api:1.6.2-11
          jaxb:2.3.9-1
          jdk-tool:83.v417146707a_3d
          joda-time-api:2.13.1-115.va_6b_5f8efb_1d8
          jquery:1.12.4-3
          jquery3-api:3.7.1-3
          jsch:0.2.16-95.v3eecb_55fa_b_78
          json-api:20250107-125.v28b_a_ffa_eb_f01
          json-path-api:2.9.0-148.v22a_7ffe323ce
          junit:1317.v5b_35d792b_06a_
          locale:566.v3323643741cc
          lockable-resources:1301.v0e3b_da_4b_4462
          log-parser:2.3.7
          mailer:489.vd4b_25144138f
          matrix-auth:3.2.4
          matrix-project:845.vffd7fa_f27555
          maven-plugin:3.25
          metrics:4.2.21-464.vc9fa_a_0d6265d
          mina-sshd-api-common:2.14.0-143.v2b_362fc39576
          mina-sshd-api-core:2.14.0-143.v2b_362fc39576
          momentjs:1.1.1
          monitoring:2.5.0
          oss-symbols-api:296.v4981240eeb_1a_
          parameterized-trigger:840.v3c7d4a_a_5e6c7
          pipeline-build-step:555.v589d5c24a_3d6
          pipeline-graph-analysis:231.v56354571a_da_0
          pipeline-groovy-lib:752.vdddedf804e72
          pipeline-input-step:513.v7eb_fe7e2c0fc
          pipeline-milestone-step:127.vb_52887ca_3b_6d
          pipeline-model-api:2.2234.v4a_b_13b_8cd590
          pipeline-model-definition:2.2234.v4a_b_13b_8cd590
          pipeline-model-extensions:2.2234.v4a_b_13b_8cd590
          pipeline-rest-api:2.37
          pipeline-stage-step:322.vecffa_99f371c
          pipeline-stage-tags-metadata:2.2234.v4a_b_13b_8cd590
          pipeline-stage-view:2.37
          pipeline-utility-steps:2.19.0
          plain-credentials:183.va_de8f1dd5a_2b_
          plugin-util-api:6.0.0
          postbuild-task:72.v75dc459e4747
          postbuildscript:3.4.1-695.vf6b_0b_8053979
          prometheus:819.v50953a_c560dd
          rich-text-publisher-plugin:1.5
          run-condition:243.v3c3f94e46a_8b_
          schedule-build:643.v58b_09e4b_748c
          scm-api:703.v72ff4b_259600
          script-security:1373.vb_b_4a_a_c26fa_00
          selected-tests-executor:1.3.3
          simple-theme-plugin:202.v6367d3dea_73b_
          snakeyaml-api:2.3-123.v13484c65210a_
          ssh-credentials:349.vb_8b_6b_9709f5b_
          sshd:3.353.v2b_d33c46e970
          structs:343.vdcf37b_a_c81d5
          support-core:1637.v6a_6b_f31d4373
          test-results-analyzer:0.4.1
          text-finder:1.31
          thinBackup:2.1.2
          token-macro:444.v52de7e9c573d
          trilead-api:2.192.vc50a_d147e369
          uno-choice:2.8.6
          variant:70.va_d9f17f859e0
          workflow-aggregator:600.vb_57cdd26fdd7
          workflow-api:1363.v03f731255494
          workflow-basic-steps:1079.vce64b_a_929c5a_
          workflow-cps:4018.vf02e01888da_f
          workflow-durable-task-step:1405.v1fcd4a_d00096
          workflow-job:1505.vea_4b_20a_4a_495
          workflow-multibranch:803.v08103b_87c280
          workflow-scm-step:437.v05a_f66b_e5ef8
          workflow-step-api:700.v6e45cb_a_5a_a_21
          workflow-support:961.v51869f7b_d409
          xtrigger-api:1.2

          James Coleman added a comment - Thanks markewaite , yes I don't think we are running Docker pipeline (unless it has a name not including pipeline or docker). Here is the list of plugins and versions, hopefully useful to check. Perhaps it is also a breaking change for workflow-cps plugin ?   Breaks the current workflow with error, I expect subsequent workflows pass ok however more testing needed on that.   Jenkins: 2.492.1 OS: Linux - 3.10.0-1160.15.2.el7.x86_64 Java: 17.0.14 - Red Hat, Inc. (OpenJDK 64-Bit Server VM) — ansible:500.v7564a_db_8feec ansicolor:1.0.6 antisamy-markup-formatter:173.v680e3a_b_69ff3 apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83 asm-api:9.7.1-97.v4cc844130d97 badge:2.7 bootstrap5-api:5.3.3-2 bouncycastle-api:2.30.1.80-256.vf98926042a_9b_ branch-api:2.1214.v3f652804588d build-monitor-plugin:1.14-961.v676e38a_7a_248 build-pipeline-plugin:2.0.2 buildresult-trigger:1.01 buildtriggerbadge:251.vdf6ef853f3f5 caffeine-api:3.2.0-161.v691ef352cee1 checks-api:367.v18b_7f530e54a_ cloudbees-disk-usage-simple:232.v713eeed2e1f4 cloudbees-folder:6.982.vf165a_16c9507 command-launcher:118.v72741845c17a_ commons-compress-api:1.27.1-2 commons-lang3-api:3.17.0-84.vb_b_938040b_078 commons-text-api:1.13.0-153.v91dcd89e2a_22 conditional-buildstep:1.5.0 credentials:1408.va_622a_b_f5b_1b_1 credentials-binding:687.v619cb_15e923f cucumber-reports:5.8.5 cvs:471.v28a_41b_03ee4f dashboard-view:2.528.v3470c02b_d7c9 data-tables-api:2.2.2-1 disable-job-button:1.v9db_352414f90 display-url-api:2.209.v582ed814ff2f durable-task:581.v299a_5609d767 echarts-api:5.6.0-1 eddsa-api:0.3.0-13.v7cb_69ed68f00 emoji-symbols-api:16.0-26.v9818ff7423f0 envinject-api:1.235.va_14c74f8f487 flatpickr-api:4.6.13-15.vf6e534338831 font-awesome-api:6.7.2-1 fstrigger:1.02 groovy:497.v7b_061a_a_de65d groovy-postbuild:272.v52a_03efb_8653 gson-api:2.12.1-113.v347686d6729f handlebars:3.0.8 instance-identity:201.vd2a_b_5a_468a_a_6 ionicons-api:82.v0597178874e1 jackson2-api:2.17.0-389.va_5c7e45cd806 jakarta-activation-api:2.1.3-2 jakarta-mail-api:2.1.3-2 javadoc:310.v032f3f16b_0f8 javax-activation-api:1.2.0-8 javax-mail-api:1.6.2-11 jaxb:2.3.9-1 jdk-tool:83.v417146707a_3d joda-time-api:2.13.1-115.va_6b_5f8efb_1d8 jquery:1.12.4-3 jquery3-api:3.7.1-3 jsch:0.2.16-95.v3eecb_55fa_b_78 json-api:20250107-125.v28b_a_ffa_eb_f01 json-path-api:2.9.0-148.v22a_7ffe323ce junit:1317.v5b_35d792b_06a_ locale:566.v3323643741cc lockable-resources:1301.v0e3b_da_4b_4462 log-parser:2.3.7 mailer:489.vd4b_25144138f matrix-auth:3.2.4 matrix-project:845.vffd7fa_f27555 maven-plugin:3.25 metrics:4.2.21-464.vc9fa_a_0d6265d mina-sshd-api-common:2.14.0-143.v2b_362fc39576 mina-sshd-api-core:2.14.0-143.v2b_362fc39576 momentjs:1.1.1 monitoring:2.5.0 oss-symbols-api:296.v4981240eeb_1a_ parameterized-trigger:840.v3c7d4a_a_5e6c7 pipeline-build-step:555.v589d5c24a_3d6 pipeline-graph-analysis:231.v56354571a_da_0 pipeline-groovy-lib:752.vdddedf804e72 pipeline-input-step:513.v7eb_fe7e2c0fc pipeline-milestone-step:127.vb_52887ca_3b_6d pipeline-model-api:2.2234.v4a_b_13b_8cd590 pipeline-model-definition:2.2234.v4a_b_13b_8cd590 pipeline-model-extensions:2.2234.v4a_b_13b_8cd590 pipeline-rest-api:2.37 pipeline-stage-step:322.vecffa_99f371c pipeline-stage-tags-metadata:2.2234.v4a_b_13b_8cd590 pipeline-stage-view:2.37 pipeline-utility-steps:2.19.0 plain-credentials:183.va_de8f1dd5a_2b_ plugin-util-api:6.0.0 postbuild-task:72.v75dc459e4747 postbuildscript:3.4.1-695.vf6b_0b_8053979 prometheus:819.v50953a_c560dd rich-text-publisher-plugin:1.5 run-condition:243.v3c3f94e46a_8b_ schedule-build:643.v58b_09e4b_748c scm-api:703.v72ff4b_259600 script-security:1373.vb_b_4a_a_c26fa_00 selected-tests-executor:1.3.3 simple-theme-plugin:202.v6367d3dea_73b_ snakeyaml-api:2.3-123.v13484c65210a_ ssh-credentials:349.vb_8b_6b_9709f5b_ sshd:3.353.v2b_d33c46e970 structs:343.vdcf37b_a_c81d5 support-core:1637.v6a_6b_f31d4373 test-results-analyzer:0.4.1 text-finder:1.31 thinBackup:2.1.2 token-macro:444.v52de7e9c573d trilead-api:2.192.vc50a_d147e369 uno-choice:2.8.6 variant:70.va_d9f17f859e0 workflow-aggregator:600.vb_57cdd26fdd7 workflow-api:1363.v03f731255494 workflow-basic-steps:1079.vce64b_a_929c5a_ workflow-cps:4018.vf02e01888da_f workflow-durable-task-step:1405.v1fcd4a_d00096 workflow-job:1505.vea_4b_20a_4a_495 workflow-multibranch:803.v08103b_87c280 workflow-scm-step:437.v05a_f66b_e5ef8 workflow-step-api:700.v6e45cb_a_5a_a_21 workflow-support:961.v51869f7b_d409 xtrigger-api:1.2
          Mark Waite made changes -
          Attachment New: plugins.txt [ 64023 ]
          Mark Waite made changes -
          Attachment New: run-jenkins.sh [ 64024 ]

          Mark Waite added a comment -

          I can't duplicate the problem in my environment. I must be missing something about how you are running the job or what the job is doing. Steps that I took while trying to duplicate the issue:

          1. Create a plugins.txt file that lists all the plugins and their versions. I used your list, though I deleted the two deprecated plugins, since they are unused
          2. Create a run-jenkins.sh shell script that starts Jenkins 2.492.1, installs those plugins, and starts the setup wizard
          3. Run the run-jenkins.sh shell script and complete the setup wizard by creating a user and installing no additional plugins
          4. Create the job Event-job with the following declarative Pipeline script
            pipeline {
                agent any
                parameters {
                    choice name: 'EVENT', description: 'Choose the event that started the job', choices: ['DeployedOK', 'NotDeployedOK']
                    string name: 'VERSION', description: 'Provide the component version', defaultValue: 'unknown'
                    string name: 'PARENT_JOB_NAME', description: 'Name of the parent job', defaultValue: 'unknown'
                }
            
                stages {
                    stage('Hello') {
                        steps {
                            echo "Hello World from ${params.EVENT}, ${params.VERSION}, ${params.PARENT_JOB_NAME}"
                        }
                    }
                }
            }
                
          5. Run the 'Event-job' once to parse the parameter definition, then run it again to confirm parameters are handled correctly
          6. Create a job ansible-pipeline with the following declarative script
            pipeline {
                agent {
                    label '!windows'
                }
            
                stages {
                    stage('ku upgrade') {
                        steps {
                            script {
                                // def inventory = readYaml file: 'inventory.yml'
                                def inventory = [
                                    all: [
                                        children: [
                                            ku: [
                                                hosts: [
                                                    ku_container: [
                                                        latest_version: "1.23.4",
                                                        current_version: "1.23.4"
                                                    ]
                                                ]
                                            ]
                                        ]
                                    ]
                                ]
                                if (inventory.all.children.ku.hosts.ku_container.latest_version == inventory.all.children.ku.hosts.ku_container.current_version) {
                                    echo 'No new version downloaded. Skip upgrade.'
                                    currentBuild.result = 'SUCCESS'
                                    return
                                }
                            }
                            timeout(time: 60, unit: 'MINUTES') {
                                build job: 'Event-job', 
                                        parameters: [
                                            string(name: 'EVENT', value: 'DeployedOK'), 
                                            string(name: 'VERSION', value: 'my-version'), 
                                            string(name: 'PARENT_JOB_NAME', value: 'build-another')]
                            }
                        }
                    }
                }
            }
                
          7. Run the ansible-pipeline job and confirm that it executes the Event-job as expected

          Pleas provide additional details so that others can duplicate the issue.

          Mark Waite added a comment - I can't duplicate the problem in my environment. I must be missing something about how you are running the job or what the job is doing. Steps that I took while trying to duplicate the issue: Create a plugins.txt file that lists all the plugins and their versions. I used your list, though I deleted the two deprecated plugins, since they are unused Create a run-jenkins.sh shell script that starts Jenkins 2.492.1, installs those plugins, and starts the setup wizard Run the run-jenkins.sh shell script and complete the setup wizard by creating a user and installing no additional plugins Create the job Event-job with the following declarative Pipeline script pipeline { agent any parameters { choice name: 'EVENT' , description: 'Choose the event that started the job' , choices: [ 'DeployedOK' , 'NotDeployedOK' ] string name: 'VERSION' , description: 'Provide the component version' , defaultValue: 'unknown' string name: 'PARENT_JOB_NAME' , description: 'Name of the parent job' , defaultValue: 'unknown' } stages { stage( 'Hello' ) { steps { echo "Hello World from ${params.EVENT}, ${params.VERSION}, ${params.PARENT_JOB_NAME}" } } } } Run the 'Event-job' once to parse the parameter definition, then run it again to confirm parameters are handled correctly Create a job ansible-pipeline with the following declarative script pipeline { agent { label '!windows' } stages { stage( 'ku upgrade' ) { steps { script { // def inventory = readYaml file: 'inventory.yml' def inventory = [ all: [ children: [ ku: [ hosts: [ ku_container: [ latest_version: "1.23.4" , current_version: "1.23.4" ] ] ] ] ] ] if (inventory.all.children.ku.hosts.ku_container.latest_version == inventory.all.children.ku.hosts.ku_container.current_version) { echo 'No new version downloaded. Skip upgrade.' currentBuild.result = 'SUCCESS' return } } timeout(time: 60, unit: 'MINUTES' ) { build job: 'Event-job' , parameters: [ string(name: 'EVENT' , value: 'DeployedOK' ), string(name: 'VERSION' , value: 'my-version' ), string(name: 'PARENT_JOB_NAME' , value: 'build-another' )] } } } } } Run the ansible-pipeline job and confirm that it executes the Event-job as expected Pleas provide additional details so that others can duplicate the issue.

          James Coleman added a comment -

          The issue happens for us on a workflow job that runs across an upgrade of our jenkins container.

          Workflow runs to upgrade our jenkins and other docker containers.

          At the initial part of workflow older versions of plugins are running, the list of versions i here below.

           

          At the ansible playbook XXX-ku-upgrade.yml then the old jenkins container is stopped

          And new jenkins container started with new plugins (and same jenkins home) ...

          And the workflow resumes .. (but hits the issue quickly and fails).

          Normally the workflow would resume and run other tasks to upgrade the full system - all docker containers of different types.

           

          Thanks very much markewaite for working on this already!

          I hope this describes it better.

          This workflow across jenkins upgrade is how we keep our jenkins up to date.

           

          VERSIONS FROM jenkins BEFORE upgrade:

           

          Jenkins: 2.492.1
          OS: Linux - 3.10.0-1160.11.1.el7.x86_64
          Java: 17.0.14 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)

          ansible:500.v7564a_db_8feec
          ansicolor:1.0.6
          antisamy-markup-formatter:173.v680e3a_b_69ff3
          apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
          asm-api:9.7.1-97.v4cc844130d97
          badge:2.7
          bootstrap5-api:5.3.3-2
          bouncycastle-api:2.30.1.80-256.vf98926042a_9b_
          branch-api:2.1214.v3f652804588d
          build-monitor-plugin:1.14-961.v676e38a_7a_248
          build-pipeline-plugin:2.0.2
          buildresult-trigger:1.01
          buildtriggerbadge:251.vdf6ef853f3f5
          caffeine-api:3.2.0-161.v691ef352cee1
          checks-api:2.2.3
          cloudbees-disk-usage-simple:232.v713eeed2e1f4
          cloudbees-folder:6.982.vf165a_16c9507
          command-launcher:118.v72741845c17a_
          commons-compress-api:1.27.1-2
          commons-lang3-api:3.17.0-84.vb_b_938040b_078
          commons-text-api:1.13.0-153.v91dcd89e2a_22
          conditional-buildstep:1.5.0
          credentials:1408.va_622a_b_f5b_1b_1
          credentials-binding:687.v619cb_15e923f
          cucumber-reports:5.8.5
          cvs:471.v28a_41b_03ee4f
          dashboard-view:2.528.v3470c02b_d7c9
          data-tables-api:2.2.2-1
          disable-job-button:1.v9db_352414f90
          display-url-api:2.209.v582ed814ff2f
          durable-task:581.v299a_5609d767
          echarts-api:5.6.0-1
          eddsa-api:0.3.0-13.v7cb_69ed68f00
          emoji-symbols-api:16.0-26.v9818ff7423f0
          envinject-api:1.235.va_14c74f8f487
          flatpickr-api:4.6.13-15.vf6e534338831
          font-awesome-api:6.7.2-1
          fstrigger:1.02
          groovy:497.v7b_061a_a_de65d
          groovy-postbuild:272.v52a_03efb_8653
          gson-api:2.12.1-113.v347686d6729f
          handlebars:3.0.8
          instance-identity:201.vd2a_b_5a_468a_a_6
          ionicons-api:82.v0597178874e1
          jackson2-api:2.17.0-389.va_5c7e45cd806
          jakarta-activation-api:2.1.3-2
          jakarta-mail-api:2.1.3-2
          javadoc:310.v032f3f16b_0f8
          javax-activation-api:1.2.0-8
          javax-mail-api:1.6.2-11
          jaxb:2.3.9-1
          jdk-tool:83.v417146707a_3d
          joda-time-api:2.13.1-115.va_6b_5f8efb_1d8
          jquery:1.12.4-3
          jquery3-api:3.7.1-3
          jsch:0.2.16-86.v42e010d9484b_
          json-api:20250107-125.v28b_a_ffa_eb_f01
          json-path-api:2.9.0-148.v22a_7ffe323ce
          junit:1317.v5b_35d792b_06a_
          locale:566.v3323643741cc
          lockable-resources:1301.v0e3b_da_4b_4462
          log-parser:2.3.7
          mailer:489.vd4b_25144138f
          matrix-auth:3.2.4
          matrix-project:845.vffd7fa_f27555
          maven-plugin:3.25
          metrics:4.2.21-464.vc9fa_a_0d6265d
          mina-sshd-api-common:2.14.0-143.v2b_362fc39576
          mina-sshd-api-core:2.14.0-143.v2b_362fc39576
          momentjs:1.1.1
          monitoring:2.5.0
          oss-symbols-api:296.v4981240eeb_1a_
          parameterized-trigger:840.v3c7d4a_a_5e6c7
          pipeline-build-step:555.v589d5c24a_3d6
          pipeline-graph-analysis:231.v56354571a_da_0
          pipeline-groovy-lib:752.vdddedf804e72
          pipeline-input-step:513.v7eb_fe7e2c0fc
          pipeline-milestone-step:127.vb_52887ca_3b_6d
          pipeline-model-api:2.2221.vc657003fb_d93
          pipeline-model-definition:2.2221.vc657003fb_d93
          pipeline-model-extensions:2.2221.vc657003fb_d93
          pipeline-rest-api:2.37
          pipeline-stage-step:322.vecffa_99f371c
          pipeline-stage-tags-metadata:2.2221.vc657003fb_d93
          pipeline-stage-view:2.37
          pipeline-utility-steps:2.19.0
          plain-credentials:183.va_de8f1dd5a_2b_
          plugin-util-api:6.0.0
          postbuild-task:72.v75dc459e4747
          postbuildscript:3.4.1-695.vf6b_0b_8053979
          prometheus:819.v50953a_c560dd
          rich-text-publisher-plugin:1.5
          run-condition:243.v3c3f94e46a_8b_
          schedule-build:643.v58b_09e4b_748c
          scm-api:703.v72ff4b_259600
          script-security:1373.vb_b_4a_a_c26fa_00
          selected-tests-executor:1.3.3
          simple-theme-plugin:202.v6367d3dea_73b_
          snakeyaml-api:2.3-123.v13484c65210a_
          ssh-credentials:349.vb_8b_6b_9709f5b_
          sshd:3.353.v2b_d33c46e970
          structs:343.vdcf37b_a_c81d5
          support-core:1637.v6a_6b_f31d4373
          test-results-analyzer:0.4.1
          text-finder:1.31
          thinBackup:2.1.2
          token-macro:444.v52de7e9c573d
          trilead-api:2.190.v1ed19f8137f8
          uno-choice:2.8.6
          variant:70.va_d9f17f859e0
          workflow-aggregator:600.vb_57cdd26fdd7
          workflow-api:1363.v03f731255494
          workflow-basic-steps:1079.vce64b_a_929c5a_
          workflow-cps:4018.vf02e01888da_f
          workflow-durable-task-step:1405.v1fcd4a_d00096
          workflow-job:1505.vea_4b_20a_4a_495
          workflow-multibranch:800.v5f0a_a_660950e
          workflow-scm-step:437.v05a_f66b_e5ef8
          workflow-step-api:700.v6e45cb_a_5a_a_21
          workflow-support:961.v51869f7b_d409
          xtrigger-api:1.2

           

          James Coleman added a comment - The issue happens for us on a workflow job that runs across an upgrade of our jenkins container. Workflow runs to upgrade our jenkins and other docker containers. At the initial part of workflow older versions of plugins are running, the list of versions i here below.   At the ansible playbook XXX-ku-upgrade.yml then the old jenkins container is stopped And new jenkins container started with new plugins (and same jenkins home) ... And the workflow resumes .. (but hits the issue quickly and fails). Normally the workflow would resume and run other tasks to upgrade the full system - all docker containers of different types.   Thanks very much markewaite for working on this already! I hope this describes it better. This workflow across jenkins upgrade is how we keep our jenkins up to date.   VERSIONS FROM jenkins BEFORE upgrade:   Jenkins: 2.492.1 OS: Linux - 3.10.0-1160.11.1.el7.x86_64 Java: 17.0.14 - Red Hat, Inc. (OpenJDK 64-Bit Server VM) — ansible:500.v7564a_db_8feec ansicolor:1.0.6 antisamy-markup-formatter:173.v680e3a_b_69ff3 apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83 asm-api:9.7.1-97.v4cc844130d97 badge:2.7 bootstrap5-api:5.3.3-2 bouncycastle-api:2.30.1.80-256.vf98926042a_9b_ branch-api:2.1214.v3f652804588d build-monitor-plugin:1.14-961.v676e38a_7a_248 build-pipeline-plugin:2.0.2 buildresult-trigger:1.01 buildtriggerbadge:251.vdf6ef853f3f5 caffeine-api:3.2.0-161.v691ef352cee1 checks-api:2.2.3 cloudbees-disk-usage-simple:232.v713eeed2e1f4 cloudbees-folder:6.982.vf165a_16c9507 command-launcher:118.v72741845c17a_ commons-compress-api:1.27.1-2 commons-lang3-api:3.17.0-84.vb_b_938040b_078 commons-text-api:1.13.0-153.v91dcd89e2a_22 conditional-buildstep:1.5.0 credentials:1408.va_622a_b_f5b_1b_1 credentials-binding:687.v619cb_15e923f cucumber-reports:5.8.5 cvs:471.v28a_41b_03ee4f dashboard-view:2.528.v3470c02b_d7c9 data-tables-api:2.2.2-1 disable-job-button:1.v9db_352414f90 display-url-api:2.209.v582ed814ff2f durable-task:581.v299a_5609d767 echarts-api:5.6.0-1 eddsa-api:0.3.0-13.v7cb_69ed68f00 emoji-symbols-api:16.0-26.v9818ff7423f0 envinject-api:1.235.va_14c74f8f487 flatpickr-api:4.6.13-15.vf6e534338831 font-awesome-api:6.7.2-1 fstrigger:1.02 groovy:497.v7b_061a_a_de65d groovy-postbuild:272.v52a_03efb_8653 gson-api:2.12.1-113.v347686d6729f handlebars:3.0.8 instance-identity:201.vd2a_b_5a_468a_a_6 ionicons-api:82.v0597178874e1 jackson2-api:2.17.0-389.va_5c7e45cd806 jakarta-activation-api:2.1.3-2 jakarta-mail-api:2.1.3-2 javadoc:310.v032f3f16b_0f8 javax-activation-api:1.2.0-8 javax-mail-api:1.6.2-11 jaxb:2.3.9-1 jdk-tool:83.v417146707a_3d joda-time-api:2.13.1-115.va_6b_5f8efb_1d8 jquery:1.12.4-3 jquery3-api:3.7.1-3 jsch:0.2.16-86.v42e010d9484b_ json-api:20250107-125.v28b_a_ffa_eb_f01 json-path-api:2.9.0-148.v22a_7ffe323ce junit:1317.v5b_35d792b_06a_ locale:566.v3323643741cc lockable-resources:1301.v0e3b_da_4b_4462 log-parser:2.3.7 mailer:489.vd4b_25144138f matrix-auth:3.2.4 matrix-project:845.vffd7fa_f27555 maven-plugin:3.25 metrics:4.2.21-464.vc9fa_a_0d6265d mina-sshd-api-common:2.14.0-143.v2b_362fc39576 mina-sshd-api-core:2.14.0-143.v2b_362fc39576 momentjs:1.1.1 monitoring:2.5.0 oss-symbols-api:296.v4981240eeb_1a_ parameterized-trigger:840.v3c7d4a_a_5e6c7 pipeline-build-step:555.v589d5c24a_3d6 pipeline-graph-analysis:231.v56354571a_da_0 pipeline-groovy-lib:752.vdddedf804e72 pipeline-input-step:513.v7eb_fe7e2c0fc pipeline-milestone-step:127.vb_52887ca_3b_6d pipeline-model-api:2.2221.vc657003fb_d93 pipeline-model-definition:2.2221.vc657003fb_d93 pipeline-model-extensions:2.2221.vc657003fb_d93 pipeline-rest-api:2.37 pipeline-stage-step:322.vecffa_99f371c pipeline-stage-tags-metadata:2.2221.vc657003fb_d93 pipeline-stage-view:2.37 pipeline-utility-steps:2.19.0 plain-credentials:183.va_de8f1dd5a_2b_ plugin-util-api:6.0.0 postbuild-task:72.v75dc459e4747 postbuildscript:3.4.1-695.vf6b_0b_8053979 prometheus:819.v50953a_c560dd rich-text-publisher-plugin:1.5 run-condition:243.v3c3f94e46a_8b_ schedule-build:643.v58b_09e4b_748c scm-api:703.v72ff4b_259600 script-security:1373.vb_b_4a_a_c26fa_00 selected-tests-executor:1.3.3 simple-theme-plugin:202.v6367d3dea_73b_ snakeyaml-api:2.3-123.v13484c65210a_ ssh-credentials:349.vb_8b_6b_9709f5b_ sshd:3.353.v2b_d33c46e970 structs:343.vdcf37b_a_c81d5 support-core:1637.v6a_6b_f31d4373 test-results-analyzer:0.4.1 text-finder:1.31 thinBackup:2.1.2 token-macro:444.v52de7e9c573d trilead-api:2.190.v1ed19f8137f8 uno-choice:2.8.6 variant:70.va_d9f17f859e0 workflow-aggregator:600.vb_57cdd26fdd7 workflow-api:1363.v03f731255494 workflow-basic-steps:1079.vce64b_a_929c5a_ workflow-cps:4018.vf02e01888da_f workflow-durable-task-step:1405.v1fcd4a_d00096 workflow-job:1505.vea_4b_20a_4a_495 workflow-multibranch:800.v5f0a_a_660950e workflow-scm-step:437.v05a_f66b_e5ef8 workflow-step-api:700.v6e45cb_a_5a_a_21 workflow-support:961.v51869f7b_d409 xtrigger-api:1.2  

          Mark Waite added a comment -

          Also, Declarative builds running across the upgrade may fail. New builds should work

          I think you are encountering that case.

          Mark Waite added a comment - Also, Declarative builds running across the upgrade may fail. New builds should work I think you are encountering that case.

            Unassigned Unassigned
            jamesc0 James Coleman
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: