[JENKINS-70267] Internal pipeline crashes leave stuck executors

Type: Bug
Resolution: Duplicate
Priority: Critical
Component/s: workflow-cps-plugin, workflow-job-plugin
Labels:
None
Environment:

Hide
Jenkins: 2.361.4
OS: Linux - 4.15.0-200-generic
---
ace-editor:1.1
active-directory:2.29
analysis-model-api:10.21.0
ant:481.v7b_09e538fcca
antisamy-markup-formatter:155.v795fb_8702324
apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61
artifact-manager-s3:670.v0558a_cb_c82c2
artifactory:3.17.4
authentication-tokens:1.4
aws-credentials:191.vcb_f183ce58b_9
aws-global-configuration:106.v106dc1d8d86e
aws-java-sdk:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-cloudformation:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-codebuild:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ec2:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ecr:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ecs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-efs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-elasticbeanstalk:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-iam:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-logs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-minimal:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-sns:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-sqs:1.12.287-357.vf82d85a_6eefd
aws-java-sdk-ssm:1.12.287-357.vf82d85a_6eefd
blueocean:1.26.0
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.26.0
blueocean-commons:1.26.0
blueocean-config:1.26.0
blueocean-core-js:1.26.0
blueocean-dashboard:1.26.0
blueocean-display-url:2.4.1
blueocean-events:1.26.0
blueocean-git-pipeline:1.26.0
blueocean-github-pipeline:1.26.0
blueocean-i18n:1.26.0
blueocean-jira:1.26.0
blueocean-jwt:1.26.0
blueocean-personalization:1.26.0
blueocean-pipeline-api-impl:1.26.0
blueocean-pipeline-editor:1.26.0
blueocean-pipeline-scm-api:1.26.0
blueocean-rest:1.26.0
blueocean-rest-impl:1.26.0
blueocean-web:1.26.0
bootstrap4-api:4.6.0-5
bootstrap5-api:5.2.1-3
bouncycastle-api:2.26
branch-api:2.1051.v9985666b_f6cc
build-timeout:1.25
buildtriggerbadge:251.vdf6ef853f3f5
built-on-column:1.3
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:1.8.0
cloudbees-bitbucket-branch-source:791.vb_eea_a_476405b
cloudbees-folder:6.758.vfd75d09eea_a_1
cobertura:1.17
code-coverage-api:3.4.0
command-launcher:90.v669d7ccb_7c31
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-27.vb_fa_3896786a_7
conditional-buildstep:1.4.2
config-file-provider:3.11.1
configuration-as-code:1569.vb_72405b_80249
copyartifact:1.48
credentials:1189.vf61b_a_5e2f62e
credentials-binding:523.vd859a_4b_122e6
cron_column:1.4
cvs:2.19.1
data-tables-api:1.12.1-4
dependency-check-jenkins-plugin:5.2.1
dependency-track:4.2.0
display-url-api:2.3.7
docker-commons:1.21
docker-workflow:563.vd5d2e5c4007f
durable-task:503.v57154d18d478
echarts-api:5.4.0-1
email-ext:2.92
emailext-template:1.5
envinject:2.892.v25453b_80e595
envinject-api:1.199.v3ce31253ed13
external-monitor-job:203.v683c09d993b_9
favorite:2.4.1
font-awesome-api:6.2.1-1
forensics-api:1.16.0
git:4.14.3
git-client:3.13.1
git-parameter:0.9.18
git-server:99.va_0826a_b_cdfa_d
github:1.36.0
github-api:1.303-400.v35c2d8258028
github-branch-source:1696.v3a_7603564d04
gitlab-api:5.0.1-78.v47a_45b_9f78b_7
gitlab-branch-source:643.vdc12a_4a_06434
gitlab-plugin:1.6.0
gradle:2.1.1
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.31
instance-identity:142.v04572ca_5b_265
ionicons-api:31.v4757b_6987003
ivy:2.4
jackson2-api:2.14.1-313.v504cdd45c18b
jakarta-activation-api:2.0.1-2
jakarta-mail-api:2.0.1-2
javadoc:226.v71211feb_e7e9
javax-activation-api:1.2.0-5
javax-mail-api:1.6.2-8
jaxb:2.3.7-1
jdk-tool:63.v62d2fd4b_4793
jenkins-design-language:1.26.0
jenkins-multijob-plugin:623.v03401733c9a_9
jersey2-api:2.37-1
jira:3.8
jjwt-api:0.11.5-77.v646c772fddb_0
jnr-posix-api:3.1.16-1
jobConfigHistory:1176.v1b_4290db_41a_5
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.6.1-2
jsch:0.1.55.61.va_e9ee26616e7
junit:1166.va_436e268e972
kerberos-sso:1.10
kubernetes:3743.v1fa_4c724c3b_7
kubernetes-client-api:5.12.2-193.v26a_6078f65a_9
kubernetes-credentials:0.9.0
ldap:2.12
lockable-resources:1069.v726298f53f8c
mailer:438.v02c7f0a_12fa_4
mapdb-api:1.0.9-28.vf251ce40855d
matrix-auth:3.1.6
matrix-project:785.v06b_7f47b_c631
maven-plugin:3.20
mercurial:1260.vdfb_723cdcc81
metrics:4.2.13-420.vea_2f17932dd6
mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a
mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a
minio:1.3.3-rc93.9e92f846d4cf
momentjs:1.1.1
monitoring:1.91.0
notification:1.15
nsiqcollector:1.3.3
oic-auth:2.3
okhttp-api:4.9.3-108.v0feda04578cf
pam-auth:1.10
parameterized-trigger:2.45
pipeline-build-step:2.18
pipeline-github-lib:38.v445716ea_edda_
pipeline-graph-analysis:195.v5812d95a_a_2f9
pipeline-groovy-lib:621.vb_44ce045b_582
pipeline-input-step:466.v6d0a_5df34f81
pipeline-milestone-step:101.vd572fef9d926
pipeline-model-api:2.2118.v31fd5b_9944b_5
pipeline-model-definition:2.2118.v31fd5b_9944b_5
pipeline-model-extensions:2.2118.v31fd5b_9944b_5
pipeline-rest-api:2.28
pipeline-stage-step:296.v5f6908f017a_5
pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5
pipeline-stage-view:2.28
pipeline-utility-steps:2.14.0
plain-credentials:139.ved2b_9cf7587b
plugin-usage-plugin:3.0
plugin-util-api:2.20.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
prism-api:1.29.0-1
pubsub-light:1.17
resource-disposer:0.20
reverse-proxy-auth-plugin:1.7.4
run-condition:1.5
s3:0.12.3436.v674b_46258039
saml:4.385.v4dea_91565e9d
scm-api:621.vda_a_b_055e58f7
script-security:1228.vd93135a_2fb_25
snakeyaml-api:1.33-90.v80dcb_3814d35
sse-gateway:1.26
ssh-agent:295.v9ca_a_1c7cc3a_a_
ssh-credentials:305.v8f4381501156
ssh-slaves:2.854.v7fd446b_337c9
sshd:3.270.vb_a_e71e64c287
structs:324.va_f5d6774f3a_d
subversion:2.16.0-SICKAG
test-results-aggregator:1.2.10
test-results-analyzer:0.4.0
test-stability:2.3
timestamper:1.21
token-macro:321.vd7cc1f2a_52c8
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
warnings-ng:9.21.0
windows-slaves:1.8.1
workflow-aggregator:590.v6a_d052e5a_a_b_5
workflow-api:1200.v8005c684b_a_c6
workflow-basic-steps:994.vd57e3ca_46d24
workflow-cps:3565.v4b_d9b_8c29a_b_3
workflow-cps-global-lib:609.vd95673f149b_b
workflow-durable-task-step:1217.v38306d8fa_b_5c
workflow-job:1254.v3f64639b_11dd
workflow-multibranch:716.vc692a_e52371b_
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.44

Show
Jenkins: 2.361.4 OS: Linux - 4.15.0-200-generic --- ace-editor:1.1 active-directory:2.29 analysis-model-api:10.21.0 ant:481.v7b_09e538fcca antisamy-markup-formatter:155.v795fb_8702324 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 artifact-manager-s3:670.v0558a_cb_c82c2 artifactory:3.17.4 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-global-configuration:106.v106dc1d8d86e aws-java-sdk:1.12.287-357.vf82d85a_6eefd aws-java-sdk-cloudformation:1.12.287-357.vf82d85a_6eefd aws-java-sdk-codebuild:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ec2:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ecr:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ecs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-efs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-elasticbeanstalk:1.12.287-357.vf82d85a_6eefd aws-java-sdk-iam:1.12.287-357.vf82d85a_6eefd aws-java-sdk-logs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-minimal:1.12.287-357.vf82d85a_6eefd aws-java-sdk-sns:1.12.287-357.vf82d85a_6eefd aws-java-sdk-sqs:1.12.287-357.vf82d85a_6eefd aws-java-sdk-ssm:1.12.287-357.vf82d85a_6eefd blueocean:1.26.0 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.26.0 blueocean-commons:1.26.0 blueocean-config:1.26.0 blueocean-core-js:1.26.0 blueocean-dashboard:1.26.0 blueocean-display-url:2.4.1 blueocean-events:1.26.0 blueocean-git-pipeline:1.26.0 blueocean-github-pipeline:1.26.0 blueocean-i18n:1.26.0 blueocean-jira:1.26.0 blueocean-jwt:1.26.0 blueocean-personalization:1.26.0 blueocean-pipeline-api-impl:1.26.0 blueocean-pipeline-editor:1.26.0 blueocean-pipeline-scm-api:1.26.0 blueocean-rest:1.26.0 blueocean-rest-impl:1.26.0 blueocean-web:1.26.0 bootstrap4-api:4.6.0-5 bootstrap5-api:5.2.1-3 bouncycastle-api:2.26 branch-api:2.1051.v9985666b_f6cc build-timeout:1.25 buildtriggerbadge:251.vdf6ef853f3f5 built-on-column:1.3 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.8.0 cloudbees-bitbucket-branch-source:791.vb_eea_a_476405b cloudbees-folder:6.758.vfd75d09eea_a_1 cobertura:1.17 code-coverage-api:3.4.0 command-launcher:90.v669d7ccb_7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-27.vb_fa_3896786a_7 conditional-buildstep:1.4.2 config-file-provider:3.11.1 configuration-as-code:1569.vb_72405b_80249 copyartifact:1.48 credentials:1189.vf61b_a_5e2f62e credentials-binding:523.vd859a_4b_122e6 cron_column:1.4 cvs:2.19.1 data-tables-api:1.12.1-4 dependency-check-jenkins-plugin:5.2.1 dependency-track:4.2.0 display-url-api:2.3.7 docker-commons:1.21 docker-workflow:563.vd5d2e5c4007f durable-task:503.v57154d18d478 echarts-api:5.4.0-1 email-ext:2.92 emailext-template:1.5 envinject:2.892.v25453b_80e595 envinject-api:1.199.v3ce31253ed13 external-monitor-job:203.v683c09d993b_9 favorite:2.4.1 font-awesome-api:6.2.1-1 forensics-api:1.16.0 git:4.14.3 git-client:3.13.1 git-parameter:0.9.18 git-server:99.va_0826a_b_cdfa_d github:1.36.0 github-api:1.303-400.v35c2d8258028 github-branch-source:1696.v3a_7603564d04 gitlab-api:5.0.1-78.v47a_45b_9f78b_7 gitlab-branch-source:643.vdc12a_4a_06434 gitlab-plugin:1.6.0 gradle:2.1.1 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.31 instance-identity:142.v04572ca_5b_265 ionicons-api:31.v4757b_6987003 ivy:2.4 jackson2-api:2.14.1-313.v504cdd45c18b jakarta-activation-api:2.0.1-2 jakarta-mail-api:2.0.1-2 javadoc:226.v71211feb_e7e9 javax-activation-api:1.2.0-5 javax-mail-api:1.6.2-8 jaxb:2.3.7-1 jdk-tool:63.v62d2fd4b_4793 jenkins-design-language:1.26.0 jenkins-multijob-plugin:623.v03401733c9a_9 jersey2-api:2.37-1 jira:3.8 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.16-1 jobConfigHistory:1176.v1b_4290db_41a_5 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.1-2 jsch:0.1.55.61.va_e9ee26616e7 junit:1166.va_436e268e972 kerberos-sso:1.10 kubernetes:3743.v1fa_4c724c3b_7 kubernetes-client-api:5.12.2-193.v26a_6078f65a_9 kubernetes-credentials:0.9.0 ldap:2.12 lockable-resources:1069.v726298f53f8c mailer:438.v02c7f0a_12fa_4 mapdb-api:1.0.9-28.vf251ce40855d matrix-auth:3.1.6 matrix-project:785.v06b_7f47b_c631 maven-plugin:3.20 mercurial:1260.vdfb_723cdcc81 metrics:4.2.13-420.vea_2f17932dd6 mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a minio:1.3.3-rc93.9e92f846d4cf momentjs:1.1.1 monitoring:1.91.0 notification:1.15 nsiqcollector:1.3.3 oic-auth:2.3 okhttp-api:4.9.3-108.v0feda04578cf pam-auth:1.10 parameterized-trigger:2.45 pipeline-build-step:2.18 pipeline-github-lib:38.v445716ea_edda_ pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:621.vb_44ce045b_582 pipeline-input-step:466.v6d0a_5df34f81 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2118.v31fd5b_9944b_5 pipeline-model-definition:2.2118.v31fd5b_9944b_5 pipeline-model-extensions:2.2118.v31fd5b_9944b_5 pipeline-rest-api:2.28 pipeline-stage-step:296.v5f6908f017a_5 pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5 pipeline-stage-view:2.28 pipeline-utility-steps:2.14.0 plain-credentials:139.ved2b_9cf7587b plugin-usage-plugin:3.0 plugin-util-api:2.20.0 popper-api:1.16.1-3 popper2-api:2.11.6-2 prism-api:1.29.0-1 pubsub-light:1.17 resource-disposer:0.20 reverse-proxy-auth-plugin:1.7.4 run-condition:1.5 s3:0.12.3436.v674b_46258039 saml:4.385.v4dea_91565e9d scm-api:621.vda_a_b_055e58f7 script-security:1228.vd93135a_2fb_25 snakeyaml-api:1.33-90.v80dcb_3814d35 sse-gateway:1.26 ssh-agent:295.v9ca_a_1c7cc3a_a_ ssh-credentials:305.v8f4381501156 ssh-slaves:2.854.v7fd446b_337c9 sshd:3.270.vb_a_e71e64c287 structs:324.va_f5d6774f3a_d subversion:2.16.0-SICKAG test-results-aggregator:1.2.10 test-results-analyzer:0.4.0 test-stability:2.3 timestamper:1.21 token-macro:321.vd7cc1f2a_52c8 trilead-api:2.84.v72119de229b_7 variant:59.vf075fe829ccb warnings-ng:9.21.0 windows-slaves:1.8.1 workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1200.v8005c684b_a_c6 workflow-basic-steps:994.vd57e3ca_46d24 workflow-cps:3565.v4b_d9b_8c29a_b_3 workflow-cps-global-lib:609.vd95673f149b_b workflow-durable-task-step:1217.v38306d8fa_b_5c workflow-job:1254.v3f64639b_11dd workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.44

Similar Issues:
Powered by SuggestiMate

Show

If a pipeline crashes due to some unexpected internal errors, jobs get stuck and block the executors from running other things.

I have built the following minimal example for a pipeline (Taken from another issue: ~~JENKINS-70080~~. This is not the main problem, just an easy way to cause the issue described later):

class TestRunner {
  private def testCases = []

  @NonCPS
  void add(newTests) {
    testCases += newTests
  }
}


node('test'){
    stage('a'){
        script {
            def t = new TestRunner()
        }
        echo "Hello"   
    }
}

This code produces the following crash:

12:33:04  [Pipeline] Start of Pipeline
12:33:04  [Pipeline] node
12:33:05  Running on test in /home/jenkins/workspace/test pipeline
12:33:05  [Pipeline] End of Pipeline
12:33:05  java.lang.VerifyError: (class: TestRunner, method: add signature: (Ljava/lang/Object;)V) Unable to pop operand off an empty stack
12:33:05  	at java.base/java.lang.Class.getDeclaredMethods0(Native Method)
12:33:05  	at java.base/java.lang.Class.privateGetDeclaredMethods(Class.java:3166)
12:33:05  	at java.base/java.lang.Class.getDeclaredMethod(Class.java:2473)
12:33:05  	at java.base/jdk.internal.reflect.ReflectionFactory.findReadWriteObjectForSerialization(ReflectionFactory.java:556)
12:33:05  	at java.base/jdk.internal.reflect.ReflectionFactory.readObjectForSerialization(ReflectionFactory.java:537)
12:33:05  	at jdk.unsupported/sun.reflect.ReflectionFactory.readObjectForSerialization(ReflectionFactory.java:144)
12:33:05  	at org.jboss.marshalling.reflect.JDKSpecific$SerMethods.<init>(JDKSpecific.java:61)
12:33:05  	at org.jboss.marshalling.reflect.SerializableClass.<init>(SerializableClass.java:84)
12:33:05  	at org.jboss.marshalling.reflect.SerializableClassRegistry$1.computeValue(SerializableClassRegistry.java:62)
12:33:05  	at org.jboss.marshalling.reflect.SerializableClassRegistry$1.computeValue(SerializableClassRegistry.java:59)
12:33:05  	at java.base/java.lang.ClassValue.getFromHashMap(ClassValue.java:228)
12:33:05  	at java.base/java.lang.ClassValue.getFromBackup(ClassValue.java:210)
12:33:05  	at java.base/java.lang.ClassValue.get(ClassValue.java:116)
12:33:05  	at org.jboss.marshalling.reflect.SerializableClassRegistry.lookup(SerializableClassRegistry.java:83)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeNewSerializableClass(RiverMarshaller.java:1514)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeNewClass(RiverMarshaller.java:1417)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeClass(RiverMarshaller.java:1268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeClassClass(RiverMarshaller.java:1256)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:166)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeArrayObject(RiverMarshaller.java:312)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:222)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.writeArrayObject(RiverMarshaller.java:312)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:222)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1080)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1080)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:65)
12:33:05  	at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:56)
12:33:05  	at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:50)
12:33:05  	at org.jboss.marshalling.river.RiverObjectOutputStream.writeObjectOverride(RiverObjectOutputStream.java:179)
12:33:05  	at java.base/java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:341)
12:33:05  	at java.base/java.util.HashMap.internalWriteEntries(HashMap.java:1858)
12:33:05  	at java.base/java.util.HashMap.writeObject(HashMap.java:1412)
12:33:05  	at org.jboss.marshalling.reflect.JDKSpecific$SerMethods.callWriteObject(JDKSpecific.java:89)
12:33:05  	at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:199)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1089)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
12:33:05  	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
12:33:05  	at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
12:33:05  	at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:116)
12:33:05  	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.lambda$writeObject$1(RiverWriter.java:144)
12:33:05  	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:331)
12:33:05  	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverWriter.writeObject(RiverWriter.java:143)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:577)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgram(CpsThreadGroup.java:554)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.saveProgramIfPossible(CpsThreadGroup.java:537)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:461)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
12:33:05  	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
12:33:05  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
12:33:05  	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
12:33:05  	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:30)
12:33:05  	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:70)
12:33:05  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
12:33:05  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
12:33:05  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
12:33:05  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
12:33:05  	at java.base/java.lang.Thread.run(Thread.java:829)
12:33:05  Finished: FAILURE

The stack trace according to the monitoring looks like this:

Stack-trace
java.lang.IllegalStateException: trying to open a build log on test pipeline #53 after it has completed
      at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:232)
      at org.jenkinsci.plugins.workflow.job.WorkflowRun$NodePrintListener.onNewHead(WorkflowRun.java:1079)
      at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1556)
      at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.notifyNewHead(CpsThreadGroup.java:492)
      at org.jenkinsci.plugins.workflow.cps.FlowHead.setNewHead(FlowHead.java:158)
      at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.launch(CpsBodyExecution.java:125)
      at org.jenkinsci.plugins.workflow.cps.CpsBodyInvoker.launch(CpsBodyInvoker.java:188)
      at org.jenkinsci.plugins.workflow.cps.CpsBodyInvoker.launch(CpsBodyInvoker.java:183)
      at org.jenkinsci.plugins.workflow.cps.CpsBodyInvoker$1.onSuccess(CpsBodyInvoker.java:159)
      at org.jenkinsci.plugins.workflow.cps.CpsBodyInvoker$1.onSuccess(CpsBodyInvoker.java:154)
      at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$5$1.run(CpsFlowExecution.java:930)
      at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:38)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:30)
      at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:70)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      at java.base/java.lang.Thread.run(Thread.java:829)

After the crash, one of the executors for a given agent will be permanently blocked. There is no way of aborting the build via UI through the common means. While the red X button is there, it has no effect instead just prompts a popup saying:

Are you sure you want to abort null?

We were able to find a workaround by explicitly removing the executor from the computer via CLI:

// Example Code, partially based on https://github.com/cloudbees/jenkins-scripts/blob/master/ProperlyStopOnlyRunningPipelines.groovy

jenkins.model.Jenkins.instanceOrNull.getComputers().each { computer ->
  computer.executors.findAll { exec -> exec.isBusy() && exec.currentExecutable }.each { exec ->
    if (exec.currentExecutable.getFullDisplayName().contains("test pipeline")) {
      println "Stopping ${exec.currentExecutable.getFullDisplayName()}"
      computer.removeExecutor(exec)
    }
  }
}

This issue keeps reappearing in our systems within different build jobs. This example was based on our Test System, where we reproduced the issue. However we have the same error in production (see attached environment).

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

prod_environment.txt
6 kB
2022-12-13 12:22

duplicates

JENKINS-71692 Pipeline sometimes leaks Execution on heavyweight executors

Resolved

relates to

JENKINS-71692 Pipeline sometimes leaks Execution on heavyweight executors

Resolved

Marian Degel added a comment - 2023-01-30 08:16

We keep getting random crashes within the pipelines on occasions, which then continue to block our executors with non-removable jobs.

Some of our newer issues:

[2023-01-28T08:57:35.706Z] [Pipeline] }
[2023-01-28T08:57:35.709Z] [Pipeline] // timeout
[2023-01-28T08:57:35.713Z] [Pipeline] }
[2023-01-28T08:57:35.716Z] [Pipeline] // stage
[2023-01-28T08:57:35.720Z] [Pipeline] }
[2023-01-28T08:57:35.722Z] [Pipeline] End of Pipeline
[2023-01-28T08:57:35.744Z] java.lang.NullPointerException
[2023-01-28T08:57:35.744Z] 	at com.dabsquared.gitlabjenkins.workflow.GitLabCommitStatusStep.access$400(GitLabCommitStatusStep.java:34)
[2023-01-28T08:57:35.744Z] 	at com.dabsquared.gitlabjenkins.workflow.GitLabCommitStatusStep$GitLabCommitStatusStepExecution$1.onSuccess(GitLabCommitStatusStep.java:103)
[2023-01-28T08:57:35.744Z] 	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:372)
[2023-01-28T08:57:35.744Z] 	at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
[2023-01-28T08:57:35.744Z] 	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:155)
[2023-01-28T08:57:35.744Z] 	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:152)
[2023-01-28T08:57:35.744Z] 	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
[2023-01-28T08:57:35.744Z] 	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
[2023-01-28T08:57:35.744Z] 	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:152)
[2023-01-28T08:57:35.745Z] 	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
[2023-01-28T08:57:35.745Z] 	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
[2023-01-28T08:57:35.745Z] 	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
[2023-01-28T08:57:35.745Z] 	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:420)
[2023-01-28T08:57:35.745Z] 	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:95)
[2023-01-28T08:57:35.745Z] 	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
[2023-01-28T08:57:35.745Z] 	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
[2023-01-28T08:57:35.745Z] 	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
[2023-01-28T08:57:35.745Z] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2023-01-28T08:57:35.745Z] 	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
[2023-01-28T08:57:35.745Z] 	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
[2023-01-28T08:57:35.745Z] 	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
[2023-01-28T08:57:35.745Z] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[2023-01-28T08:57:35.745Z] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[2023-01-28T08:57:35.745Z] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[2023-01-28T08:57:35.745Z] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[2023-01-28T08:57:35.745Z] 	at java.base/java.lang.Thread.run(Thread.java:829)
[2023-01-28T08:57:35.755Z] Finished: FAILURE

It appears, as if Jenkins suddenly cannot handle any exceptions that originate from the pipeline script itself. This behavior is new in our opinion.

Marian Degel added a comment - 2023-01-30 08:16 We keep getting random crashes within the pipelines on occasions, which then continue to block our executors with non-removable jobs. Some of our newer issues: [2023-01-28T08:57:35.706Z] [Pipeline] } [2023-01-28T08:57:35.709Z] [Pipeline] // timeout [2023-01-28T08:57:35.713Z] [Pipeline] } [2023-01-28T08:57:35.716Z] [Pipeline] // stage [2023-01-28T08:57:35.720Z] [Pipeline] } [2023-01-28T08:57:35.722Z] [Pipeline] End of Pipeline [2023-01-28T08:57:35.744Z] java.lang.NullPointerException [2023-01-28T08:57:35.744Z] at com.dabsquared.gitlabjenkins.workflow.GitLabCommitStatusStep.access$400(GitLabCommitStatusStep.java:34) [2023-01-28T08:57:35.744Z] at com.dabsquared.gitlabjenkins.workflow.GitLabCommitStatusStep$GitLabCommitStatusStepExecution$1.onSuccess(GitLabCommitStatusStep.java:103) [2023-01-28T08:57:35.744Z] at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:372) [2023-01-28T08:57:35.744Z] at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73) [2023-01-28T08:57:35.744Z] at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:155) [2023-01-28T08:57:35.744Z] at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:152) [2023-01-28T08:57:35.744Z] at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136) [2023-01-28T08:57:35.744Z] at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275) [2023-01-28T08:57:35.744Z] at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:152) [2023-01-28T08:57:35.745Z] at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) [2023-01-28T08:57:35.745Z] at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) [2023-01-28T08:57:35.745Z] at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187) [2023-01-28T08:57:35.745Z] at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:420) [2023-01-28T08:57:35.745Z] at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:95) [2023-01-28T08:57:35.745Z] at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330) [2023-01-28T08:57:35.745Z] at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294) [2023-01-28T08:57:35.745Z] at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67) [2023-01-28T08:57:35.745Z] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [2023-01-28T08:57:35.745Z] at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139) [2023-01-28T08:57:35.745Z] at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) [2023-01-28T08:57:35.745Z] at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) [2023-01-28T08:57:35.745Z] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [2023-01-28T08:57:35.745Z] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [2023-01-28T08:57:35.745Z] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [2023-01-28T08:57:35.745Z] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [2023-01-28T08:57:35.745Z] at java.base/java.lang.Thread.run(Thread.java:829) [2023-01-28T08:57:35.755Z] Finished: FAILURE It appears, as if Jenkins suddenly cannot handle any exceptions that originate from the pipeline script itself. This behavior is new in our opinion.

Allan BURDAJEWICZ added a comment - 2023-08-29 11:27

degelma Is this still happening in recent version of the workflow plugin stack ? In particular since https://github.com/jenkinsci/workflow-job-plugin/releases/tag/1301.v054d9cea_9593 ?

Allan BURDAJEWICZ added a comment - 2023-08-29 11:27 degelma Is this still happening in recent version of the workflow plugin stack ? In particular since https://github.com/jenkinsci/workflow-job-plugin/releases/tag/1301.v054d9cea_9593 ?

Marian Degel added a comment - 2023-08-31 13:43

allan_burdajewicz: We have not been able to observe this behavior any longer.

It's hard to give a specific timepoint, as the issues were appearing infrequently and the only way easy way to reproduce them was our sample from ~~JENKINS-70080~~.
Thus, the main issues stopped for us around the time when ~~JENKINS-70080~~ was fixed, as this was the main culprit.

I think we had seen a few random occasions after that, but they also quickly stopped, which would coincide with your mentioned plugin patch.
We haven't seen the issue reappearing since then.

Marian Degel added a comment - 2023-08-31 13:43 allan_burdajewicz : We have not been able to observe this behavior any longer. It's hard to give a specific timepoint, as the issues were appearing infrequently and the only way easy way to reproduce them was our sample from JENKINS-70080 . Thus, the main issues stopped for us around the time when JENKINS-70080 was fixed, as this was the main culprit. I think we had seen a few random occasions after that, but they also quickly stopped, which would coincide with your mentioned plugin patch. We haven't seen the issue reappearing since then.

Devin Nusbaum added a comment - 2023-08-31 20:25 - edited

I wonder if my proposed fix for ~~JENKINS-71692~~ would also address this case, or if it is distinct. To check, we would have to roll back the fix associated with ~~JENKINS-70080~~ and then see if the reproducer above triggers CpsVmExecutorService.reportProblem (eventually) or really only happens during serialization. If the former is true, I think this can be closed as a duplicate of ~~JENKINS-71692~~, otherwise we would need to look into making changes so that exceptions here end up making their way to CpsVmExecutorService.reportProblem, at least in this case.

From a quick look I suspect that CpsThreadGroup.propagateToWorkflow instead needs to be treated as a fatal VM thread error that kills the build, or only logged while the Pipeline is allowed to continue running normally.

Devin Nusbaum added a comment - 2023-08-31 20:25 - edited I wonder if my proposed fix for JENKINS-71692 would also address this case, or if it is distinct. To check, we would have to roll back the fix associated with JENKINS-70080 and then see if the reproducer above triggers CpsVmExecutorService.reportProblem (eventually) or really only happens during serialization. If the former is true, I think this can be closed as a duplicate of JENKINS-71692 , otherwise we would need to look into making changes so that exceptions here end up making their way to CpsVmExecutorService.reportProblem , at least in this case. From a quick look I suspect that CpsThreadGroup.propagateToWorkflow instead needs to be treated as a fatal VM thread error that kills the build, or only logged while the Pipeline is allowed to continue running normally.

Devin Nusbaum added a comment - 2023-08-31 20:43

Oh and the NPE in GitLabCommitStatusStep is kind of similar, but looks like a distinct issue. https://github.com/jenkinsci/gitlab-plugin/pull/1410 should have fixed it (really the plugin could probably use a simpler API like BodyExecutionCallback.TailCall), but in general we should make CpsBodyExecution.onSuccess and CpsBodyExecution.onFailure robust against exceptions thrown by the callbacks.

Devin Nusbaum added a comment - 2023-08-31 20:43 Oh and the NPE in GitLabCommitStatusStep is kind of similar, but looks like a distinct issue. https://github.com/jenkinsci/gitlab-plugin/pull/1410 should have fixed it (really the plugin could probably use a simpler API like BodyExecutionCallback.TailCall ), but in general we should make CpsBodyExecution.onSuccess and CpsBodyExecution.onFailure robust against exceptions thrown by the callbacks.

Devin Nusbaum added a comment - 2023-08-31 21:57

Ok, I was able to confirm that both the issue in the description and the one with the GitLab plugin BodyExecutionCallback are fixed by my proposed fix in ~~JENKINS-71692~~, so I am closing this as a duplicate of that issue. Thanks for your report degelma!

Note though my reproduction of the issue in the description required a slightly different approach: I had to load the class indirectly via load otherwise things failed before node even started. Unfortunately I do not think we can fully reproduce that issue in a test, at least not trivially, because you cannot simply throw a Throwable from methods like writeReplace or writeObject to get the same behavior, it has to come from a place where jboss-marshalling does not catch the exception and wrap it in a non-Throwable so we can bypass this logic. I do still think propagateErrorToWorkflow seems problematic, but as long as it allows all of the threads to keep running I guess it should work fine.

Devin Nusbaum added a comment - 2023-08-31 21:57 Ok, I was able to confirm that both the issue in the description and the one with the GitLab plugin BodyExecutionCallback are fixed by my proposed fix in JENKINS-71692 , so I am closing this as a duplicate of that issue. Thanks for your report degelma ! Note though my reproduction of the issue in the description required a slightly different approach: I had to load the class indirectly via load otherwise things failed before node even started. Unfortunately I do not think we can fully reproduce that issue in a test, at least not trivially, because you cannot simply throw a Throwable from methods like writeReplace or writeObject to get the same behavior, it has to come from a place where jboss-marshalling does not catch the exception and wrap it in a non- Throwable so we can bypass this logic . I do still think propagateErrorToWorkflow seems problematic, but as long as it allows all of the threads to keep running I guess it should work fine.

Details

Description

Attachments

Attachments

Issue Links

Activity

Collapse comment: Marian Degel added a comment - 2023-01-30 08:16

Expand comment: Marian Degel added a comment - 2023-01-30 08:16

Collapse comment: Allan BURDAJEWICZ added a comment - 2023-08-29 11:27

Expand comment: Allan BURDAJEWICZ added a comment - 2023-08-29 11:27

Collapse comment: Marian Degel added a comment - 2023-08-31 13:43

Expand comment: Marian Degel added a comment - 2023-08-31 13:43

Collapse comment: Devin Nusbaum added a comment - 2023-08-31 20:25, Edited by Devin Nusbaum - 2023-08-31 20:25

Expand comment: Devin Nusbaum added a comment - 2023-08-31 20:25, Edited by Devin Nusbaum - 2023-08-31 20:25

Collapse comment: Devin Nusbaum added a comment - 2023-08-31 20:43

Expand comment: Devin Nusbaum added a comment - 2023-08-31 20:43

Collapse comment: Devin Nusbaum added a comment - 2023-08-31 21:57

Expand comment: Devin Nusbaum added a comment - 2023-08-31 21:57

People

Dates