-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Base OS is Alma 9.6
Jenkins v2.516.3-lts-alpine (and builds) run via/on docker 28.4.0
Plugins:
analysis-model-api: 13.8.0-902.v26f80296f743; antisamy-markup-formatter: 173.v680e3a_b_69ff3; apache-httpcomponents-client-4-api: 4.5.14-269.vfa_2321039a_83; apache-httpcomponents-client-5-api: 5.5-170.v023de017ccd7; asm-api: 9.8-163.vb_2a_96d3f9c3c; authentication-tokens: 1.144.v5ff4a_5ec5c33; bootstrap5-api: 5.3.8-876.vb_c62a_27d9a_77; bouncycastle-api: 2.30.1.81-264.v95c79c0e772c; branch-api: 2.1253.v6e7f7519f710; caffeine-api: 3.2.2-178.v353b_8428ed56; checks-api: 373.vfe7645102093; cloudbees-folder: 6.1042.v37b_8229708e4; commons-collections4-api: 4.5.0-8.va_d5448ef9011; commons-compress-api: 1.28.0-1; commons-lang3-api: 3.18.0-98.v3a_674c06072d; commons-text-api: 1.14.0-194.v804a_dc3a_1b_d8; configuration-as-code: 1985.vdda_32d0c4ea_b_; credentials: 1447.v4cb_b_539b_5321; credentials-binding: 702.vfe613e537e88; data-tables-api: 2.3.4-1400.vb_1e3e3c4dfc8; display-url-api: 2.217.va_6b_de84cc74b_; docker-commons: 457.v0f62a_94f11a_3; docker-java-api: 3.5.3-122.v156e51f30c0a_; docker-workflow: 621.va_73f881d9232; durable-task: 605.v9a_b_9040c9970; echarts-api: 6.0.0-1146.v5c8f3b_8f0573; eddsa-api: 0.3.0.1-19.vc432d923e5ee; font-awesome-api: 7.0.1-859.v128d3a_efb_6e5; forensics-api: 3.1754.v2a_6613b_77002; git: 5.7.0; git-client: 6.4.0; gson-api: 2.13.2-173.va_a_092315913c; htmlpublisher: 427; instance-identity: 203.v15e81a_1b_7a_38; ionicons-api: 94.vcc3065403257; jackson2-api: 2.20.0-411.v6ef8fdee4fe9; jakarta-activation-api: 2.1.3-2; jakarta-mail-api: 2.1.3-3; javax-activation-api: 1.2.0-8; javax-mail-api: 1.6.2-11; jaxb: 2.3.9-133.vb_ec76a_73f706; jersey2-api: 2.47-165.ve7809a_3e87e0; jira: 3.19; joda-time-api: 2.14.0-149.v1c3ce991d1b_9; jquery3-api: 3.7.1-594.vb_3864f326cf0; json-api: 20250517-173.v596efb_962a_31; json-path-api: 2.9.0-190.veefca_05d5477; jsoup: 1.21.2-66.v6ea_38164b_8a_2; junit: 1355.v45e2ea_65863c; mailer: 522.va_995fa_cfb_8b_d; matrix-auth: 3.2.8; matrix-project: 858.vb_b_eb_9a_7ea_99e; metrics: 4.2.33-484.v2fcd689980d1; mina-sshd-api-common: 2.16.0-167.va_269f38cc024; mina-sshd-api-core: 2.16.0-167.va_269f38cc024; pipeline-build-step: 571.v08a_fffd4b_0ce; pipeline-graph-analysis: 245.v88f03631a_b_21; pipeline-graph-view: 642.v39f37c8e1e70; pipeline-groovy-lib: 763.v13008816b_de7; pipeline-input-step: 534.v352f0a_e98918; pipeline-milestone-step: 138.v78ca_76831a_43; pipeline-model-api: 2.2273.v643f36ed9e94; pipeline-model-definition: 2.2273.v643f36ed9e94; pipeline-model-extensions: 2.2273.v643f36ed9e94; pipeline-stage-step: 322.vecffa_99f371c; pipeline-stage-tags-metadata: 2.2273.v643f36ed9e94; plain-credentials: 199.v9f8e1f741799; plugin-util-api: 6.1167.v022176c7e0ca_; PrioritySorter: 863.v4a_b_974a_5d042; prism-api: 1.30.0-609.vf0a_df102d9a_f; saml: 4.583.vc68232f7018a_; scm-api: 707.v749f968369d4; script-security: 1378.vf25626395f49; sloccount: 1.27; snakeyaml-api: 2.3-125.v4d77857a_b_402; ssh-credentials: 361.vb_f6760818e8c; sshd: 3.374.v19b_d59ce6610; ssh-slaves: 3.1071.v0d059c7b_c555; stashNotifier: 1.528.v9637674a_673f; structs: 353.v261ea_40a_80fb_; token-macro: 477.vd4f0dc3cb_cf1; trilead-api: 2.209.v0e69b_c43c245; variant: 70.va_d9f17f859e0; warnings-ng: 12.9783.ve1cb_9f060738; workflow-aggregator: 608.v67378e9d3db_1; workflow-api: 1384.vdc05a_48f535f; workflow-basic-steps: 1079.vce64b_a_929c5a_; workflow-cps: 4183.v94b_6fd39da_c1; workflow-durable-task-step: 1464.v2d3f5c68f84c; workflow-job: 1546.v62a_c59c112dd; workflow-multibranch: 821.vc3b_4ea_780798; workflow-scm-step: 452.vdf1ca_c8d3a_87; workflow-step-api: 706.v518c5dcb_24c0; workflow-support: 989.va_20a_1a_57710a_
Base OS is Alma 9.6 Jenkins v2.516.3-lts-alpine (and builds) run via/on docker 28.4.0 Plugins: analysis-model-api: 13.8.0-902.v26f80296f743; antisamy-markup-formatter: 173.v680e3a_b_69ff3; apache-httpcomponents-client-4-api: 4.5.14-269.vfa_2321039a_83; apache-httpcomponents-client-5-api: 5.5-170.v023de017ccd7; asm-api: 9.8-163.vb_2a_96d3f9c3c; authentication-tokens: 1.144.v5ff4a_5ec5c33; bootstrap5-api: 5.3.8-876.vb_c62a_27d9a_77; bouncycastle-api: 2.30.1.81-264.v95c79c0e772c; branch-api: 2.1253.v6e7f7519f710; caffeine-api: 3.2.2-178.v353b_8428ed56; checks-api: 373.vfe7645102093; cloudbees-folder: 6.1042.v37b_8229708e4; commons-collections4-api: 4.5.0-8.va_d5448ef9011; commons-compress-api: 1.28.0-1; commons-lang3-api: 3.18.0-98.v3a_674c06072d; commons-text-api: 1.14.0-194.v804a_dc3a_1b_d8; configuration-as-code: 1985.vdda_32d0c4ea_b_; credentials: 1447.v4cb_b_539b_5321; credentials-binding: 702.vfe613e537e88; data-tables-api: 2.3.4-1400.vb_1e3e3c4dfc8; display-url-api: 2.217.va_6b_de84cc74b_; docker-commons: 457.v0f62a_94f11a_3; docker-java-api: 3.5.3-122.v156e51f30c0a_; docker-workflow: 621.va_73f881d9232; durable-task: 605.v9a_b_9040c9970; echarts-api: 6.0.0-1146.v5c8f3b_8f0573; eddsa-api: 0.3.0.1-19.vc432d923e5ee; font-awesome-api: 7.0.1-859.v128d3a_efb_6e5; forensics-api: 3.1754.v2a_6613b_77002; git: 5.7.0; git-client: 6.4.0; gson-api: 2.13.2-173.va_a_092315913c; htmlpublisher: 427; instance-identity: 203.v15e81a_1b_7a_38; ionicons-api: 94.vcc3065403257; jackson2-api: 2.20.0-411.v6ef8fdee4fe9; jakarta-activation-api: 2.1.3-2; jakarta-mail-api: 2.1.3-3; javax-activation-api: 1.2.0-8; javax-mail-api: 1.6.2-11; jaxb: 2.3.9-133.vb_ec76a_73f706; jersey2-api: 2.47-165.ve7809a_3e87e0; jira: 3.19; joda-time-api: 2.14.0-149.v1c3ce991d1b_9; jquery3-api: 3.7.1-594.vb_3864f326cf0; json-api: 20250517-173.v596efb_962a_31; json-path-api: 2.9.0-190.veefca_05d5477; jsoup: 1.21.2-66.v6ea_38164b_8a_2; junit: 1355.v45e2ea_65863c; mailer: 522.va_995fa_cfb_8b_d; matrix-auth: 3.2.8; matrix-project: 858.vb_b_eb_9a_7ea_99e; metrics: 4.2.33-484.v2fcd689980d1; mina-sshd-api-common: 2.16.0-167.va_269f38cc024; mina-sshd-api-core: 2.16.0-167.va_269f38cc024; pipeline-build-step: 571.v08a_fffd4b_0ce; pipeline-graph-analysis: 245.v88f03631a_b_21; pipeline-graph-view: 642.v39f37c8e1e70; pipeline-groovy-lib: 763.v13008816b_de7; pipeline-input-step: 534.v352f0a_e98918; pipeline-milestone-step: 138.v78ca_76831a_43; pipeline-model-api: 2.2273.v643f36ed9e94; pipeline-model-definition: 2.2273.v643f36ed9e94; pipeline-model-extensions: 2.2273.v643f36ed9e94; pipeline-stage-step: 322.vecffa_99f371c; pipeline-stage-tags-metadata: 2.2273.v643f36ed9e94; plain-credentials: 199.v9f8e1f741799; plugin-util-api: 6.1167.v022176c7e0ca_; PrioritySorter: 863.v4a_b_974a_5d042; prism-api: 1.30.0-609.vf0a_df102d9a_f; saml: 4.583.vc68232f7018a_; scm-api: 707.v749f968369d4; script-security: 1378.vf25626395f49; sloccount: 1.27; snakeyaml-api: 2.3-125.v4d77857a_b_402; ssh-credentials: 361.vb_f6760818e8c; sshd: 3.374.v19b_d59ce6610; ssh-slaves: 3.1071.v0d059c7b_c555; stashNotifier: 1.528.v9637674a_673f; structs: 353.v261ea_40a_80fb_; token-macro: 477.vd4f0dc3cb_cf1; trilead-api: 2.209.v0e69b_c43c245; variant: 70.va_d9f17f859e0; warnings-ng: 12.9783.ve1cb_9f060738; workflow-aggregator: 608.v67378e9d3db_1; workflow-api: 1384.vdc05a_48f535f; workflow-basic-steps: 1079.vce64b_a_929c5a_; workflow-cps: 4183.v94b_6fd39da_c1; workflow-durable-task-step: 1464.v2d3f5c68f84c; workflow-job: 1546.v62a_c59c112dd; workflow-multibranch: 821.vc3b_4ea_780798; workflow-scm-step: 452.vdf1ca_c8d3a_87; workflow-step-api: 706.v518c5dcb_24c0; workflow-support: 989.va_20a_1a_57710a_
We periodically experience build failures due to what looks like build node comms issues. However, as far as we can tell the build nodes remain up and accessible. We run a jenkins controller via docker and have a number of build nodes (mix of vms and metal) registered. The agent runs as a systemd service. Given that a build can take upwards of 3h due to complexity and number of components being built, failures at the end (eg archiving artifacts) are inconvenient as there is no way to resume (that we know of) and the build process just needs to be restarted.
Attached are two stack traces showing two different failures that we recently experienced. At this point we are not sure what to look at next to try and track this down as it is intermittent. Some times it will happen with frequency other times, it might be days before we see it again so we do not have a consistent trigger either.
jenkins-stack1.txt – unclear what to look at next
jenkins-stack2.txt + jenkins-build3.txt – looks like a timeout but unclear what would trigger this. no firewalls should be in the path between these nodes.
