-
Bug
-
Resolution: Fixed
-
Blocker
-
Jenkins: 2.307
OS: Linux - 5.8.0-1040-azure
---
blueocean-commons:1.24.8
azure-keyvault:123.v1aba71c2d365
credentials:2.5
resource-disposer:0.16
jjwt-api:0.11.2-9.c8b45b8bb173
build-pipeline-plugin:1.5.8
pipeline-input-step:2.12
azure-commons:1.1.3
matrix-project:1.19
workflow-support:3.8
sshd:3.1.0
token-macro:266.v44a80cf277fd
font-awesome-api:5.15.3-4
conditional-buildstep:1.4.1
favorite:2.3.3
pipeline-stage-tags-metadata:1.9.1
matrix-auth:2.6.8
plain-credentials:1.7
lockable-resources:2.11
popper-api:1.16.1-2
workflow-multibranch:2.26
snakeyaml-api:1.29.1
htmlpublisher:1.25
blueocean-rest:1.24.8
momentjs:1.1.1
pipeline-utility-steps:2.8.0
workflow-aggregator:2.6
authentication-tokens:1.4
jackson2-api:2.12.4
apache-httpcomponents-client-4-api:4.5.13-1.0
workflow-basic-steps:2.24
purge-job-history:1.6
pipeline-build-step:2.15
jdk-tool:1.5
github:1.34.0
git-server:1.10
ace-editor:1.1
jquery-detached:1.2.1
ws-cleanup:0.39
oic-auth:1.8
junit:1.52
workflow-durable-task-step:2.39
powershell:1.5
copyartifact:1.46.1
ldap:2.7
script-security:1.78
workflow-job:2.41
config-file-provider:3.8.1
run-condition:1.5
pipeline-model-definition:1.9.1
plugin-util-api:2.4.0
pipeline-multibranch-defaults:2.1
msbuild:1.30
github-api:1.123
handy-uri-templates-2-api:2.1.8-1.0
bootstrap5-api:5.1.0-1
pipeline-stage-step:2.5
popper2-api:2.9.3-1
uno-choice:2.5.6
pipeline-milestone-step:1.3.2
windows-azure-storage:359.v09573e5ff73f
jquery:1.12.4-1
workflow-cps-global-lib:2.21
workflow-cps:2.93
durable-task:1.39
variant:1.4
jquery3-api:3.6.0-2
jsch:0.1.55.2
git-parameter:0.9.13
azure-sdk:30.vf3165534d6e8
azure-artifact-manager:94.vcb8b76839313
pipeline-stage-view:2.19
scm-api:2.6.5
pipeline-model-api:1.9.1
parameterized-trigger:2.41
azure-ad:179.vf6841393099e
credentials-binding:1.27
mailer:1.34
git:4.8.2
workflow-scm-step:2.13
checks-api:1.7.2
antisamy-markup-formatter:2.1
handlebars:3.0.8
display-url-api:2.3.5
saml:2.0.7
command-launcher:1.6
pipeline-model-extensions:1.9.1
okhttp-api:3.14.9
git-client:3.9.0
ssh-credentials:1.19
bootstrap4-api:4.6.0-3
trilead-api:1.0.13
maven-plugin:3.12
workflow-api:2.46
workflow-step-api:2.24
pipeline-rest-api:2.19
bouncycastle-api:2.23
branch-api:2.6.5
echarts-api:5.1.2-9
build-timestamp:1.0.3
jaxb:2.3.0.1
structs:1.23
pipeline-graph-analysis:1.11
cloudbees-folder:6.16
caffeine-api:2.9.2-29.v717aac953ff3
github-branch-source:2.11.2
javadoc:1.6
azure-credentials:182.v3ccd4a755864
Jenkins: 2.307 OS: Linux - 5.8.0-1040-azure --- blueocean-commons:1.24.8 azure-keyvault:123.v1aba71c2d365 credentials:2.5 resource-disposer:0.16 jjwt-api:0.11.2-9.c8b45b8bb173 build-pipeline-plugin:1.5.8 pipeline-input-step:2.12 azure-commons:1.1.3 matrix-project:1.19 workflow-support:3.8 sshd:3.1.0 token-macro:266.v44a80cf277fd font-awesome-api:5.15.3-4 conditional-buildstep:1.4.1 favorite:2.3.3 pipeline-stage-tags-metadata:1.9.1 matrix-auth:2.6.8 plain-credentials:1.7 lockable-resources:2.11 popper-api:1.16.1-2 workflow-multibranch:2.26 snakeyaml-api:1.29.1 htmlpublisher:1.25 blueocean-rest:1.24.8 momentjs:1.1.1 pipeline-utility-steps:2.8.0 workflow-aggregator:2.6 authentication-tokens:1.4 jackson2-api:2.12.4 apache-httpcomponents-client-4-api:4.5.13-1.0 workflow-basic-steps:2.24 purge-job-history:1.6 pipeline-build-step:2.15 jdk-tool:1.5 github:1.34.0 git-server:1.10 ace-editor:1.1 jquery-detached:1.2.1 ws-cleanup:0.39 oic-auth:1.8 junit:1.52 workflow-durable-task-step:2.39 powershell:1.5 copyartifact:1.46.1 ldap:2.7 script-security:1.78 workflow-job:2.41 config-file-provider:3.8.1 run-condition:1.5 pipeline-model-definition:1.9.1 plugin-util-api:2.4.0 pipeline-multibranch-defaults:2.1 msbuild:1.30 github-api:1.123 handy-uri-templates-2-api:2.1.8-1.0 bootstrap5-api:5.1.0-1 pipeline-stage-step:2.5 popper2-api:2.9.3-1 uno-choice:2.5.6 pipeline-milestone-step:1.3.2 windows-azure-storage:359.v09573e5ff73f jquery:1.12.4-1 workflow-cps-global-lib:2.21 workflow-cps:2.93 durable-task:1.39 variant:1.4 jquery3-api:3.6.0-2 jsch:0.1.55.2 git-parameter:0.9.13 azure-sdk:30.vf3165534d6e8 azure-artifact-manager:94.vcb8b76839313 pipeline-stage-view:2.19 scm-api:2.6.5 pipeline-model-api:1.9.1 parameterized-trigger:2.41 azure-ad:179.vf6841393099e credentials-binding:1.27 mailer:1.34 git:4.8.2 workflow-scm-step:2.13 checks-api:1.7.2 antisamy-markup-formatter:2.1 handlebars:3.0.8 display-url-api:2.3.5 saml:2.0.7 command-launcher:1.6 pipeline-model-extensions:1.9.1 okhttp-api:3.14.9 git-client:3.9.0 ssh-credentials:1.19 bootstrap4-api:4.6.0-3 trilead-api:1.0.13 maven-plugin:3.12 workflow-api:2.46 workflow-step-api:2.24 pipeline-rest-api:2.19 bouncycastle-api:2.23 branch-api:2.6.5 echarts-api:5.1.2-9 build-timestamp:1.0.3 jaxb:2.3.0.1 structs:1.23 pipeline-graph-analysis:1.11 cloudbees-folder:6.16 caffeine-api:2.9.2-29.v717aac953ff3 github-branch-source:2.11.2 javadoc:1.6 azure-credentials:182.v3ccd4a755864
Since the last update, 2.307, I'm unable to open jobs webpage when they ran at least once. When I click on the job it waits a minute or so and the it returns a 504 Gateway Time-out error.
[JENKINS-66433] Unable to open job webpage
Neil
added a comment - I can confirm that the problem persists on the lts version as well. We downgraded to 2.306 in our helm chart and the memory consumption was cut by more than 50%, fixing the problem with our pods restarting.
Neil
added a comment - I can confirm that the problem persists on the lts version as well. We downgraded to 2.306 in our helm chart and the memory consumption was cut by more than 50%, fixing the problem with our pods restarting. Per the thread dump, it seems to be between core and Azure Artifact Manager:
46362msec elapsed in Handling GET /job/Client/ from 188.10.26.98 : Jetty (winstone)-111 Job/index.jelly WorkflowJob/main.jelly
…
reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:87)
reactor.core.publisher.Flux.blockLast(Flux.java:2519)
com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:94)
com.azure.core.util.paging.ContinuablePagedByIteratorBase.hasNext(ContinuablePagedByIteratorBase.java:53)
com.microsoft.jenkins.artifactmanager.AzureBlobVirtualFile.run(AzureBlobVirtualFile.java:116)
hudson.model.Run.getArtifactsUpTo(Run.java:1160)
java.base@11.0.11/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
FYI timja
Changelog of 2.307 states:
Removed: The Woodstox implementation of the StAX API has been removed from Jenkins core. Users of the Azure Artifact Manager, Azure Container Agents, Azure Storage, and Azure SDK API plugins must upgrade those plugins to the latest versions in lockstep with this core upgrade.
Related?
nsewardep I don't think you're seeing the same problem on the LTS version. The change in 2.307 was not backported to the LTS version. You must be seeing a different issue that needs further explanation in a new bug report.
Neil
added a comment - markewaite added the issue https://issues.jenkins.io/browse/JENKINS-66491 
I don't see any further logging information that will help at this time. Based on the comment from danielbeck, it looks like the getArtifactsUpTo() call to AzureBlobVirtualFile is being reported in the nodes/master/thread-dump.txt file of the attached support bundle support_2021-08-23_10.49.55.zip
. Many calls to BlockingSingleSubscriber.blockingGet are listed in the slow-requests/20210823-316.txt file as well.
Can you confirm that your Azure File Storage is responding to requests? We had issues in the past with Jenkins infrastructure use of Azure File Storage where the storage system was not responding to requests due to too many locks being acquired.
We may need help from timja or others with more knowledge of Azure file storage.
I assume that you have updated your plugins since the original list was provided and that you've removed the build pipeline plugin that has an unresolved security vulnerability. When I ran Jenkins 2.309 with your plugin list, there were several plugins that needed to be updated, though none of them were in the list of plugins mentioned in the 2.307 changelog. I don't expect that any of those updates affect the issue here, but that is what I observed while trying to duplicate the issue.
I tried to duplicate the issue as reported but was unable to do so. I assume that is because I do not have Azure File Storage defined that I can configure in the test.
Hey markewaite , thanks for your reply. Yes, I updated all plugins, if you feel it'd be helpful I can upgrade to 2.309 create a new support bundle and upload it here, then I'll revert back to 2.306 which is the latest version which correctly run on our infrastructure.
The Azure Storage looks fine. From the Manage Jenkins page I can configure it and connect to it and it works.
basil FYI, I’ll try look at this but I’ve got a few issues I’m looking at currently along with getting all the docker images updated
Basil Crow
added a comment - Jackson 2.12.4 is still using Thread.currentThread().getContextClassLoader(), which means it is using SJSXP on Jenkins 2.307 and later. This isn't what azure-core is usually run against; azure-core normally runs against Woodstox, which is probably what it is tested against as well (and what it runs against on Jenkins 2.306, albeit a much older version of Woodstox than what azure-core recommends).
Jackson 2.13 (scheduled for release this month) should eliminate the usage of Thread.currentThread().getContextClassLoader(), which means Jackson 2.13 will be using Woodstox on Jenkins 2.307 or later (and not only that, but a recent version). I just posted a PR to get an incremental build of the jackson2-api plugin based on Jackson 2.13.0-rc2. The incremental is (confusingly named) jackson2-api-2.12.5-rc209.e0f7755544a6.hpi and can be downloaded from here: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/jackson2-api/2.12.5-rc209.e0f7755544a6/
Can you try running with this incremental build of the Jackson 2 API plugin (based on Jackson 2.13.0-rc2) and the latest version of Jenkins core and let me know whether the problem is resolved? For instructions on how to install a custom build, see: https://www.jenkins.io/doc/book/managing/plugins/#advanced-installation
Basil Crow
added a comment - Jackson 2.12.4 is still using Thread.currentThread().getContextClassLoader() , which means it is using SJSXP on Jenkins 2.307 and later. This isn't what azure-core is usually run against; azure-core normally runs against Woodstox, which is probably what it is tested against as well (and what it runs against on Jenkins 2.306, albeit a much older version of Woodstox than what azure-core recommends).
Jackson 2.13 (scheduled for release this month) should eliminate the usage of Thread.currentThread().getContextClassLoader() , which means Jackson 2.13 will be using Woodstox on Jenkins 2.307 or later (and not only that, but a recent version). I just posted a PR to get an incremental build of the jackson2-api plugin based on Jackson 2.13.0-rc2. The incremental is (confusingly named) jackson2-api-2.12.5-rc209.e0f7755544a6.hpi and can be downloaded from here: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/jackson2-api/2.12.5-rc209.e0f7755544a6/
Can you try running with this incremental build of the Jackson 2 API plugin (based on Jackson 2.13.0-rc2) and the latest version of Jenkins core and let me know whether the problem is resolved? For instructions on how to install a custom build, see: https://www.jenkins.io/doc/book/managing/plugins/#advanced-installation Hey basil , sorry for the delay but I was out of city for the weekend; your incremental build worked like a charm with 2.309.
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.13 says that Jackson 2.13.0 was released on Sep 30, 2021.
Unfortunately, I don't see any evidence of that beyond the wiki page. https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core does not show a 2.13.0 release, nor do any of the related locations like https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind show a 2.13.0 release.
Hopefully released artifacts are coming soon.
Basil Crow
added a comment - I updated jenkinsci/jackson2-api-plugin#91 to use the final release of 2.13.0. New incremental is at: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/jackson2-api/2.12.5-rc209.5e59ca1526d1/
Basil Crow
added a comment - I updated jenkinsci/jackson2-api-plugin#91 to use the final release of 2.13.0. New incremental is at: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/jackson2-api/2.12.5-rc209.5e59ca1526d1/ attoy can you confirm that the 2.13.0 release of jackson2-api resolves it also? See https://github.com/jenkinsci/jackson2-api-plugin/releases/tag/2.13.0-226.v0c5dd2d2fd2a for release details.
Downgrading to 2.306 fixed the issue.