• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core

      Since the last update, 2.307, I'm unable to open jobs webpage when they ran at least once. When I click on the job it waits a minute or so and the it returns a 504 Gateway Time-out error.

          [JENKINS-66433] Unable to open job webpage

          Mark Waite added a comment - - edited

          I don't see any further logging information that will help at this time. Based on the comment from danielbeck, it looks like the getArtifactsUpTo() call to AzureBlobVirtualFile is being reported in the nodes/master/thread-dump.txt file of the attached support bundle support_2021-08-23_10.49.55.zip . Many calls to BlockingSingleSubscriber.blockingGet are listed in the slow-requests/20210823-316.txt file as well.

          Can you confirm that your Azure File Storage is responding to requests? We had issues in the past with Jenkins infrastructure use of Azure File Storage where the storage system was not responding to requests due to too many locks being acquired.

          We may need help from timja or others with more knowledge of Azure file storage.

          I assume that you have updated your plugins since the original list was provided and that you've removed the build pipeline plugin that has an unresolved security vulnerability. When I ran Jenkins 2.309 with your plugin list, there were several plugins that needed to be updated, though none of them were in the list of plugins mentioned in the 2.307 changelog. I don't expect that any of those updates affect the issue here, but that is what I observed while trying to duplicate the issue.

          I tried to duplicate the issue as reported but was unable to do so. I assume that is because I do not have Azure File Storage defined that I can configure in the test.

          Mark Waite added a comment - - edited I don't see any further logging information that will help at this time. Based on the comment from danielbeck , it looks like the getArtifactsUpTo() call to AzureBlobVirtualFile is being reported in the nodes/master/thread-dump.txt file of the attached support bundle support_2021-08-23_10.49.55.zip . Many calls to BlockingSingleSubscriber.blockingGet are listed in the slow-requests/20210823-316.txt file as well. Can you confirm that your Azure File Storage is responding to requests? We had issues in the past with Jenkins infrastructure use of Azure File Storage where the storage system was not responding to requests due to too many locks being acquired. We may need help from timja or others with more knowledge of Azure file storage. I assume that you have updated your plugins since the original list was provided and that you've removed the build pipeline plugin that has an unresolved security vulnerability. When I ran Jenkins 2.309 with your plugin list, there were several plugins that needed to be updated, though none of them were in the list of plugins mentioned in the 2.307 changelog. I don't expect that any of those updates affect the issue here, but that is what I observed while trying to duplicate the issue. I tried to duplicate the issue as reported but was unable to do so. I assume that is because I do not have Azure File Storage defined that I can configure in the test.

          Hey markewaite , thanks for your reply. Yes, I updated all plugins, if you feel it'd be helpful I can upgrade to 2.309 create a new support bundle and upload it here, then I'll revert back to 2.306 which is the latest version which correctly run on our infrastructure.

          The Azure Storage looks fine. From the Manage Jenkins page I can configure it and connect to it and it works.

          Leandro Rebosio added a comment - Hey markewaite  , thanks for your reply. Yes, I updated all plugins, if you feel it'd be helpful I can upgrade to 2.309 create a new support bundle and upload it here, then I'll revert back to 2.306 which is the latest version which correctly run on our infrastructure. The Azure Storage looks fine. From the Manage Jenkins page I can configure it and connect to it and it works.

          Tim Jacomb added a comment -

          basil FYI, I’ll try look at this but I’ve got a few issues I’m looking at currently along with getting all the docker images updated

          Tim Jacomb added a comment - basil FYI, I’ll try look at this but I’ve got a few issues I’m looking at currently along with getting all the docker images updated

          Basil Crow added a comment -

          Jackson 2.12.4 is still using Thread.currentThread().getContextClassLoader(), which means it is using SJSXP on Jenkins 2.307 and later. This isn't what azure-core is usually run against; azure-core normally runs against Woodstox, which is probably what it is tested against as well (and what it runs against on Jenkins 2.306, albeit a much older version of Woodstox than what azure-core recommends).

          Jackson 2.13 (scheduled for release this month) should eliminate the usage of Thread.currentThread().getContextClassLoader(), which means Jackson 2.13 will be using Woodstox on Jenkins 2.307 or later (and not only that, but a recent version). I just posted a PR to get an incremental build of the jackson2-api plugin based on Jackson 2.13.0-rc2. The incremental is (confusingly named) jackson2-api-2.12.5-rc209.e0f7755544a6.hpi and can be downloaded from here: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/jackson2-api/2.12.5-rc209.e0f7755544a6/

          Can you try running with this incremental build of the Jackson 2 API plugin (based on Jackson 2.13.0-rc2) and the latest version of Jenkins core and let me know whether the problem is resolved? For instructions on how to install a custom build, see: https://www.jenkins.io/doc/book/managing/plugins/#advanced-installation

          Basil Crow added a comment - Jackson 2.12.4 is still using Thread.currentThread().getContextClassLoader() , which means it is using SJSXP on Jenkins 2.307 and later. This isn't what azure-core is usually run against; azure-core normally runs against Woodstox, which is probably what it is tested against as well (and what it runs against on Jenkins 2.306, albeit a much older version of Woodstox than what azure-core recommends). Jackson 2.13 (scheduled for release this month) should eliminate the usage of Thread.currentThread().getContextClassLoader() , which means Jackson 2.13 will be using Woodstox on Jenkins 2.307 or later (and not only that, but a recent version). I just posted a PR to get an incremental build of the jackson2-api plugin based on Jackson 2.13.0-rc2. The incremental is (confusingly named) jackson2-api-2.12.5-rc209.e0f7755544a6.hpi and can be downloaded from here: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/jackson2-api/2.12.5-rc209.e0f7755544a6/ Can you try running with this incremental build of the Jackson 2 API plugin (based on Jackson 2.13.0-rc2) and the latest version of Jenkins core and let me know whether the problem is resolved? For instructions on how to install a custom build, see: https://www.jenkins.io/doc/book/managing/plugins/#advanced-installation

          Hey basil , sorry for the delay but I was out of city for the weekend; your incremental build worked like a charm with 2.309.

          Leandro Rebosio added a comment - Hey basil  , sorry for the delay but I was out of city for the weekend; your incremental build worked like a charm with 2.309.

          Mark Waite added a comment -

          https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.13 says that Jackson 2.13.0 was released on Sep 30, 2021.

          Unfortunately, I don't see any evidence of that beyond the wiki page. https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core does not show a 2.13.0 release, nor do any of the related locations like https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind show a 2.13.0 release.

          Hopefully released artifacts are coming soon.

          Mark Waite added a comment - https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.13 says that Jackson 2.13.0 was released on Sep 30, 2021. Unfortunately, I don't see any evidence of that beyond the wiki page. https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core does not show a 2.13.0 release, nor do any of the related locations like https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind show a 2.13.0 release. Hopefully released artifacts are coming soon.

          Daniel Beck added a comment -

          Daniel Beck added a comment - https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.13.0/ ?

          Basil Crow added a comment -

          Basil Crow added a comment - I updated jenkinsci/jackson2-api-plugin#91 to use the final release of 2.13.0. New incremental is at: https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/jackson2-api/2.12.5-rc209.5e59ca1526d1/

          Thank you basil ; updated and working good!

          Leandro Rebosio added a comment - Thank you basil  ; updated and working good!

          Mark Waite added a comment -

          attoy can you confirm that the 2.13.0 release of jackson2-api resolves it also? See https://github.com/jenkinsci/jackson2-api-plugin/releases/tag/2.13.0-226.v0c5dd2d2fd2a for release details.

          Mark Waite added a comment - attoy can you confirm that the 2.13.0 release of jackson2-api resolves it also? See https://github.com/jenkinsci/jackson2-api-plugin/releases/tag/2.13.0-226.v0c5dd2d2fd2a for release details.

            Unassigned Unassigned
            attoy Leandro Rebosio
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: