Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60536

Jenkins job hangs in Jira call after git rev-list command

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • jira-plugin

      Recently Jenkins master  when a job is triggered it hangs around checkout

       Command that it seems to get stuck on:

      git rev-list --no-walk a4539ebbe8fe6da843fb1fe4c7b0ce5cf79a0647 # timeout=10

      This issue only started happening sporadically from 2019-12-17 and is impacting our CI/CD pipeline. We rely on Jenkins to deploy to our environments.

       

      If you tried to kill the job, it is unresponsive.

      If you restart Jenkins master then the job that gets stuck is no longer exists

       

      Job Build Logs:

      Started by upstream project "parent-job" build number 5
originally caused by:
 Branch indexing
Checking out git <REPO> into /var/lib/jenkins/workspace/acceptance-tests@script to read acceptance-tests/Jenkinsfile-local
using credential jenkins-git
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url git@bitbucket.org:obillex/fundit.git # timeout=10
Fetching upstream changes from git@bitbucket.org:obillex/fundit.git
 > git --version # timeout=10
using GIT_SSH to set credentials Jenkins access to git
 > git fetch --tags --progress -- <REPO>+refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse b6a1b7a159b86dd5a52b8385670397d568345c53^{commit} # timeout=10
Checking out Revision b6a1b7a159b86dd5a52b8385670397d568345c53 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f b6a1b7a159b86dd5a52b8385670397d568345c53 # timeout=10
Commit message: "Commit message"
 > git rev-list --no-walk a4539ebbe8fe6da843fb1fe4c7b0ce5cf79a0647 # timeout=10

      This issue seems to be a regression of this resolved issue JENKINS-43106

        1. About Jenkins 2.208 [Jenkins].pdf
          63 kB
        2. Screenshot from 2019-12-19 10-27-27.png
          Screenshot from 2019-12-19 10-27-27.png
          268 kB
        3. image-2020-02-26-08-28-37-010.png
          image-2020-02-26-08-28-37-010.png
          25 kB
        4. image-2020-02-26-08-29-08-607.png
          image-2020-02-26-08-29-08-607.png
          228 kB
        5. image-2020-02-26-08-29-32-302.png
          image-2020-02-26-08-29-32-302.png
          227 kB
        6. image-2020-02-26-08-29-59-154.png
          image-2020-02-26-08-29-59-154.png
          237 kB

          [JENKINS-60536] Jenkins job hangs in Jira call after git rev-list command

          Emma Phillips added a comment -

          Upgraded to the latest version of Jenkins 2.209 and within 5 minutes of running the new version the issue has occurred again.

           

          You cannot stop/kill the job once it gets into this state. When restarting, the job ceases to exist

          Emma Phillips added a comment - Upgraded to the latest version of Jenkins 2.209 and within 5 minutes of running the new version the issue has occurred again.   You cannot stop/kill the job once it gets into this state. When restarting, the job ceases to exist

          Mark Waite added a comment -

          If this is an instance of JENKINS-43106, then the Jira 2.5.0 plugin would be installed on the system. Libraries included in the Jira 2.5.0 plugin were causing the issue. Since you state in the environment section of the bug report that you updated all the plugins on the system, I doubt that you are running that old version of the Jira plugin. Could you provide the list of installed plugins (Manage Jenkins - System Information)?

          Since you're running on Amazon EC2 with an EBS backed volume, is there any indication from the AWS reports or environment which might hint that you are exhausting some I/O limit?

          When the git rev-list command hangs, are other jobs able to continue, or do they also hang?

          When the git rev-list command hangs, can a user on the Jenkins master perform command line git operations still in the JENKINS_HOME directory structure? For example, can that user perform a git status or a git rev-list command in any other workspace?

          If you create a similar environment on a different EC2 machine, can you see the same failure?

          If you create a similar environment on a local machine, can you see the same failure?

          Mark Waite added a comment - If this is an instance of JENKINS-43106 , then the Jira 2.5.0 plugin would be installed on the system. Libraries included in the Jira 2.5.0 plugin were causing the issue. Since you state in the environment section of the bug report that you updated all the plugins on the system, I doubt that you are running that old version of the Jira plugin. Could you provide the list of installed plugins (Manage Jenkins - System Information)? Since you're running on Amazon EC2 with an EBS backed volume, is there any indication from the AWS reports or environment which might hint that you are exhausting some I/O limit? When the git rev-list command hangs, are other jobs able to continue, or do they also hang? When the git rev-list command hangs, can a user on the Jenkins master perform command line git operations still in the JENKINS_HOME directory structure? For example, can that user perform a git status or a git rev-list command in any other workspace? If you create a similar environment on a different EC2 machine, can you see the same failure? If you create a similar environment on a local machine, can you see the same failure?

          Kieran Shaw added a comment -

          Hi Mark,

           

          Thanks for picking this up so quickly. I'm working with Emma on this, below is our plugin list. 

          It seems quite intermittent, some jobs are able to start successfully and complete once started, but others just can't seem to get started and then can't be killed.

          Whilst there is a hung job, other jobs can run ok. We fire up a new EC2 slave for every job.

          We are running on EC2 and EBS backed storage, but it shows no signs of unusual behaviour and our burst balance is fine.

          Using the Monitoring plugin, I can see the thread for one of the hung jobs, this is what it is showing:

           

          Name  â†“ Version    Enabled   
          ace-editor 1.1 true
          ansicolor 0.6.2 true
          ant 1.10 true
          antisamy-markup-formatter 1.6 true
          apache-httpcomponents-client-4-api 4.5.10-2.0 true
          authentication-tokens 1.3 true
          aws-credentials 1.28 true
          aws-java-sdk 1.11.687 true
          blueocean 1.21.0 true
          blueocean-autofavorite 1.2.4 true
          blueocean-bitbucket-pipeline 1.21.0 true
          blueocean-commons 1.21.0 true
          blueocean-config 1.21.0 true
          blueocean-core-js 1.21.0 true
          blueocean-dashboard 1.21.0 true
          blueocean-display-url 2.3.0 true
          blueocean-events 1.21.0 true
          blueocean-git-pipeline 1.21.0 true
          blueocean-github-pipeline 1.21.0 true
          blueocean-i18n 1.21.0 true
          blueocean-jira 1.21.0 true
          blueocean-jwt 1.21.0 true
          blueocean-personalization 1.21.0 true
          blueocean-pipeline-api-impl 1.21.0 true
          blueocean-pipeline-editor 1.21.0 true
          blueocean-pipeline-scm-api 1.21.0 true
          blueocean-rest 1.21.0 true
          blueocean-rest-impl 1.21.0 true
          blueocean-web 1.21.0 true
          bouncycastle-api 2.17 true
          branch-api 2.5.5 true
          build-pipeline-plugin 1.5.8 true
          build-timeout 1.19 true
          cloudbees-bitbucket-branch-source 2.6.0 true
          cloudbees-folder 6.10.1 true
          command-launcher 1.4 true
          conditional-buildstep 1.3.6 true
          config-file-provider 3.6.2 true
          credentials 2.3.0 true
          credentials-binding 1.20 true
          display-url-api 2.3.2 true
          docker-commons 1.15 true
          docker-workflow 1.21 true
          durable-task 1.33 true
          ec2 1.47 true
          email-ext 2.68 true
          favorite 2.3.2 true
          git 4.0.0 true
          git-client 3.0.0 true
          git-server 1.9 true
          github 1.29.5 true
          github-api 1.95 true
          github-branch-source 2.5.8 true
          gradle 1.35 true
          h2-api 1.4.199 true
          handlebars 1.1.1 true
          handy-uri-templates-2-api 2.1.8-1.0 true
          htmlpublisher 1.21 true
          jackson2-api 2.10.1 true
          javadoc 1.5 true
          jaxb 2.3.0.1 true
          jdk-tool 1.4 true
          jenkins-design-language 1.21.0 true
          jira 3.0.11 true
          jira-steps 1.5.1 true
          jquery 1.12.4-1 true
          jquery-detached 1.2.1 true
          jsch 0.1.55.1 true
          junit 1.28 true
          ldap 1.21 true
          lockable-resources 2.7 true
          mailer 1.29 true
          matrix-auth 2.5 true
          matrix-project 1.14 true
          maven-plugin 3.4 true
          mercurial 2.8 true
          momentjs 1.1.1 true
          monitoring 1.80.0 true
          node-iterator-api 1.5.0 true
          nodejs 1.3.4 true
          parameterized-trigger 2.36 true
          performance 3.17 true
          pipeline-aws 1.39 true
          pipeline-build-step 2.10 true
          pipeline-github-lib 1.0 true
          pipeline-graph-analysis 1.10 true
          pipeline-input-step 2.11 true
          pipeline-maven 3.8.2 true
          pipeline-milestone-step 1.3.1 true
          pipeline-model-api 1.5.0 true
          pipeline-model-declarative-agent 1.1.1 true
          pipeline-model-definition 1.5.0 true
          pipeline-model-extensions 1.5.0 true
          pipeline-rest-api 2.12 true
          pipeline-stage-step 2.3 true
          pipeline-stage-tags-metadata 1.5.0 true
          pipeline-stage-view 2.12 true
          plain-credentials 1.5 true
          pubsub-light 1.13 true
          resource-disposer 0.14 true
          run-condition 1.2 true
          saml 1.1.4 true
          scm-api 2.6.3 true
          script-security 1.68 true
          slack 2.35 true
          sonar 2.10 true
          sse-gateway 1.20 true
          ssh-agent 1.17 true
          ssh-credentials 1.18 true
          ssh-slaves 1.31.0 true
          structs 1.20 true
          test-results-analyzer 0.3.5 true
          timestamper 1.10 true
          token-macro 2.10 true
          trilead-api 1.0.5 true
          variant 1.3 true
          workflow-aggregator 2.6 true
          workflow-api 2.38 true
          workflow-basic-steps 2.18 true
          workflow-cps 2.78 true
          workflow-cps-global-lib 2.15 true
          workflow-durable-task-step 2.35 true
          workflow-job 2.36 true
          workflow-multibranch 2.21 true
          workflow-scm-step 2.9 true
          workflow-step-api 2.21 true
          workflow-support 3.3 true
          ws-cleanup 0.38 true

          Kieran Shaw added a comment - Hi Mark,   Thanks for picking this up so quickly. I'm working with Emma on this, below is our plugin list.  It seems quite intermittent, some jobs are able to start successfully and complete once started, but others just can't seem to get started and then can't be killed. Whilst there is a hung job, other jobs can run ok. We fire up a new EC2 slave for every job. We are running on EC2 and EBS backed storage, but it shows no signs of unusual behaviour and our burst balance is fine. Using the Monitoring plugin, I can see the thread for one of the hung jobs, this is what it is showing:   Name  â†“ Version    Enabled    ace-editor 1.1 true ansicolor 0.6.2 true ant 1.10 true antisamy-markup-formatter 1.6 true apache-httpcomponents-client-4-api 4.5.10-2.0 true authentication-tokens 1.3 true aws-credentials 1.28 true aws-java-sdk 1.11.687 true blueocean 1.21.0 true blueocean-autofavorite 1.2.4 true blueocean-bitbucket-pipeline 1.21.0 true blueocean-commons 1.21.0 true blueocean-config 1.21.0 true blueocean-core-js 1.21.0 true blueocean-dashboard 1.21.0 true blueocean-display-url 2.3.0 true blueocean-events 1.21.0 true blueocean-git-pipeline 1.21.0 true blueocean-github-pipeline 1.21.0 true blueocean-i18n 1.21.0 true blueocean-jira 1.21.0 true blueocean-jwt 1.21.0 true blueocean-personalization 1.21.0 true blueocean-pipeline-api-impl 1.21.0 true blueocean-pipeline-editor 1.21.0 true blueocean-pipeline-scm-api 1.21.0 true blueocean-rest 1.21.0 true blueocean-rest-impl 1.21.0 true blueocean-web 1.21.0 true bouncycastle-api 2.17 true branch-api 2.5.5 true build-pipeline-plugin 1.5.8 true build-timeout 1.19 true cloudbees-bitbucket-branch-source 2.6.0 true cloudbees-folder 6.10.1 true command-launcher 1.4 true conditional-buildstep 1.3.6 true config-file-provider 3.6.2 true credentials 2.3.0 true credentials-binding 1.20 true display-url-api 2.3.2 true docker-commons 1.15 true docker-workflow 1.21 true durable-task 1.33 true ec2 1.47 true email-ext 2.68 true favorite 2.3.2 true git 4.0.0 true git-client 3.0.0 true git-server 1.9 true github 1.29.5 true github-api 1.95 true github-branch-source 2.5.8 true gradle 1.35 true h2-api 1.4.199 true handlebars 1.1.1 true handy-uri-templates-2-api 2.1.8-1.0 true htmlpublisher 1.21 true jackson2-api 2.10.1 true javadoc 1.5 true jaxb 2.3.0.1 true jdk-tool 1.4 true jenkins-design-language 1.21.0 true jira 3.0.11 true jira-steps 1.5.1 true jquery 1.12.4-1 true jquery-detached 1.2.1 true jsch 0.1.55.1 true junit 1.28 true ldap 1.21 true lockable-resources 2.7 true mailer 1.29 true matrix-auth 2.5 true matrix-project 1.14 true maven-plugin 3.4 true mercurial 2.8 true momentjs 1.1.1 true monitoring 1.80.0 true node-iterator-api 1.5.0 true nodejs 1.3.4 true parameterized-trigger 2.36 true performance 3.17 true pipeline-aws 1.39 true pipeline-build-step 2.10 true pipeline-github-lib 1.0 true pipeline-graph-analysis 1.10 true pipeline-input-step 2.11 true pipeline-maven 3.8.2 true pipeline-milestone-step 1.3.1 true pipeline-model-api 1.5.0 true pipeline-model-declarative-agent 1.1.1 true pipeline-model-definition 1.5.0 true pipeline-model-extensions 1.5.0 true pipeline-rest-api 2.12 true pipeline-stage-step 2.3 true pipeline-stage-tags-metadata 1.5.0 true pipeline-stage-view 2.12 true plain-credentials 1.5 true pubsub-light 1.13 true resource-disposer 0.14 true run-condition 1.2 true saml 1.1.4 true scm-api 2.6.3 true script-security 1.68 true slack 2.35 true sonar 2.10 true sse-gateway 1.20 true ssh-agent 1.17 true ssh-credentials 1.18 true ssh-slaves 1.31.0 true structs 1.20 true test-results-analyzer 0.3.5 true timestamper 1.10 true token-macro 2.10 true trilead-api 1.0.5 true variant 1.3 true workflow-aggregator 2.6 true workflow-api 2.38 true workflow-basic-steps 2.18 true workflow-cps 2.78 true workflow-cps-global-lib 2.15 true workflow-durable-task-step 2.35 true workflow-job 2.36 true workflow-multibranch 2.21 true workflow-scm-step 2.9 true workflow-step-api 2.21 true workflow-support 3.3 true ws-cleanup 0.38 true

          Kieran Shaw added a comment -

          We can also run git commands just fine in the workspace directory of the job that is trying to be run.

          Kieran Shaw added a comment - We can also run git commands just fine in the workspace directory of the job that is trying to be run.

          Kieran Shaw added a comment -

          We also just had a case of a job managing to get past the master checkout and assigned a slave and then hung in the same place doing the rev-list on the slave.

          Kieran Shaw added a comment - We also just had a case of a job managing to get past the master checkout and assigned a slave and then hung in the same place doing the rev-list on the slave.

          Mark Waite added a comment -

          Based on a comparison of your installed plugins and my installed plugins, you might consider the following as ways to explore the problem further. I don't have any reason to believe that any of these are the root of the problem, I'm just guessing in an attempt to further explore what might be causing the issue.

          Wild Guesses

          • The jira-steps plugin is installed in your environment and not in mine. Since a different jira plugin was deeply involved in JENKINS-43106, you might attempt to remove the jira-steps plugin to see if that helps
          • The build-pipeline-plugin is installed in your environment and not in mine. It has a known security issue that should be unrelated to this issue. However, if you can remove that plugin without harming your users, it may be worth the attempt
          • The test-results-analyzer plugin is installed in your environment and not in mine. I don't see any reason that would affect this case, since it seems to be a presentation and reporting plugin

          Mark Waite added a comment - Based on a comparison of your installed plugins and my installed plugins, you might consider the following as ways to explore the problem further. I don't have any reason to believe that any of these are the root of the problem, I'm just guessing in an attempt to further explore what might be causing the issue. Wild Guesses The jira-steps plugin is installed in your environment and not in mine. Since a different jira plugin was deeply involved in JENKINS-43106 , you might attempt to remove the jira-steps plugin to see if that helps The build-pipeline-plugin is installed in your environment and not in mine. It has a known security issue that should be unrelated to this issue. However, if you can remove that plugin without harming your users, it may be worth the attempt The test-results-analyzer plugin is installed in your environment and not in mine. I don't see any reason that would affect this case, since it seems to be a presentation and reporting plugin

          Kieran Shaw added a comment -

          Tracked down another locked up thread and it does indeed look like Jira:

           

          Executor #-1 for master : executing feature/acceptance-local #541
org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605)
org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511)
org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75)
org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101)
org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423)
org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseHttpPromiseAsyncClient.java:36)
com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417)
com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364)
com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297)
com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianHttpClientDecorator.java:83)
com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253)
com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.java:69)
com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.java:109)
com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94)
hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:196)
hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136)
io.jenkins.blueocean.service.embedded.jira.JiraSCMListener.onChangeLogParsed(JiraSCMListener.java:52)
org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:853)
org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1000(WorkflowRun.java:133)
org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1116)
org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140)
org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:155)
org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:69)
org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:299)
hudson.model.ResourceController.execute(ResourceController.java:97)
hudson.model.Executor.run(Executor.java:428)

          I think it's the bit that parses the changelog and tries to link to Jira issues.

          Kieran Shaw added a comment - Tracked down another locked up thread and it does indeed look like Jira:   Executor #-1  for  master : executing feature/acceptance-local #541 org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605) org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511) org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75) org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101) org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423) org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280) org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295) org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377) org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129) org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75) org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108) com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseHttpPromiseAsyncClient.java:36) com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417) com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364) com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297) com.atlassian.jira. rest .client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianHttpClientDecorator.java:83) com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253) com.atlassian.jira. rest .client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.java:69) com.atlassian.jira. rest .client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.java:109) com.atlassian.jira. rest .client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94) hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:196) hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136) io.jenkins.blueocean.service.embedded.jira.JiraSCMListener.onChangeLogParsed(JiraSCMListener.java:52) org.jenkinsci.plugins.workflow.job.WorkflowRun.onCheckout(WorkflowRun.java:853) org.jenkinsci.plugins.workflow.job.WorkflowRun.access$1000(WorkflowRun.java:133) org.jenkinsci.plugins.workflow.job.WorkflowRun$SCMListenerImpl.onCheckout(WorkflowRun.java:1116) org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:140) org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:155) org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:69) org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:299) hudson.model.ResourceController.execute(ResourceController.java:97) hudson.model.Executor.run(Executor.java:428) I think it's the bit that parses the changelog and tries to link to Jira issues.

          Mark Waite added a comment -

          Interesting and very encouraging. If that is a Jenkins pipeline job, you might insert an echo statement after the checkout scm step so that you have evidence that the checkout completed and the hang is in a different location that is unrelated to the git plugin.

          Mark Waite added a comment - Interesting and very encouraging. If that is a Jenkins pipeline job, you might insert an echo statement after the checkout scm step so that you have evidence that the checkout completed and the hang is in a different location that is unrelated to the git plugin.

          Kieran Shaw added a comment -

          Have removed the Jira site and all is working again. Put it back in, stuff breaks. So I think I've proven it, but I have no idea why it is locking up. It's like there is no timeout in the Jira query.

          Kieran Shaw added a comment - Have removed the Jira site and all is working again. Put it back in, stuff breaks. So I think I've proven it, but I have no idea why it is locking up. It's like there is no timeout in the Jira query.

          Kieran Shaw added a comment -

          I have both the Jira steps and the Jira plugin installed. On the configuration screen you can do a connection test on the Jira steps plugin and it works just fine (I've got it configured with basic auth), but on the Jira plugin "validate settings" button it just ends up timing out with a 504 gateway timeout after 1 minute. Same credentials, same URL. So the connection method the Jira plugin is using under the hood must be different I guess?

          Kieran Shaw added a comment - I have both the Jira steps and the Jira plugin installed. On the configuration screen you can do a connection test on the Jira steps plugin and it works just fine (I've got it configured with basic auth), but on the Jira plugin "validate settings" button it just ends up timing out with a 504 gateway timeout after 1 minute. Same credentials, same URL. So the connection method the Jira plugin is using under the hood must be different I guess?

          Mike Poulson added a comment - - edited

          We saw this issue manifest in jobs also hanging at "Started by user" with the only thing in the console log.    We have repro on fully updated 2.210 system with Jira plugin running 3.0.11.

          By dumping the java threads (sudo jstack <pid> > jstack-out.txt) we discovered the following thread consuming a high amount of CPU.   We were able to see the issue start on ~12/18/2019 after no changes to our master or the plugins.  This was easily seen based on CPU history logs which show the system would consume a full CPU when it was hung.

          By removing the config file hudson.plugins.jira.JiraGlobalConfiguration.xml and restarting it resolved the issue.

          // "Handling GET /job/App/job/master/3/ from 172.30.4.236 : qtp328638398-38531 WorkflowRun/index.jelly GitChangeSetList/digest.jelly" #38531 prio=5 os_prio=0 cpu=58.61ms elapsed=2320.00s tid=0x00007f446840b800 nid=0x23a3 waiting on condition  [0x00007f440703d000]"Handling GET /job/App/job/master/3/ from 172.30.4.236 : qtp328638398-38531 WorkflowRun/index.jelly GitChangeSetList/digest.jelly" #38531 prio=5 os_prio=0 cpu=58.61ms elapsed=2320.00s tid=0x00007f446840b800 nid=0x23a3 waiting on condition  [0x00007f440703d000]   java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method) - parking to wait for  <0x00000006bfcf6888> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.5/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.5/AbstractQueuedSynchronizer.java:885) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.5/AbstractQueuedSynchronizer.java:917) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.5/AbstractQueuedSynchronizer.java:1240) at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.5/ReentrantLock.java:267) at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129) at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108) at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseHttpPromiseAsyncClient.java:36) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297) at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianHttpClientDecorator.java:83) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253) at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.java:69) at com.atlassian.jira.rest.client.internal.async.AsynchronousProjectRestClient.getAllProjects(AsynchronousProjectRestClient.java:61) at hudson.plugins.jira.JiraRestService.getProjectsKeys(JiraRestService.java:182) at hudson.plugins.jira.JiraSession.getProjectKeys(JiraSession.java:65) at hudson.plugins.jira.JiraSite.getProjectKeys(JiraSite.java:815) at hudson.plugins.jira.JiraChangeLogAnnotator.hasProjectForIssue(JiraChangeLogAnnotator.java:146) at hudson.plugins.jira.JiraChangeLogAnnotator.annotate(JiraChangeLogAnnotator.java:75) at hudson.scm.ChangeLogSet$Entry.getMsgAnnotated(ChangeLogSet.java:252) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.5/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.5/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.5/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.5/Method.java:566) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsString(ExpressionSupport.java:46) at org.apache.commons.jelly.tags.core.ExprTag.doTag(ExprTag.java:42) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56) at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43) at org.kohsuke.stapler.Facet.handleIndexRequest(Facet.java:282) at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:99) at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:32) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:456) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:246) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:512) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1592) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1296) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1562) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1211) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:500) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386) at org.eclipse.jetty.server.HttpChannel$$Lambda$175/0x00000008403e7840.dispatch(Unknown Source) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.lang.Thread.run(java.base@11.0.5/Thread.java:834)
   Locked ownable synchronizers: - <0x000000060ab49f70> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

           

          Mike Poulson added a comment - - edited We saw this issue manifest in jobs also hanging at "Started by user" with the only thing in the console log.    We have repro on fully updated 2.210 system with Jira plugin running 3.0.11. By dumping the java threads (sudo jstack <pid> > jstack-out.txt) we discovered the following thread consuming a high amount of CPU.   We were able to see the issue start on ~12/18/2019 after no changes to our master or the plugins.  This was easily seen based on CPU history logs which show the system would consume a full CPU when it was hung. By removing the config file hudson.plugins.jira.JiraGlobalConfiguration.xml and restarting it resolved the issue. // "Handling GET /job/App/job/master/3/ from 172.30.4.236 : qtp328638398-38531 WorkflowRun/index.jelly GitChangeSetList/digest.jelly" #38531 prio=5 os_prio=0 cpu=58.61ms elapsed=2320.00s tid=0x00007f446840b800 nid=0x23a3 waiting on condition  [0x00007f440703d000] "Handling GET /job/App/job/master/3/ from 172.30.4.236 : qtp328638398-38531 WorkflowRun/index.jelly GitChangeSetList/digest.jelly" #38531 prio=5 os_prio=0 cpu=58.61ms elapsed=2320.00s tid=0x00007f446840b800 nid=0x23a3 waiting on condition  [0x00007f440703d000]   java.lang. Thread .State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method) - parking to wait for  <0x00000006bfcf6888> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.5/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.5/AbstractQueuedSynchronizer.java:885) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.5/AbstractQueuedSynchronizer.java:917) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.5/AbstractQueuedSynchronizer.java:1240) at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.5/ReentrantLock.java:267) at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129) at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108) at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseHttpPromiseAsyncClient.java:36) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297) at com.atlassian.jira. rest .client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianHttpClientDecorator.java:83) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253) at com.atlassian.jira. rest .client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.java:69) at com.atlassian.jira. rest .client.internal.async.AsynchronousProjectRestClient.getAllProjects(AsynchronousProjectRestClient.java:61) at hudson.plugins.jira.JiraRestService.getProjectsKeys(JiraRestService.java:182) at hudson.plugins.jira.JiraSession.getProjectKeys(JiraSession.java:65) at hudson.plugins.jira.JiraSite.getProjectKeys(JiraSite.java:815) at hudson.plugins.jira.JiraChangeLogAnnotator.hasProjectForIssue(JiraChangeLogAnnotator.java:146) at hudson.plugins.jira.JiraChangeLogAnnotator.annotate(JiraChangeLogAnnotator.java:75) at hudson.scm.ChangeLogSet$Entry.getMsgAnnotated(ChangeLogSet.java:252) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.5/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.5/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.5/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.5/Method.java:566) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsString(ExpressionSupport.java:46) at org.apache.commons.jelly.tags.core.ExprTag.doTag(ExprTag.java:42) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:147) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161) at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56) at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43) at org.kohsuke.stapler.Facet.handleIndexRequest(Facet.java:282) at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:99) at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:32) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:456) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:280) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:246) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:105) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:118) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:512) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1592) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1296) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1562) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1211) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:500) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:386) at org.eclipse.jetty.server.HttpChannel$$Lambda$175/0x00000008403e7840.dispatch(Unknown Source) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:562) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:378) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.lang. Thread .run(java.base@11.0.5/ Thread .java:834)    Locked ownable synchronizers: - <0x000000060ab49f70> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)  

          Mark Waite added a comment -

          A new Jira integration has been created by Atlassian. You might consider trying the new integration, available from https://plugins.jenkins.io/atlassian-jira-software-cloud . A blog post announcing the new plugin will likely be published early in 2020.

          Mark Waite added a comment - A new Jira integration has been created by Atlassian. You might consider trying the new integration, available from https://plugins.jenkins.io/atlassian-jira-software-cloud . A blog post announcing the new plugin will likely be published early in 2020.

          benoit guerin added a comment -

          Hello markewaite

          Thanks for the news, and the pointers.

          This new plugin only handles build and deployment informations. My company uses the legacy Jira plugin to add comments on issues, and also the jira-steps plugin to enforce our workflow. I do not see how I will do this with this new plugin, until all these features are implemented on it. And I also think I am not alone with these usecases.

          This new Jira plugin is too young enough, and not complete enough to be considered as a drop in remplacement for the legacy Jira one. Plus the legacy is a hard dependency for blue ocean. 

          benoit guerin added a comment - Hello markewaite Thanks for the news, and the pointers. This new plugin only handles build and deployment informations. My company uses the legacy Jira plugin to add comments on issues, and also the jira-steps plugin to enforce our workflow. I do not see how I will do this with this new plugin, until all these features are implemented on it. And I also think I am not alone with these usecases. This new Jira plugin is too young enough, and not complete enough to be considered as a drop in remplacement for the legacy Jira one. Plus the legacy is a hard dependency for blue ocean. 

          Mark Waite added a comment -

          Thanks. You are correct. I had not read enough of the details of the new Jira plugin from Atlassian.

          Mark Waite added a comment - Thanks. You are correct. I had not read enough of the details of the new Jira plugin from Atlassian.

          David France added a comment - - edited

          We've being seeing much the same problem stack that Kieran reported, trying to progress issues by a workflow action in Jira.  We first became aware of the problem on 20/12/2019.

          The locking thread:

          "Handling GET /view/Pipelines/job/.../457/ from a.b.c.d : qtp64133603-15 AbstractBuild/index.jelly SubversionChangeLogS
et/digest.jelly" #15 prio=5 os_prio=0 cpu=8985.66ms elapsed=5244.95s tid=0x0000000064d07000 nid=0x1fdc waiting for monitor entry  [0
x00000000665b7000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605)
        - waiting to lock <0x000000008613f660> (a org.apache.http.nio.reactor.ssl.SSLIOSession)
        at org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511)
        at org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75)
        at org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101)
        at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423)
        at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j
ava:295)
        at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
        at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH
ttpPromiseAsyncClient.java:36)
        at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417)
        at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364)
        at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297)
        at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH
ttpClientDecorator.java:83)
        at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253)
        at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j
ava:69)
        at com.atlassian.jira.rest.client.internal.async.AsynchronousIssueRestClient.getIssue(AsynchronousIssueRestClient.java:186)
        at com.atlassian.jira.rest.client.internal.async.AsynchronousIssueRestClient.getIssue(AsynchronousIssueRestClient.java:177)
        at hudson.plugins.jira.JiraRestService.getIssue(JiraRestService.java:154)
        at hudson.plugins.jira.JiraSession.getIssue(JiraSession.java:126)
        at hudson.plugins.jira.JiraSite.lambda$getIssue$0(JiraSite.java:878)
        at hudson.plugins.jira.JiraSite$$Lambda$303/0x000000002262d840.call(Unknown Source)
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
        - locked <0x000000008644f218> (a com.google.common.cache.LocalCache$StrongAccessEntry)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
        at hudson.plugins.jira.JiraSite.getIssue(JiraSite.java:873)
        at hudson.plugins.jira.JiraChangeLogAnnotator.annotate(JiraChangeLogAnnotator.java:107)
        at hudson.scm.ChangeLogSet$Entry.getMsgAnnotated(ChangeLogSet.java:252)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.5/Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.5/NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.5/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.5/Method.java:566)
        at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
        at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
        at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
        at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
        at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
        at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
        at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
        at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
...

           
          And a waiting thread:

          "Executor #1 for master : executing deployToUAT #1619" #647 daemon prio=5 os_prio=0 cpu=436.80ms elapsed=236.46s tid=0x000000006
2b92000 nid=0x2590 waiting on condition  [0x000000006644d000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method)
        - parking to wait for  <0x0000000083d46f58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.5/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.5/AbstractQueuedSynchronizer.j
ava:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.5/AbstractQueuedSynchronizer.java:917)        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.5/AbstractQueuedSynchronizer.java:1240)
        at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.5/ReentrantLock.java:267)
        at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278)
        at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j
ava:295)
        at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
        at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
        at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
        at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
        at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH
ttpPromiseAsyncClient.java:36)
        at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417)
        at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364)
        at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297)
        at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH
ttpClientDecorator.java:83)
        at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253)
        at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j
ava:69)
        at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.
java:109)
        at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94
)
        at hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:196)
        at hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136)
        at hudson.plugins.jira.JiraSite.progressMatchingIssues(JiraSite.java:1043)
        at hudson.plugins.jira.JiraIssueUpdateBuilder.perform(JiraIssueUpdateBuilder.java:105)
        at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
        at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
        at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
        at hudson.model.Build$BuildExecution.build(Build.java:206)
        at hudson.model.Build$BuildExecution.doRun(Build.java:163)
        at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
        at hudson.model.Run.execute(Run.java:1853)
        at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
        at hudson.model.ResourceController.execute(ResourceController.java:97)
        at hudson.model.Executor.run(Executor.java:427)

           

           The problem looks to be a problem with an earlier connection in the connection pool in the HttpAsyncClient. The earlier open connection is failing to be closed before processing the new request (see the first thread stack above).

          We did a couple of experiments in the jira-plugin source, and managed to find a way get the connection to complete.

          The first hope was that changing the HttpClientOptions.maxConnectionsPerHost value would stop the connection pool holding the connections, but forcing this (in the code) to either 1 or 2000 had no effect - it was still trying to close the previous (now available) connection.

          Instead, as a rough workaround we tried bypassing the connection pooling, by creating a new HttpAsyncClientBuilder against a new PoolingNHttpClientConnectionManager for each request. The change is fairly basic, but has enabled Jenkins to stay up so far.

          The change is on this fork. (Essentially it extracts a method in the constructor, to create the HttpAsyncClientBuilder, then also calls it from doExecute().)

          As mentioned, this was just a minimal/dirty change to get us up and running for now - it just bypasses connection reuse at the point of the request. For example an HttpAsyncClient without connection pooling would be cleaner.

          For reference this is working against JIRA Cloud.  We're not sure why the connections started failing to close, but this was enough to force our builds through for the time being.

          David France added a comment - - edited We've being seeing much the same problem stack that Kieran reported, trying to progress issues by a workflow action in Jira.  We first became aware of the problem on 20/12/2019. The locking thread: "Handling GET /view/Pipelines/job/.../457/ from a.b.c.d : qtp64133603-15 AbstractBuild/index.jelly SubversionChangeLogS et/digest.jelly" #15 prio=5 os_prio=0 cpu=8985.66ms elapsed=5244.95s tid=0x0000000064d07000 nid=0x1fdc waiting for monitor entry [0 x00000000665b7000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605) - waiting to lock <0x000000008613f660> (a org.apache.http.nio.reactor.ssl.SSLIOSession) at org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511) at org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75) at org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101) at org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423) at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j ava:295) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129) at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108) at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH ttpPromiseAsyncClient.java:36) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297) at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH ttpClientDecorator.java:83) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253) at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j ava:69) at com.atlassian.jira.rest.client.internal.async.AsynchronousIssueRestClient.getIssue(AsynchronousIssueRestClient.java:186) at com.atlassian.jira.rest.client.internal.async.AsynchronousIssueRestClient.getIssue(AsynchronousIssueRestClient.java:177) at hudson.plugins.jira.JiraRestService.getIssue(JiraRestService.java:154) at hudson.plugins.jira.JiraSession.getIssue(JiraSession.java:126) at hudson.plugins.jira.JiraSite.lambda$getIssue$0(JiraSite.java:878) at hudson.plugins.jira.JiraSite$$Lambda$303/0x000000002262d840.call(Unknown Source) at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313) - locked <0x000000008644f218> (a com.google.common.cache.LocalCache$StrongAccessEntry) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) at com.google.common.cache.LocalCache.get(LocalCache.java:3965) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764) at hudson.plugins.jira.JiraSite.getIssue(JiraSite.java:873) at hudson.plugins.jira.JiraChangeLogAnnotator.annotate(JiraChangeLogAnnotator.java:107) at hudson.scm.ChangeLogSet$Entry.getMsgAnnotated(ChangeLogSet.java:252) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.5/Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.5/NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.5/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.5/Method.java:566) at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314) at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185) at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) ...   And a waiting thread: "Executor #1 for master : executing deployToUAT #1619" #647 daemon prio=5 os_prio=0 cpu=436.80ms elapsed=236.46s tid=0x000000006 2b92000 nid=0x2590 waiting on condition [0x000000006644d000] java.lang.Thread.State: WAITING (parking) at jdk.internal.misc.Unsafe.park(java.base@11.0.5/Native Method) - parking to wait for <0x0000000083d46f58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(java.base@11.0.5/LockSupport.java:194) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.5/AbstractQueuedSynchronizer.j ava:885) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.5/AbstractQueuedSynchronizer.java:917) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.5/AbstractQueuedSynchronizer.java:1240) at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.5/ReentrantLock.java:267) at org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:278) at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.j ava:295) at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377) at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129) at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75) at org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108) at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseH ttpPromiseAsyncClient.java:36) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417) at com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297) at com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianH ttpClientDecorator.java:83) at com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253) at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.j ava:69) at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient. java:109) at com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94 ) at hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:196) at hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136) at hudson.plugins.jira.JiraSite.progressMatchingIssues(JiraSite.java:1043) at hudson.plugins.jira.JiraIssueUpdateBuilder.perform(JiraIssueUpdateBuilder.java:105) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1853) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:427)    The problem looks to be a problem with an earlier connection in the connection pool in the HttpAsyncClient . The earlier open connection is failing to be closed before processing the new request (see the first thread stack above). We did a couple of experiments in the jira-plugin source, and managed to find a way get the connection to complete. The first hope was that changing the HttpClientOptions.maxConnectionsPerHost value would stop the connection pool holding the connections, but forcing this (in the code) to either 1 or 2000 had no effect - it was still trying to close the previous (now available) connection. Instead, as a rough workaround we tried bypassing the connection pooling, by creating a new HttpAsyncClientBuilder against a new PoolingNHttpClientConnectionManager for each request. The change is fairly basic, but has enabled Jenkins to stay up so far. The change is on this fork . (Essentially it extracts a method in the constructor, to create the HttpAsyncClientBuilder , then also calls it from doExecute() .) As mentioned, this was just a minimal/dirty change to get us up and running for now - it just bypasses connection reuse at the point of the request. For example an HttpAsyncClient without connection pooling would be cleaner. For reference this is working against JIRA Cloud.  We're not sure why the connections started failing to close, but this was enough to force our builds through for the time being.

          Craig Rodrigues added a comment - - edited

          dfrance can you submit a PR of what you've got working to https://github.com/jenkinsci/jira-plugin ?

          Craig Rodrigues added a comment - - edited dfrance can you submit a PR of what you've got working to https://github.com/jenkinsci/jira-plugin ?

          Taylor Nelson added a comment -

          dfrance this is affecting our team as well – same stack trace as yours. Your fix did unclog our pipelines temporarily, and then we ran into issues regarding JENKINS-56987 sometime last week which completely broke the plugin again.

          The current state of the pull request for 56987 involves deleting all the apache files that you modified in your fork to get the connection to stop freezing. So now we are freezing again. It's been a long few weeks with these plugin problems!

           

          Taylor Nelson added a comment - dfrance this is affecting our team as well – same stack trace as yours. Your fix did unclog our pipelines temporarily, and then we ran into issues regarding JENKINS-56987 sometime last week which completely broke the plugin again. The current state of the pull request for 56987 involves deleting all the apache files that you modified in your fork to get the connection to stop freezing. So now we are freezing again. It's been a long few weeks with these plugin problems!  

          Craig Rodrigues added a comment -

          taylornelson dfrance Take a look at: https://github.com/jenkinsci/jira-plugin/pull/213 and try it out if you want. That is a patch I have which works after the Atlassian GDPR changes in JENKINS-56987 . I'm testing this now against my JIRA Cloud setup.

          One thing I did in that patch was I deleted the imported copy of the ApacheAsyncHttpClient code. I'm not sure if that was the right approach, but it allows me to get a setup that worked in my JIRA Cloud setup.

          Craig Rodrigues added a comment - taylornelson dfrance Take a look at: https://github.com/jenkinsci/jira-plugin/pull/213 and try it out if you want. That is a patch I have which works after the Atlassian GDPR changes in JENKINS-56987 . I'm testing this now against my JIRA Cloud setup. One thing I did in that patch was I deleted the imported copy of the ApacheAsyncHttpClient code. I'm not sure if that was the right approach, but it allows me to get a setup that worked in my JIRA Cloud setup.

          Taylor Nelson added a comment - - edited

          rodrigc Yes I definitely saw your pull request. Thanks for all your work on that! I built the plugin with your PR #213 yesterday morning, but this issue here persists even with your changes.

          We use the plugin for anything from updating a fixVersion to adding comments on tasks, and for some reason these jobs started occasionally freezing and would require us to restart our Jenkins server to unclog the pipeline. (With the exact stack trace David has above – blocked thread cannot close HTTP connection and another thread waiting forever, etc)

          I was going to combine your changes in PR 213 with David's changes on PR 207, but 207 modifies that AsyncHttpClient class that you have deleted in your PR. So now I have to choose to build a plugin that doesn't work, but doesn't freeze OR a plugin that works but freezes and blocks all of our executors – connection freeze fix or GDPR fix, respectively.

          I realize I'm not contributing to the fix, I don't have any experience at all building Jenkins plugins. Just hoping my comments will give more context to the problem we are having. I'm sort of surprised this issue isn't affecting more people.

           

          Edit: I just saw you added some commits related to the apache dependencies being updated. Will try building again

          Taylor Nelson added a comment - - edited rodrigc Yes I definitely saw your pull request. Thanks for all your work on that! I built the plugin with your PR #213 yesterday morning, but this issue here persists even with your changes. We use the plugin for anything from updating a fixVersion to adding comments on tasks, and for some reason these jobs started occasionally freezing and would require us to restart our Jenkins server to unclog the pipeline. (With the exact stack trace David has above – blocked thread cannot close HTTP connection and another thread waiting forever, etc) I was going to combine your changes in PR 213 with David's changes on PR 207, but 207 modifies that AsyncHttpClient class that you have deleted in your PR. So now I have to choose to build a plugin that doesn't work, but doesn't freeze OR a plugin that works but freezes and blocks all of our executors – connection freeze fix or GDPR fix, respectively. I realize I'm not contributing to the fix, I don't have any experience at all building Jenkins plugins. Just hoping my comments will give more context to the problem we are having. I'm sort of surprised this issue isn't affecting more people.   Edit: I just saw you added some commits related to the apache dependencies being updated. Will try building again

          Craig Rodrigues added a comment - - edited

          Cool, let me know how it works for you.

          You can also post messages in the Gitter chat channel https://gitter.im/jenkinsci/jira-plugin

          The more people who collaborate on this the better!

          Craig Rodrigues added a comment - - edited Cool, let me know how it works for you. You can also post messages in the Gitter chat channel https://gitter.im/jenkinsci/jira-plugin The more people who collaborate on this the better!

          Taylor Nelson added a comment -

          I pulled the latest changes from your pull request 213 and built the plugin. Unfortunately it did not work. It looks like you merged David France's changes into your PR (David's changes solved the freezing problem for me before these GDPR problems) so I'm not sure why it's not working now. Maybe 213 is still using the "default" apache dependencies instead of the custom Jira plugin ones?

          Here are the details before I reboot the server again:

           

          The BLOCKED thread:

           

          The WAITING thread:

          Taylor Nelson added a comment - I pulled the latest changes from your pull request 213 and built the plugin. Unfortunately it did not work. It looks like you merged David France's changes into your PR (David's changes solved the freezing problem for me before these GDPR problems) so I'm not sure why it's not working now. Maybe 213 is still using the "default" apache dependencies instead of the custom Jira plugin ones? Here are the details before I reboot the server again:   The BLOCKED thread:   The WAITING thread:

          David France added a comment - - edited

          Just to report, we've tried merging https://github.com/rodrigc/jira-plugin/tree/jira-rest-client.version_5.2.0 across to https://github.com/xft-devs/jira-plugin

          It was able to run a workflow action against Jira Cloud OK again here - the fix for JENKINS-56987 looks good, thanks. Time will tell if it is still stable with the connections...

          The comment about deleting ApacheAsyncHttpClient didn't make sense during the merge. The class was still present on the jira-rest-client.version_5.2.0 branch, so the earlier change came across without a problem.

          For reference, this is building, with the same test failures before and after:

          [ERROR] Failures:
[ERROR] hudson.plugins.jira.JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents(hudson.plugins.jira.JiraCreateIssueNotifierTest)
[ERROR]   Run 1: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 2: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 3: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 4: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR]   Run 5: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[INFO]
[INFO]
[ERROR] Tests run: 199, Failures: 1, Errors: 0, Skipped: 0

          I've not investigated these deeper.

          David France added a comment - - edited Just to report, we've tried merging https://github.com/rodrigc/jira-plugin/tree/jira-rest-client.version_5.2.0 across to https://github.com/xft-devs/jira-plugin It was able to run a workflow action against Jira Cloud OK again here - the fix for JENKINS-56987 looks good, thanks. Time will tell if it is still stable with the connections... The comment about deleting ApacheAsyncHttpClient didn't make sense during the merge. The class was still present on the jira-rest-client.version_5.2.0 branch, so the earlier change came across without a problem. For reference, this is building, with the same test failures before and after: [ERROR] Failures: [ERROR] hudson.plugins.jira.JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents(hudson.plugins.jira.JiraCreateIssueNotifierTest) [ERROR] Run 1: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1> [ERROR] Run 2: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1> [ERROR] Run 3: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1> [ERROR] Run 4: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1> [ERROR] Run 5: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1> [INFO] [INFO] [ERROR] Tests run: 199, Failures: 1, Errors: 0, Skipped: 0 I've not investigated these deeper.

          Taylor Nelson added a comment -

          dfrance how is the plugin working for you now? Any blocked threads/stuck executors?

           

          I merged your changes with Craig's changes and was still getting blocked threads. If it's going smoothly for you, maybe I did the merge incorrectly.

          Taylor Nelson added a comment - dfrance how is the plugin working for you now? Any blocked threads/stuck executors?   I merged your changes with Craig's changes and was still getting blocked threads. If it's going smoothly for you, maybe I did the merge incorrectly.

          David France added a comment -

          The merged version has been fine for the last week - can't see any stuck threads,  To be fair though, we haven't gone back and checked if the freezing behaviour would come back without this.

           

          The merge was pushed up to https://github.com/xft-devs/jira-plugin .  If you haven't already, it may be worth trying pulling or cloning that again.

          David France added a comment - The merged version has been fine for the last week - can't see any stuck threads,  To be fair though, we haven't gone back and checked if the freezing behaviour would come back without this.   The merge was pushed up to https://github.com/xft-devs/jira-plugin  .  If you haven't already, it may be worth trying pulling or cloning that again.

          Hugo SANSON added a comment - - edited

          Bumping up this issue as i believe it is not solved with 3.0.13 nor 3.0.15, 
          I remember we had a fix preparing for this issue before the Atlassian GDPR changes rolled out,
          and it was working for a week with this PR https://github.com/jenkinsci/jira-plugin/pull/207
          but apparently it didn't move forward a notch since then.

           

          Our environement, jenkins 2.204.5 with open jdk 11 - Trying to connect to Jira Cloud with API Token and username

          Sympthoms are 

          First connection after reboot works for validate and jiraSearch

          But all subsequent connections threads hang at this point,  with status BLOCKED on 
          org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605)

          org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#2]
org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605)
org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511)
org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75)
org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101)
org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423)
org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseHttpPromiseAsyncClient.java:38)
com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417)
com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364)
com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297)
com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianHttpClientDecorator.java:83)
com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253)
com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.java:69)
com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.java:109)
com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94)
hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:206)
hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136)
hudson.plugins.jira.pipeline.SearchIssuesStep$SearchStepExecution.run(SearchIssuesStep.java:84)
hudson.plugins.jira.pipeline.SearchIssuesStep$SearchStepExecution.run(SearchIssuesStep.java:61)
org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
hudson.security.ACL.impersonate(ACL.java:290)
org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
java.base@11.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
java.base@11.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:264)
java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@11.0.6/java.lang.Thread.run(Thread.java:834)

          Issue looks similar to taylornelson's 
          Any help would be appreciated, 
          I'm willing to fix and build from source given some pointers

          Hugo SANSON added a comment - - edited Bumping up this issue as i believe it is not solved with 3.0.13 nor 3.0.15,  I remember we had a fix preparing for this issue before the Atlassian GDPR changes rolled out, and it was working for a week with this PR  https://github.com/jenkinsci/jira-plugin/pull/207 but apparently it didn't move forward a notch since then.   Our environement, jenkins  2.204.5  with open jdk 11 - Trying to connect to Jira Cloud with API Token and username Sympthoms are  First connection after reboot works for validate and jiraSearch But all subsequent connections threads hang at this point,  with status BLOCKED on  org.apache.http.nio.reactor.ssl.SSLIOSession.close( SSLIOSession.java:605 ) org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#2] org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605) org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511) org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75) org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101) org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423) org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280) org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295) org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377) org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129) org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75) org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108) com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseHttpPromiseAsyncClient.java:38) com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417) com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364) com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297) com.atlassian.jira. rest .client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianHttpClientDecorator.java:83) com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253) com.atlassian.jira. rest .client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.java:69) com.atlassian.jira. rest .client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.java:109) com.atlassian.jira. rest .client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94) hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:206) hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136) hudson.plugins.jira.pipeline.SearchIssuesStep$SearchStepExecution.run(SearchIssuesStep.java:84) hudson.plugins.jira.pipeline.SearchIssuesStep$SearchStepExecution.run(SearchIssuesStep.java:61) org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) hudson.security.ACL.impersonate(ACL.java:290) org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) java.base@11.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) java.base@11.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base@11.0.6/java.lang. Thread .run( Thread .java:834) Issue looks similar to taylornelson 's  Any help would be appreciated,  I'm willing to fix and build from source given some pointers

          benoit guerin added a comment -

          Agreed, issue is still present

          benoit guerin added a comment - Agreed, issue is still present

          Hugo SANSON added a comment - - edited

          Using this fork you guys made at https://github.com/xft-devs/jira-plugin

          The issue looks fixed or at least
          Our use cases with Jenkins pipeline JiraSearch and JiraIssueUpdater against Jira Cloud work fine.
          No more threads hanging or blocked

           To be able to build this fork, i had to change a few versions in the pom.xml

          configuration-as-code.version from 1.8 to 1.36
jackson2-api from 2.8.11.2 to 2.10.2
io.jenkins.configuration-as-code to io.jenkins.configuration-as-code.test-harness
junit from 1.3 to 1.20
org.jenkins-ci.plugins.credentials from 2.1.19 to 2.2.1

          These versions changes match the https://github.com/jenkinsci/jira-plugin branch feature/4.0 changes

          Hugo SANSON added a comment - - edited Using this fork you guys made at  https://github.com/xft-devs/jira-plugin The issue looks fixed or at least Our use cases with Jenkins pipeline JiraSearch and JiraIssueUpdater against Jira Cloud work fine. No more threads hanging or blocked  To be able to build this fork, i had to change a few versions in the pom.xml configuration-as-code.version from 1.8 to 1.36 jackson2-api from 2.8.11.2 to 2.10.2 io.jenkins.configuration-as-code to io.jenkins.configuration-as-code.test-harness junit from 1.3 to 1.20 org.jenkins-ci.plugins.credentials from 2.1.19 to 2.2.1 These versions changes match the https://github.com/jenkinsci/jira-plugin  branch feature/4.0 changes

          benoit guerin added a comment -

          Same here, modifications from xft-devs solve issue. Thanks to them !

          I rebase theirs changes on master, here is the PR : https://github.com/jenkinsci/jira-plugin/pull/230

          benoit guerin added a comment - Same here, modifications from xft-devs solve issue. Thanks to them ! I rebase theirs changes on master, here is the PR :  https://github.com/jenkinsci/jira-plugin/pull/230

          David Scheidt added a comment -

          Is there any timeline on when the changes in  https://github.com/jenkinsci/jira-plugin/pull/230 will be released?  Not being able to integrate with Jira is big limitation in our workflow.

          David Scheidt added a comment - Is there any timeline on when the changes in   https://github.com/jenkinsci/jira-plugin/pull/230 will be released?  Not being able to integrate with Jira is big limitation in our workflow.

          David France added a comment -

          Pleased to report this was looking stable just now. This was with the 3.0.17 release. Many thanks.

          David France added a comment - Pleased to report this was looking stable just now. This was with the 3.0.17 release. Many thanks.

            olamy Olivier Lamy
            honestem Emma Phillips
            Votes:
            8 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: