-
Bug
-
Resolution: Fixed
-
Critical
-
OS: Amazon Linux2 CPE: cpe:2.3:o:amazon:amazon_linux:2
Java Version: OpenJDK Runtime Environment Corretto-11.0.5.10.1 (build 11.0.5+10-LTS)
Jenkins & plugin versions: See attached PDF
About Jenkins installation:
Jenkins (master) is installed as a service on an AWS EC2 instance with an EBS backed volume for Jenkins data.
Jenkins master then sits behind a load balancer
We use AWS EC2 Cloud plugin for build nodes
Things tried:
* Updated plugins that had available pluginsOS: Amazon Linux2 CPE: cpe:2.3:o:amazon:amazon_linux:2 Java Version: OpenJDK Runtime Environment Corretto-11.0.5.10.1 (build 11.0.5+10-LTS) Jenkins & plugin versions: See attached PDF About Jenkins installation: Jenkins (master) is installed as a service on an AWS EC2 instance with an EBS backed volume for Jenkins data. Jenkins master then sits behind a load balancer We use AWS EC2 Cloud plugin for build nodes Things tried: * Updated plugins that had available plugins
Recently Jenkins master when a job is triggered it hangs around checkout
Command that it seems to get stuck on:
git rev-list --no-walk a4539ebbe8fe6da843fb1fe4c7b0ce5cf79a0647 # timeout=10
This issue only started happening sporadically from 2019-12-17 and is impacting our CI/CD pipeline. We rely on Jenkins to deploy to our environments.
If you tried to kill the job, it is unresponsive.
If you restart Jenkins master then the job that gets stuck is no longer exists
Job Build Logs:
Started by upstream project "parent-job" build number 5 originally caused by: Branch indexing Checking out git <REPO> into /var/lib/jenkins/workspace/acceptance-tests@script to read acceptance-tests/Jenkinsfile-local using credential jenkins-git > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url git@bitbucket.org:obillex/fundit.git # timeout=10 Fetching upstream changes from git@bitbucket.org:obillex/fundit.git > git --version # timeout=10 using GIT_SSH to set credentials Jenkins access to git > git fetch --tags --progress -- <REPO>+refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse b6a1b7a159b86dd5a52b8385670397d568345c53^{commit} # timeout=10 Checking out Revision b6a1b7a159b86dd5a52b8385670397d568345c53 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f b6a1b7a159b86dd5a52b8385670397d568345c53 # timeout=10 Commit message: "Commit message" > git rev-list --no-walk a4539ebbe8fe6da843fb1fe4c7b0ce5cf79a0647 # timeout=10
This issue seems to be a regression of this resolved issue JENKINS-43106
- is related to
-
-
- Closed
-
- links to
[JENKINS-60536] Jenkins job hangs in Jira call after git rev-list command
Just to report, we've tried merging https://github.com/rodrigc/jira-plugin/tree/jira-rest-client.version_5.2.0 across to https://github.com/xft-devs/jira-plugin
It was able to run a workflow action against Jira Cloud OK again here - the fix for JENKINS-56987 looks good, thanks. Time will tell if it is still stable with the connections...
The comment about deleting ApacheAsyncHttpClient didn't make sense during the merge. The class was still present on the jira-rest-client.version_5.2.0 branch, so the earlier change came across without a problem.
For reference, this is building, with the same test failures before and after:
[ERROR] Failures: [ERROR] hudson.plugins.jira.JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents(hudson.plugins.jira.JiraCreateIssueNotifierTest) [ERROR] Run 1: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1> [ERROR] Run 2: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1> [ERROR] Run 3: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1> [ERROR] Run 4: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1> [ERROR] Run 5: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1> [INFO] [INFO] [ERROR] Tests run: 199, Failures: 1, Errors: 0, Skipped: 0
I've not investigated these deeper.
David France
added a comment - - edited Just to report, we've tried merging https://github.com/rodrigc/jira-plugin/tree/jira-rest-client.version_5.2.0 across to https://github.com/xft-devs/jira-plugin
It was able to run a workflow action against Jira Cloud OK again here - the fix for JENKINS-56987 looks good, thanks. Time will tell if it is still stable with the connections...
The comment about deleting ApacheAsyncHttpClient didn't make sense during the merge. The class was still present on the jira-rest-client.version_5.2.0 branch, so the earlier change came across without a problem.
For reference, this is building, with the same test failures before and after:
[ERROR] Failures:
[ERROR] hudson.plugins.jira.JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents(hudson.plugins.jira.JiraCreateIssueNotifierTest)
[ERROR] Run 1: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR] Run 2: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR] Run 3: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR] Run 4: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[ERROR] Run 5: JiraCreateIssueNotifierTest.performFailureSuccessIssueClosedWithComponents:192 expected:<0> but was:<1>
[INFO]
[INFO]
[ERROR] Tests run: 199, Failures: 1, Errors: 0, Skipped: 0
I've not investigated these deeper. 
Taylor Nelson
added a comment - dfrance how is the plugin working for you now? Any blocked threads/stuck executors?
I merged your changes with Craig's changes and was still getting blocked threads. If it's going smoothly for you, maybe I did the merge incorrectly.
Taylor Nelson
added a comment - dfrance how is the plugin working for you now? Any blocked threads/stuck executors?
I merged your changes with Craig's changes and was still getting blocked threads. If it's going smoothly for you, maybe I did the merge incorrectly. David France
added a comment - The merged version has been fine for the last week - can't see any stuck threads, To be fair though, we haven't gone back and checked if the freezing behaviour would come back without this.
The merge was pushed up to https://github.com/xft-devs/jira-plugin . If you haven't already, it may be worth trying pulling or cloning that again.
David France
added a comment - The merged version has been fine for the last week - can't see any stuck threads, To be fair though, we haven't gone back and checked if the freezing behaviour would come back without this.
The merge was pushed up to https://github.com/xft-devs/jira-plugin . If you haven't already, it may be worth trying pulling or cloning that again. Bumping up this issue as i believe it is not solved with 3.0.13 nor 3.0.15,
I remember we had a fix preparing for this issue before the Atlassian GDPR changes rolled out,
and it was working for a week with this PR https://github.com/jenkinsci/jira-plugin/pull/207
but apparently it didn't move forward a notch since then.
Our environement, jenkins 2.204.5 with open jdk 11 - Trying to connect to Jira Cloud with API Token and username
Sympthoms are
First connection after reboot works for validate and jiraSearch
But all subsequent connections threads hang at this point, with status BLOCKED on
org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605)
org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#2] org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605) org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511) org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75) org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101) org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423) org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280) org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295) org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377) org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129) org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141) org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75) org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108) com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseHttpPromiseAsyncClient.java:38) com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417) com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364) com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297) com.atlassian.jira.rest.client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianHttpClientDecorator.java:83) com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253) com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.java:69) com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.java:109) com.atlassian.jira.rest.client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94) hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:206) hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136) hudson.plugins.jira.pipeline.SearchIssuesStep$SearchStepExecution.run(SearchIssuesStep.java:84) hudson.plugins.jira.pipeline.SearchIssuesStep$SearchStepExecution.run(SearchIssuesStep.java:61) org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) hudson.security.ACL.impersonate(ACL.java:290) org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) java.base@11.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) java.base@11.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:264) java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base@11.0.6/java.lang.Thread.run(Thread.java:834)
Issue looks similar to taylornelson's
Any help would be appreciated,
I'm willing to fix and build from source given some pointers
Hugo SANSON
added a comment - - edited Bumping up this issue as i believe it is not solved with 3.0.13 nor 3.0.15,
I remember we had a fix preparing for this issue before the Atlassian GDPR changes rolled out,
and it was working for a week with this PR https://github.com/jenkinsci/jira-plugin/pull/207
but apparently it didn't move forward a notch since then.
Our environement, jenkins 2.204.5 with open jdk 11 - Trying to connect to Jira Cloud with API Token and username
Sympthoms are
First connection after reboot works for validate and jiraSearch
But all subsequent connections threads hang at this point, with status BLOCKED on
org.apache.http.nio.reactor.ssl.SSLIOSession.close( SSLIOSession.java:605 )
org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#2]
org.apache.http.nio.reactor.ssl.SSLIOSession.close(SSLIOSession.java:605)
org.apache.http.impl.nio.NHttpConnectionBase.close(NHttpConnectionBase.java:511)
org.apache.http.impl.nio.conn.CPoolEntry.closeConnection(CPoolEntry.java:75)
org.apache.http.impl.nio.conn.CPoolEntry.close(CPoolEntry.java:101)
org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractNIOConnPool.java:423)
org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:280)
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestConnection(PoolingNHttpClientConnectionManager.java:295)
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnection(AbstractClientExchangeHandler.java:377)
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(DefaultClientExchangeHandlerImpl.java:129)
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:141)
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:75)
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHttpAsyncClient.java:108)
com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.execute(SettableFuturePromiseHttpPromiseAsyncClient.java:38)
com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.doExecute(ApacheAsyncHttpClient.java:417)
com.atlassian.httpclient.apache.httpcomponents.ApacheAsyncHttpClient.execute(ApacheAsyncHttpClient.java:364)
com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.execute(DefaultRequest.java:297)
com.atlassian.jira. rest .client.internal.async.AtlassianHttpClientDecorator$AuthenticatedRequestBuilder.execute(AtlassianHttpClientDecorator.java:83)
com.atlassian.httpclient.apache.httpcomponents.DefaultRequest$DefaultRequestBuilder.get(DefaultRequest.java:253)
com.atlassian.jira. rest .client.internal.async.AbstractAsynchronousRestClient.getAndParse(AbstractAsynchronousRestClient.java:69)
com.atlassian.jira. rest .client.internal.async.AsynchronousSearchRestClient.searchJqlImplGet(AsynchronousSearchRestClient.java:109)
com.atlassian.jira. rest .client.internal.async.AsynchronousSearchRestClient.searchJql(AsynchronousSearchRestClient.java:94)
hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:206)
hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136)
hudson.plugins.jira.pipeline.SearchIssuesStep$SearchStepExecution.run(SearchIssuesStep.java:84)
hudson.plugins.jira.pipeline.SearchIssuesStep$SearchStepExecution.run(SearchIssuesStep.java:61)
org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
hudson.security.ACL.impersonate(ACL.java:290)
org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
java.base@11.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
java.base@11.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:264)
java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base@11.0.6/java.lang. Thread .run( Thread .java:834)
Issue looks similar to taylornelson 's
Any help would be appreciated,
I'm willing to fix and build from source given some pointers 
benoit guerin
added a comment - Agreed, issue is still present Using this fork you guys made at https://github.com/xft-devs/jira-plugin
The issue looks fixed or at least
Our use cases with Jenkins pipeline JiraSearch and JiraIssueUpdater against Jira Cloud work fine.
No more threads hanging or blocked
To be able to build this fork, i had to change a few versions in the pom.xml
configuration-as-code.version from 1.8 to 1.36 jackson2-api from 2.8.11.2 to 2.10.2 io.jenkins.configuration-as-code to io.jenkins.configuration-as-code.test-harness junit from 1.3 to 1.20 org.jenkins-ci.plugins.credentials from 2.1.19 to 2.2.1
These versions changes match the https://github.com/jenkinsci/jira-plugin branch feature/4.0 changes
Hugo SANSON
added a comment - - edited Using this fork you guys made at https://github.com/xft-devs/jira-plugin
The issue looks fixed or at least
Our use cases with Jenkins pipeline JiraSearch and JiraIssueUpdater against Jira Cloud work fine.
No more threads hanging or blocked
To be able to build this fork, i had to change a few versions in the pom.xml
configuration-as-code.version from 1.8 to 1.36
jackson2-api from 2.8.11.2 to 2.10.2
io.jenkins.configuration-as-code to io.jenkins.configuration-as-code.test-harness
junit from 1.3 to 1.20
org.jenkins-ci.plugins.credentials from 2.1.19 to 2.2.1
These versions changes match the https://github.com/jenkinsci/jira-plugin branch feature/4.0 changes benoit guerin
added a comment - Same here, modifications from xft-devs solve issue. Thanks to them !
I rebase theirs changes on master, here is the PR : https://github.com/jenkinsci/jira-plugin/pull/230
benoit guerin
added a comment - Same here, modifications from xft-devs solve issue. Thanks to them !
I rebase theirs changes on master, here is the PR : https://github.com/jenkinsci/jira-plugin/pull/230 
David Scheidt
added a comment - Is there any timeline on when the changes in https://github.com/jenkinsci/jira-plugin/pull/230 will be released? Not being able to integrate with Jira is big limitation in our workflow.
David Scheidt
added a comment - Is there any timeline on when the changes in https://github.com/jenkinsci/jira-plugin/pull/230 will be released? Not being able to integrate with Jira is big limitation in our workflow. David France
added a comment - Pleased to report this was looking stable just now. This was with the 3.0.17 release. Many thanks.
David France
added a comment - Pleased to report this was looking stable just now. This was with the 3.0.17 release. Many thanks. 
I pulled the latest changes from your pull request 213 and built the plugin. Unfortunately it did not work. It looks like you merged David France's changes into your PR (David's changes solved the freezing problem for me before these GDPR problems) so I'm not sure why it's not working now. Maybe 213 is still using the "default" apache dependencies instead of the custom Jira plugin ones?
Here are the details before I reboot the server again:
The BLOCKED thread:
The WAITING thread: