I don't see at clear indication that this is related to the git plugin.

Some tests to check more deeply:

• use jgit instead of command line git, does the behavior change inn a relevant way?
• use a different server for scm, soya the behavior change?
• use different plugin versions, soya behaviour change?

I don't see any indication that it is related to git either - but the issue occurs at the handoff point between the SCM system that pulled the script and actually invoking the script so I thought it worth mentioning.

I have tried a different server, as mentioned above (switching from remote git server to local file url).

I have tried different plugin versions, as mentioned above (upgrading plugins over the course of time that this issue has affected us).

I have NOT however tried switching from default CLI git to jgit, will do so.

Can you get a thread dump (i.e., from JENKINS_URL/threadDump) when you've got a hung job?

I will grab one the next time this happens.

The switch to jgit did not change anything, we have a hang and I am parsing the thread dump right now. I'm not familiar with the underlying code, but just reading the stack trace, it looks to me like we are deadlocking as a pair of jobs (both reading the same Jenkinsfile but with different configs) are launched near simultaneously by an SCM poll job.

No other jobs were running at the time, and I was unable to kill either task normally or by script console and had to restart tomcat this time in order to clean them up.

Attached is the thread dump from the moment I identified the problem.

jenkins-thread-dump-20170829.txt

We had another mysterious hang from a pair of jobs a few hours ago, which I did not catch immediately - so the thread dump is from about 3 hours after the pipelines got stuck. This time, there are no obvious stack traces pointing at the individual jobs and one miscellaneous build was running at the time of the snapshot. Additionally, I was able to issue a kill via script console on these (unlike the most recent incident).

I am not seeing anything particularly interesting in this dump. I don't know which worker was running the one build, but win04 was definitely running one of them this time as well as last time.

jenkins-thread-dump-20170905.txt

And another hang this morning. This time it was a different pair of jobs, running on OSX workers instead of Windows. A third job from the same codebase was running happily at the time of this thread dump. Nothing terribly interesting here as far as I can tell at a cursory glance.

jenkins-thread-dump-20170907.txt

I am wondering if the problem is with the Parameterized Trigger plugin. The jobs that lock up are (always?) being launched as a post-build trigger to a sync job. I have updated these parent jobs to use two triggers instead of one in the hope that splitting them up slightly more will help avoid any possible race condition.

I'm also seeing this on some builds today.

Jenkins 2.89.3, all plugins up to date.  git-client plugin is at 2.7.1 - downgrading to 2.7.0 seems to have helped.

Thread dump:

Thread #8
	at DSL.git(running in thread: org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution [#114])
	at WorkflowScript.run(WorkflowScript:38)
	at DSL.ws(Native Method)
	at WorkflowScript.run(WorkflowScript:37)
	at DSL.node(running on buildnode3-2)
	at WorkflowScript.run(WorkflowScript:36)
	at DSL.stage(Native Method)
	at WorkflowScript.run(WorkflowScript:35)
	at DSL.lock(Native Method)
	at WorkflowScript.run(WorkflowScript:32)

The git-client plugin downgrade was not sufficient, I had several builds hang in the same way in the meantime

We also began to experience this same problem recently, just after Jenkins upgrade from 2.100 to 2.103 and an upgrade of several plugins (to many to list or remember).

For the record, we do still observe this issue intermittently - but builds over the new year were less frequent, so it is possible that we just missed the right sets of circumstances due to decreased activity. A week ago, I upgraded the master to 2.101 (and all plugins to current), and have not observed the issue since this particular upgrade.

My current workaround is to avoid using the checkout step (it doesn't scale well enough for my needs anyway) and instead do this manually with an sh step.  I will report back here if this doesn't avoid the issue.

I also have same issue after upgrade to 2.89.3 LTS along with bunch of plugin. Now most of the pipeline job stuck at check out step

I have the same problem after upgrading all the plugins and Core to 2.89.3 LTS

markewaite

{quote}

The JENKINS-45447 and JENKINS-47169 fix in git client plugin 2.7.0 and in git plugin 3.7.0 might be visible as a hang in `git rev-list`. 

{quote}

I was thinking about reverting back to git client plugin 2.6.0 and git plugin 3.6.4.

We have even seen this issue on projects that has no git tags.

djviking if you're seeing the problem on a git repository which has no tags or only a few tags, then the changes in git client plugin 2.7.0 and git plugin 3.7.0 are not relevant to the problem. Reverting to an earlier version of the git plugin and the git client plugin is unlikely to resolve the issue.

We have the same issue with Jenkins 2.105 (first time observed with 2.103), Git plugin 3.7.0 and Git Client Plugin 2.7.1.

The build is running on the OpenShift slave. The code seem to get checked out correctly, but job is stuck.

However, in our case the issue is not intermittent. It always occur when build is triggered after some changes are made to the repository. It happens when it is triggered manually or by the github webhook.

Also, what i noticed is that when the checkout is stuck the last git command in the log has the incorrect commit ID:

Checking out Revision 2d4ce8b023949526bdf7a6dc0658cd1a276f5e10 (master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 2d4ce8b023949526bdf7a6dc0658cd1a276f5e10
Commit message: "JT-1: Forcing checkout"
> git rev-list --no-walk 8f5141bdcb00abddd5961a0dff6589f2654b5a87 # timeout=10

See, that the id of the git rev-list is different from the previous git checkout.

If i abort the stuck build, subsequent manual build works just fine and the commit ID is consistently the same throughout the log.

There is nothing interesting in the thread dump.

We have the same observations as ikolomiyets mentions.

Have attached some generated jstack output from both Jenkins og Slave.

• jstack -F <slave-pid>
• jstack -F <jenkins-pid>

Running with downgraded Git plugin 3.6.4 and Git Client plugin 2.6.0 we are hardly experiencing the problem. It is far less frequent.

A build that didn't hang, where the git commit hash on rev-list where different from checkout.

 > git checkout -f bc4dd5d1939f16e0124251327b0490a0059b353d
Commit message: "Fix something"
 > git rev-list 1ced60c576f7dcd1e030b92fc08a405e4c8a5800 # timeout=10

In my case i was able to fix by downgrade the jira-plugin to 2.5 as there was a bug with the JiraChangeLogAnnotator. If any one upgrade to 2.5.1 should downgrade to 2.5 to fix the issue SEVERE: ChangeLogAnnotator hudson.plugins.jira.JiraChangeLogAnnotator@xxx failed to annotate message.

You're legend! Downgrading jira-plugin to 2.5 did the trick.

Why would the JIRA plugin affect this? We are not even using it in our pipeline.

djviking refer to JENKINS-48357 for details of the impact of Jira plugin 2.5.1. This bug was reported before the release of Jira plugin 2.5.1. I suspect that Jira plugin 2.5.1 related bugs are a "side path" from this bug. The real bug is likely still in the system as reported originally.

markewaite I hit this too - I am trying downgrade to 2.5.0 and will report back if it is all ok after that. This may be a dupe of https://issues.jenkins-ci.org/browse/JENKINS-43106 but I put a support bundle there. in what log did people see the JIRA error? 

I swear that JIRA plugin (and the Atlassian dependencies) are a disaster. 

We have also downgraded JIRA plugin to 2.5.0 and it seems to have resolved the problem.

markewaite I'm downgrading the priority to reflect the fix via JIRA... not clear if this should stay as a pipeline bug though or if we can close it out.

Seeing the same issue here... no luck with the workarounds around downgrading the JRIA plugin