git operations occasionally hang in Jenkins on Windows

XMLWordPrintable

Details

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Cannot Reproduce
• Component/s:
• Labels:
• Environment:
• Similar Issues:

Description

See elaborate description, dumps and discussion at:
http://stackoverflow.com/questions/10732940/git-operations-occasionally-hang-in-jenkins-on-windows

Basically, there is a Jenkins environment with some jobs based on GIT with periodic polling.
Each of the main GIT operations in the system (periodic polling, the pull at the beginning of a job, and the tag push at the end of a job) all sometimes hang, causing builds not to happen.

I suspect that the GIT plugin is holding handles to some GIT internal files, and this locks out the GIT command-line program and causes it to hang at the beginning of a run.
This suspicion is based on a different issue I encountered - I could not wipe the workspace for one of the jobs because a GIT pack file was locked. It turns out that the process locking the file was Tomcat itself and not the GIT process as I expected.

Activity

Hide
Mark Waite added a comment -

Based on the significant changes in git-client-plugin and git-plugin to support credentials and to support JGit, and the lack of an activity on this bug for multiple months, I'm closing the bug.

Show
Mark Waite added a comment - Based on the significant changes in git-client-plugin and git-plugin to support credentials and to support JGit, and the lack of an activity on this bug for multiple months, I'm closing the bug.
Hide
ikedam added a comment - - edited

In my environment, hangs is not reproduced since introducing following configuration:

• Create a batch file
@echo off
ssh -i C:\path\to\id_rsa -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=3 -oServerAliveInterval=1 -oServerAliveCountMax=3 %*

• Set GIT_SSH to the path to the file above.

My environemnt is:
OS: Windows7/64bit
Git: msysgit 1.8.0-preview20121022(32bit)
Git plugin: 1.5.0

I want to know whether this also works for others environments. Please try it.

Show
ikedam added a comment - - edited In my environment, hangs is not reproduced since introducing following configuration: Create a batch file @echo off ssh -i C:\path\to\id_rsa -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=3 -oServerAliveInterval=1 -oServerAliveCountMax=3 %* Set GIT_SSH to the path to the file above. My environemnt is: OS: Windows7/64bit Git: msysgit 1.8.0-preview20121022(32bit) Git plugin: 1.5.0 I want to know whether this also works for others environments. Please try it.
Hide

It also happens in my environment:
OS: Windows7/64bit
Git: msysgit 1.8.0-preview20121022(32bit)
Path to git: cmd/git.exe
Node: slave (master is linux)
Scheme for git: ssh
Authentication: key pairs without a passphrase(set HOME environment variable)
When it happens: polling, SCM checkout in builds
Note: It seems happen when multiple git.exe (and ssh.exe) runs.

It may be caused not by git itself, but by ssh.
I cannot see what is happening in backgrounds, for I cannot get outputs from git or ssh... it may be a kind of race conditions, for it happens when multiple git processes run.

I'll try workarounds I've ever heard.

• use plink.exe by setting GIT_SSH
• use bin/git.exe instead of cmd/git.exe

I also think these are worth to try:

• Wrap git with a script for excluding execution.
• Add a timeout feature to git plugin.
Show
ikedam added a comment - It also happens in my environment: OS: Windows7/64bit Git: msysgit 1.8.0-preview20121022(32bit) Path to git: cmd/git.exe Node: slave (master is linux) Scheme for git: ssh Authentication: key pairs without a passphrase(set HOME environment variable) When it happens: polling, SCM checkout in builds Note: It seems happen when multiple git.exe (and ssh.exe) runs. It may be caused not by git itself, but by ssh. I cannot see what is happening in backgrounds, for I cannot get outputs from git or ssh... it may be a kind of race conditions, for it happens when multiple git processes run. I'll try workarounds I've ever heard. use plink.exe by setting GIT_SSH use bin/git.exe instead of cmd/git.exe I also think these are worth to try: Wrap git with a script for excluding execution. Add a timeout feature to git plugin.
Hide
Juraj Michalek added a comment - - edited

Solution suggested by VonC works for me (http://stackoverflow.com/questions/10732940/git-operations-occasionally-hang-in-jenkins-on-windows):

• Setting the Path to Git executable to C:\Program Files (x86)\Git\cmd\git.exe (i.e. cmd and not bin!)
Show
Juraj Michalek added a comment - - edited Solution suggested by VonC works for me ( http://stackoverflow.com/questions/10732940/git-operations-occasionally-hang-in-jenkins-on-windows): Setting the Path to Git executable to C:\Program Files (x86)\Git\cmd\git.exe (i.e. cmd and not bin!)
Hide
Guillaume Collic added a comment -

We've also hit the issue while on Jenkins "1.485", GIT plugin "1.1.24", and msysgit "1.7.11.msysgit.1", using git.cmd

Show
Guillaume Collic added a comment - We've also hit the issue while on Jenkins "1.485", GIT plugin "1.1.24", and msysgit "1.7.11.msysgit.1", using git.cmd
Hide

We've also hit the issue while on Jenkins 1.467 looking at changing to GIT plugin 1.1.19 - 1.1.22

I think the below might be a duplicate of this one.
https://issues.jenkins-ci.org/browse/JENKINS-14151

Show
VladDRussian added a comment - We've also hit the issue while on Jenkins 1.467 looking at changing to GIT plugin 1.1.19 - 1.1.22 I think the below might be a duplicate of this one. https://issues.jenkins-ci.org/browse/JENKINS-14151
Hide
Theral Mackey added a comment -

We hit the same problem, project builds will randomly fail and once one project fails it will not build again until the processes holding the git pack files locked are killed off so the files can be removed. Other projects continue building fine. In our case, this involved one git.exe process and an ssh.exe process (win7-64 w/cygwin). Once removed the Jenkins process was restarted and the build that was failing built successfully.

Show
Theral Mackey added a comment - We hit the same problem, project builds will randomly fail and once one project fails it will not build again until the processes holding the git pack files locked are killed off so the files can be removed. Other projects continue building fine. In our case, this involved one git.exe process and an ssh.exe process (win7-64 w/cygwin). Once removed the Jenkins process was restarted and the build that was failing built successfully.

People

Assignee:
Nicolas De Loof
Reporter:
Oran t