-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
Windows 2008 64-bit
Tomcat 6 (runs as a Windows service with a special "build" domain user)
Jenkins 1.461
Git plugin version 1.1.17
Git version 1.7.10-preview20120409
Remote repository is Assembla
The "mydomain\build" user is properly configured with SSH keys (command-line access works 100%, jenkins works except when it hangs)Windows 2008 64-bit Tomcat 6 (runs as a Windows service with a special "build" domain user) Jenkins 1.461 Git plugin version 1.1.17 Git version 1.7.10-preview20120409 Remote repository is Assembla The "mydomain\build" user is properly configured with SSH keys (command-line access works 100%, jenkins works except when it hangs)
See elaborate description, dumps and discussion at:
http://stackoverflow.com/questions/10732940/git-operations-occasionally-hang-in-jenkins-on-windows
Basically, there is a Jenkins environment with some jobs based on GIT with periodic polling.
Each of the main GIT operations in the system (periodic polling, the pull at the beginning of a job, and the tag push at the end of a job) all sometimes hang, causing builds not to happen.
I suspect that the GIT plugin is holding handles to some GIT internal files, and this locks out the GIT command-line program and causes it to hang at the beginning of a run.
This suspicion is based on a different issue I encountered - I could not wipe the workspace for one of the jobs because a GIT pack file was locked. It turns out that the process locking the file was Tomcat itself and not the GIT process as I expected.
- is related to
-
-
- Closed
-
[JENKINS-14046] git operations occasionally hang in Jenkins on Windows
VladDRussian
added a comment - We've also hit the issue while on Jenkins 1.467 looking at changing to GIT plugin 1.1.19 - 1.1.22
I think the below might be a duplicate of this one.
https://issues.jenkins-ci.org/browse/JENKINS-14151
VladDRussian
added a comment - We've also hit the issue while on Jenkins 1.467 looking at changing to GIT plugin 1.1.19 - 1.1.22
I think the below might be a duplicate of this one.
https://issues.jenkins-ci.org/browse/JENKINS-14151 We've also hit the issue while on Jenkins "1.485", GIT plugin "1.1.24", and msysgit "1.7.11.msysgit.1", using git.cmd
Solution suggested by VonC works for me (http://stackoverflow.com/questions/10732940/git-operations-occasionally-hang-in-jenkins-on-windows):
- Setting the Path to Git executable to C:\Program Files (x86)\Git\cmd\git.exe (i.e. cmd and not bin!)
Juraj Michalek
added a comment - - edited Solution suggested by VonC works for me ( http://stackoverflow.com/questions/10732940/git-operations-occasionally-hang-in-jenkins-on-windows):
Setting the Path to Git executable to C:\Program Files (x86)\Git\cmd\git.exe (i.e. cmd and not bin!)
ikedam
added a comment - It also happens in my environment:
OS: Windows7/64bit
Git: msysgit 1.8.0-preview20121022(32bit)
Path to git: cmd/git.exe
Node: slave (master is linux)
Scheme for git: ssh
Authentication: key pairs without a passphrase(set HOME environment variable)
When it happens: polling, SCM checkout in builds
Note: It seems happen when multiple git.exe (and ssh.exe) runs.
It may be caused not by git itself, but by ssh.
I cannot see what is happening in backgrounds, for I cannot get outputs from git or ssh... it may be a kind of race conditions, for it happens when multiple git processes run.
I'll try workarounds I've ever heard.
- use plink.exe by setting GIT_SSH
- use bin/git.exe instead of cmd/git.exe
I also think these are worth to try:
- Wrap git with a script for excluding execution.
- Add a timeout feature to git plugin.
ikedam
added a comment - It also happens in my environment:
OS: Windows7/64bit
Git: msysgit 1.8.0-preview20121022(32bit)
Path to git: cmd/git.exe
Node: slave (master is linux)
Scheme for git: ssh
Authentication: key pairs without a passphrase(set HOME environment variable)
When it happens: polling, SCM checkout in builds
Note: It seems happen when multiple git.exe (and ssh.exe) runs.
It may be caused not by git itself, but by ssh.
I cannot see what is happening in backgrounds, for I cannot get outputs from git or ssh... it may be a kind of race conditions, for it happens when multiple git processes run.
I'll try workarounds I've ever heard.
use plink.exe by setting GIT_SSH
use bin/git.exe instead of cmd/git.exe
I also think these are worth to try:
Wrap git with a script for excluding execution.
Add a timeout feature to git plugin.
In my environment, hangs is not reproduced since introducing following configuration:
- Create a batch file
@echo off ssh -i C:\path\to\id_rsa -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=3 -oServerAliveInterval=1 -oServerAliveCountMax=3 %*
- Set GIT_SSH to the path to the file above.
My environemnt is:
OS: Windows7/64bit
Git: msysgit 1.8.0-preview20121022(32bit)
Git plugin: 1.5.0
I want to know whether this also works for others environments. Please try it.
ikedam
added a comment - - edited In my environment, hangs is not reproduced since introducing following configuration:
Create a batch file
@echo off
ssh -i C:\path\to\id_rsa -oBatchMode=yes -oStrictHostKeyChecking=no -oConnectTimeout=3 -oServerAliveInterval=1 -oServerAliveCountMax=3 %*
Set GIT_SSH to the path to the file above.
My environemnt is:
OS: Windows7/64bit
Git: msysgit 1.8.0-preview20121022(32bit)
Git plugin: 1.5.0
I want to know whether this also works for others environments. Please try it. Based on the significant changes in git-client-plugin and git-plugin to support credentials and to support JGit, and the lack of an activity on this bug for multiple months, I'm closing the bug.
We hit the same problem, project builds will randomly fail and once one project fails it will not build again until the processes holding the git pack files locked are killed off so the files can be removed. Other projects continue building fine. In our case, this involved one git.exe process and an ssh.exe process (win7-64 w/cygwin). Once removed the Jenkins process was restarted and the build that was failing built successfully.