• Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Blocker Blocker
    • git-plugin
    • Windows Server 2008R2; Jenkins 1.54.3; Git Plugin 2.2.1

      We met randomly failure of git scm, it hung at the fetch process for a long time and will timeout. When it timeout it says

      02:56:20 Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress ssh://bmcdiags@.../ghts/ta +refs/heads/:refs/remotes/origin/" returned status code -1:
      02:56:20 stdout:
      02:56:20 stderr: Could not create directory 'c/Users/Administrator/.ssh'.

          [JENKINS-24454] Windows GIT SCM fetch code hung

          sharon xia created issue -

          sharon xia added a comment -

          02:36:16 Started by upstream project "echidna-patch-quality" build number 335
          02:36:16 originally caused by:
          02:36:16 Started by command line by xxx
          02:36:16 [EnvInject] - Loading node environment variables.
          02:36:17 Building remotely on ECHIDNA-QUALITY (6.1 windows-6.1 windows amd64-windows amd64-windows-6.1 amd64) in workspace c:\buildfarm-slave\workspace\echidna-patch-compile
          02:36:18 > git rev-parse --is-inside-work-tree
          02:36:19 Fetching changes from the remote Git repository
          02:36:19 > git config remote.origin.url ssh://@...:/ghts/ta
          02:36:20 Fetching upstream changes from ssh://@...:/ghts/ta
          02:36:20 > git --version
          02:36:20 > git fetch --tags --progress ssh://@...:/ghts/ta +refs/heads/:refs/remotes/origin/
          02:56:20 ERROR: Timeout after 20 minutes
          02:56:20 FATAL: Failed to fetch from ssh://@...:/ghts/ta
          02:56:20 hudson.plugins.git.GitException: Failed to fetch from ssh://bmcdiags@10.110.61.117:30000/ghts/ta
          02:56:20 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623)
          02:56:20 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855)
          02:56:20 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880)
          02:56:20 at hudson.model.AbstractProject.checkout(AbstractProject.java:1414)
          02:56:20 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:671)
          02:56:20 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
          02:56:20 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:580)
          02:56:20 at hudson.model.Run.execute(Run.java:1684)
          02:56:20 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          02:56:20 at hudson.model.ResourceController.execute(ResourceController.java:88)
          02:56:20 at hudson.model.Executor.run(Executor.java:231)
          02:56:20 Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress ssh://@...:/ghts/ta +refs/heads/:refs/remotes/origin/" returned status code -1:
          02:56:20 stdout:
          02:56:20 stderr: Could not create directory 'c/Users/Administrator/.ssh'.
          02:56:20
          02:56:20 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1325)
          02:56:20 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1186)
          02:56:20 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87)
          02:56:20 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:257)
          02:56:20 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
          02:56:20 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
          02:56:20 at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          02:56:20 at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          02:56:20 at hudson.remoting.Request$2.run(Request.java:326)
          02:56:20 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          02:56:20 at java.util.concurrent.FutureTask.run(Unknown Source)
          02:56:20 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          02:56:20 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          02:56:20 at hudson.remoting.Engine$1$1.run(Engine.java:63)
          02:56:20 at java.lang.Thread.run(Unknown Source)

          sharon xia added a comment - 02:36:16 Started by upstream project "echidna-patch-quality" build number 335 02:36:16 originally caused by: 02:36:16 Started by command line by xxx 02:36:16 [EnvInject] - Loading node environment variables. 02:36:17 Building remotely on ECHIDNA-QUALITY (6.1 windows-6.1 windows amd64-windows amd64-windows-6.1 amd64) in workspace c:\buildfarm-slave\workspace\echidna-patch-compile 02:36:18 > git rev-parse --is-inside-work-tree 02:36:19 Fetching changes from the remote Git repository 02:36:19 > git config remote.origin.url ssh:// @...: /ghts/ta 02:36:20 Fetching upstream changes from ssh:// @...: /ghts/ta 02:36:20 > git --version 02:36:20 > git fetch --tags --progress ssh:// @...:/ghts/ta +refs/heads/:refs/remotes/origin/ 02:56:20 ERROR: Timeout after 20 minutes 02:56:20 FATAL: Failed to fetch from ssh:// @...: /ghts/ta 02:56:20 hudson.plugins.git.GitException: Failed to fetch from ssh://bmcdiags@10.110.61.117:30000/ghts/ta 02:56:20 at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:623) 02:56:20 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:855) 02:56:20 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:880) 02:56:20 at hudson.model.AbstractProject.checkout(AbstractProject.java:1414) 02:56:20 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:671) 02:56:20 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88) 02:56:20 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:580) 02:56:20 at hudson.model.Run.execute(Run.java:1684) 02:56:20 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 02:56:20 at hudson.model.ResourceController.execute(ResourceController.java:88) 02:56:20 at hudson.model.Executor.run(Executor.java:231) 02:56:20 Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress ssh:// @...:/ghts/ta +refs/heads/:refs/remotes/origin/ " returned status code -1: 02:56:20 stdout: 02:56:20 stderr: Could not create directory 'c/Users/Administrator/.ssh'. 02:56:20 02:56:20 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1325) 02:56:20 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1186) 02:56:20 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(CliGitAPIImpl.java:87) 02:56:20 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:257) 02:56:20 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) 02:56:20 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) 02:56:20 at hudson.remoting.UserRequest.perform(UserRequest.java:118) 02:56:20 at hudson.remoting.UserRequest.perform(UserRequest.java:48) 02:56:20 at hudson.remoting.Request$2.run(Request.java:326) 02:56:20 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) 02:56:20 at java.util.concurrent.FutureTask.run(Unknown Source) 02:56:20 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) 02:56:20 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) 02:56:20 at hudson.remoting.Engine$1$1.run(Engine.java:63) 02:56:20 at java.lang.Thread.run(Unknown Source)

          Daniel Beck added a comment -

          Not a core issue, and likely not a Git issue either. Have you tried creating the directory Jenkins fails to create? Or tried Google?

          Daniel Beck added a comment - Not a core issue, and likely not a Git issue either. Have you tried creating the directory Jenkins fails to create? Or tried Google?
          Daniel Beck made changes -
          Component/s Original: core [ 15593 ]
          Component/s Original: scm-api [ 18054 ]
          Assignee Original: Nicolas De Loof [ ndeloof ]

          Mark Waite added a comment -

          You might review this stackoverflow posting for ideas of things you might try. It is not exactly the same case, but it was one of the first items when I searched google with that error message.

          Mark Waite added a comment - You might review this stackoverflow posting for ideas of things you might try. It is not exactly the same case, but it was one of the first items when I searched google with that error message.

          Mark Waite added a comment -

          Since you report it is a random failure, and it seems to be Windows specific, you might also try to accelerate the frequency at which you encounter the problem by defining multiple jobs which use the same ssh authenticated repository, with a local reference repository (to reduce the cloning data transfer), then run the jobs concurrently.

          If the problem is a file locking problem with the C:\Users\Administrator\.ssh directory, or with a file in that directory, then running many jobs in parallel should make it happen much more often, and may give you a chance to see other hints which may suggest what is causing the locking problem.

          Mark Waite added a comment - Since you report it is a random failure, and it seems to be Windows specific, you might also try to accelerate the frequency at which you encounter the problem by defining multiple jobs which use the same ssh authenticated repository, with a local reference repository (to reduce the cloning data transfer), then run the jobs concurrently. If the problem is a file locking problem with the C:\Users\Administrator\.ssh directory, or with a file in that directory, then running many jobs in parallel should make it happen much more often, and may give you a chance to see other hints which may suggest what is causing the locking problem.
          Mark Waite made changes -
          Summary Original: GIT SCM Pooling hung New: GIT SCM Polling hung

          sharon xia added a comment -

          You are right Waite! We are running 3 jobs concurrently on the same win 7 slave pulling from the same code source. And two of them will success and one will fail. The failure is random. I have to workaround to make the job running in serial now to see whether it will get rid of this issue. Is there any other log I can provide?

          sharon xia added a comment - You are right Waite! We are running 3 jobs concurrently on the same win 7 slave pulling from the same code source. And two of them will success and one will fail. The failure is random. I have to workaround to make the job running in serial now to see whether it will get rid of this issue. Is there any other log I can provide?

          Mark Waite added a comment -

          The message is coming from the stderr output of the "git fetch" command as far as I can tell. That would usually mean that a change to fix the issue would be needed inside the "git" program, external to Jenkins.

          If the C:\users\Administrator\.ssh directory does not exist, you can create it by logging in as Administrator, and entering the command "ssh-keygen" from a "Git Bash" shell.

          Mark Waite added a comment - The message is coming from the stderr output of the "git fetch" command as far as I can tell. That would usually mean that a change to fix the issue would be needed inside the "git" program, external to Jenkins. If the C:\users\Administrator\.ssh directory does not exist, you can create it by logging in as Administrator, and entering the command "ssh-keygen" from a "Git Bash" shell.

          sharon xia added a comment -

          Of course this file is existed otherwise the other two jobs cannot be succeedd and we will not be able to clone the file. This is the folder for placing the ssh public key and known host.

          The strange thing is we didn't meet this issue before July. The changes we did: upgrade git plugin from 1.1.6 to 2.2.1 and upgrade Jenkins from 1.532.3 to 1.554.3, change the git repository url(I don't think this is related with this issue as the other two jobs clone succeed.)

          sharon xia added a comment - Of course this file is existed otherwise the other two jobs cannot be succeedd and we will not be able to clone the file. This is the folder for placing the ssh public key and known host. The strange thing is we didn't meet this issue before July. The changes we did: upgrade git plugin from 1.1.6 to 2.2.1 and upgrade Jenkins from 1.532.3 to 1.554.3, change the git repository url(I don't think this is related with this issue as the other two jobs clone succeed.)

            Unassigned Unassigned
            sharon_xia sharon xia
            Votes:
            4 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: