-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 1.473 on Debian
-
Powered by SuggestiMate
From time to time I reveive error when getting SCM changes from Git repository and the option "SCM checkout retry count" (I have set it to 3) is not working.
ERROR: Problem fetching from origin / origin - could be unavailable. Continuing anyway
hudson.plugins.git.GitException: Command "git fetch -t git@github.com:xxx/xxx +refs/heads/:refs/remotes/origin/" returned status code 128:
stdout:
stderr: ssh_exchange_identification: Connection closed by remote host
fatal: The remote end hung up unexpectedly
at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:818)
at hudson.plugins.git.GitAPI.launchCommand(GitAPI.java:780)
at hudson.plugins.git.GitAPI.fetch(GitAPI.java:196)
at hudson.plugins.git.GitAPI.fetch(GitAPI.java:1026)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:812)
at hudson.plugins.git.GitSCM.access$100(GitSCM.java:90)
at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1096)
at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1064)
at hudson.FilePath.act(FilePath.java:842)
at hudson.FilePath.act(FilePath.java:824)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1064)
at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:117)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1249)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:589)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:494)
at hudson.model.Run.execute(Run.java:1488)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)
ERROR: Could not fetch from any repository
FATAL: Could not fetch from any repository
hudson.plugins.git.GitException: Could not fetch from any repository
at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1103)
at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1064)
at hudson.FilePath.act(FilePath.java:842)
at hudson.FilePath.act(FilePath.java:824)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1064)
at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:117)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1249)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:589)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:494)
at hudson.model.Run.execute(Run.java:1488)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:236)
[JENKINS-14575] SCM checkout retry count not working
I'm also seeing this with Jenkins ver. 1.480.1 on Debian.
This is the error I get:
08:46:36 Cloning the remote Git repository
08:46:36 Cloning repository ssh://git@git.normation.com/repo.git
08:46:36 git --version
08:46:36 git version 1.7.2.5
08:46:57 ERROR: Error cloning remote repo 'origin' : Could not clone ssh://git@git.normation.com/repo.git
08:46:57 hudson.plugins.git.GitException: Could not clone ssh://git@git.normation.com/repo.git
08:46:57 at hudson.plugins.git.GitAPI.clone(GitAPI.java:273)
08:46:57 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1044)
08:46:57 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:986)
08:46:57 at hudson.FilePath.act(FilePath.java:842)
08:46:57 at hudson.FilePath.act(FilePath.java:824)
08:46:57 at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:986)
08:46:57 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1142)
08:46:57 at hudson.model.AbstractProject.checkout(AbstractProject.java:1256)
08:46:57 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:589)
08:46:57 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
08:46:57 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:494)
08:46:57 at hudson.model.Run.execute(Run.java:1502)
08:46:57 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
08:46:57 at hudson.model.ResourceController.execute(ResourceController.java:88)
08:46:57 at hudson.model.Executor.run(Executor.java:236)
08:46:57 Caused by: hudson.plugins.git.GitException: Command "git clone --progress -o origin ssh://git@git.normation.com/repo.git /var/lib/jenkins/jobs/rudder-server-packages-2.5-branch-RPM-SLES11/workspace" returned status code 128:
08:46:57 stdout: Cloning into /var/lib/jenkins/jobs/rudder-server-packages-2.5-branch-RPM-SLES11/workspace...
08:46:57
08:46:57 stderr: ssh: connect to host git.normation.com port 22: Connection timed out
08:46:57 fatal: The remote end hung up unexpectedly
08:46:57
08:46:57 at hudson.plugins.git.GitAPI.launchCommandIn(GitAPI.java:897)
08:46:57 at hudson.plugins.git.GitAPI.access$000(GitAPI.java:42)
08:46:57 at hudson.plugins.git.GitAPI$1.invoke(GitAPI.java:269)
08:46:57 at hudson.plugins.git.GitAPI$1.invoke(GitAPI.java:248)
08:46:57 at hudson.FilePath.act(FilePath.java:842)
08:46:57 at hudson.FilePath.act(FilePath.java:824)
08:46:57 at hudson.plugins.git.GitAPI.clone(GitAPI.java:248)
08:46:57 ... 14 more
08:46:57 Trying next repository
08:46:57 ERROR: Could not clone repository
08:46:57 FATAL: Could not clone
08:46:57 hudson.plugins.git.GitException: Could not clone
08:46:57 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:1056)
08:46:57 at hudson.plugins.git.GitSCM$2.invoke(GitSCM.java:986)
08:46:57 at hudson.FilePath.act(FilePath.java:842)
08:46:57 at hudson.FilePath.act(FilePath.java:824)
08:46:57 at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:986)
08:46:57 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1142)
08:46:57 at hudson.model.AbstractProject.checkout(AbstractProject.java:1256)
08:46:57 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:589)
08:46:57 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
08:46:57 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:494)
08:46:57 at hudson.model.Run.execute(Run.java:1502)
08:46:57 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
08:46:57 at hudson.model.ResourceController.execute(ResourceController.java:88)
08:46:57 at hudson.model.Executor.run(Executor.java:236)
I am prepared to sponsor this: http://www.freedomsponsors.org/core/offer/214/scm-checkout-retry-count-not-working
git plugin 1.3.0 uses JGit to fetch (in combination with git-client 1.0.4), can you give it a try ?
I tried git plugin and this seems to have made things worse:
I now get this exception when cloning on a windows 2003 machine:
FATAL: Invalid path: installation/full/service.vmoptions
org.eclipse.jgit.dircache.InvalidPathException: Invalid path: installation/full/service.vmoptions
at org.eclipse.jgit.dircache.DirCacheCheckout.processEntry(DirCacheCheckout.java:526)
at org.eclipse.jgit.dircache.DirCacheCheckout.preScanTwoTrees(DirCacheCheckout.java:248)
at org.eclipse.jgit.dircache.DirCacheCheckout.doCheckout(DirCacheCheckout.java:405)
at org.eclipse.jgit.dircache.DirCacheCheckout.checkout(DirCacheCheckout.java:391)
at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:240)
at org.jenkinsci.plugins.gitclient.JGitAPIImpl.checkout(JGitAPIImpl.java:64)
at hudson.plugins.git.GitAPI.checkout(GitAPI.java:208)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1261)
at hudson.plugins.git.GitSCM.access$1200(GitSCM.java:57)
at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1220)
at hudson.plugins.git.GitSCM$4.invoke(GitSCM.java:1196)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2348)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:287)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at hudson.remoting.Engine$1$1.run(Engine.java:60)
at java.lang.Thread.run(Unknown Source)
On a windows 2008 server the clone process stalls after:
Using strategy: Default
Fetching changes from 1 remote Git repository
Ubuntu based servers in the same multi configuration build are cloning fine.
This solves the issue:
https://github.com/jenkinsci/git-plugin/pull/143
We're waiting for the merge.
Code changed in jenkins
User: Adam PAPAI
Path:
src/main/java/hudson/plugins/git/GitSCM.java
http://jenkins-ci.org/commit/git-plugin/d82a9f3c06e33d5d547a7b1e15cf7c72b7773c58
Log:
JENKINS-14575 Wrap GitException into IOException
The retry logic does not catch the GitException, so we have to wrap into a IOException. Clone and Fetch also affected. As of now the scmRetry will work with the git plugin.
Code changed in jenkins
User: Adam PAPAI
Path:
src/main/java/hudson/plugins/git/GitSCM.java
http://jenkins-ci.org/commit/git-plugin/33e66e1bfae8ddaca3fd799394461f3328f2358f
Log:
JENKINS-14575 Wrap GitException into IOException
Do the same if the repo exists and don't have to clone.
Code changed in jenkins
User: Adam PAPAI
Path:
src/main/java/hudson/plugins/git/GitSCM.java
http://jenkins-ci.org/commit/git-plugin/3f135dad0b3739ea5a26deaff970877c8f6bae30
Log:
JENKINS-14575 We don't need to catch it here.
Code changed in jenkins
User: Nicolas De loof
Path:
.gitignore
src/main/java/hudson/plugins/git/GitSCM.java
http://jenkins-ci.org/commit/git-plugin/f49bc41caf1def82d40aef3f6cbe08547cff48d2
Log:
Merge pull request #143 from woohgit/master
JENKINS-14575 Fix to make the retry logic work with GitSCM
Compare: https://github.com/jenkinsci/git-plugin/compare/086f1851e0bd...f49bc41caf1d
I confirmed the retry happens as expected with git-client-plugin 1.0.6 and git-plugin 1.4.0.
Doesn't work anymore with git-plugin v>= 2.0.
The same problem as earlier, gets fixed by changing throwing from GitException -> IOException.
git/src/main/java/hudson/plugins/git/GitSCM.java:629:
catch (GitException ex)
{ throw new IOException("Failed to fetch from "+url.toString(), ex); }Doesn't work anymore with git-plugin v>= 2.0.
The same problem as earlier, gets fixed by changing throwing from GitException -> IOException.
git/src/main/java/hudson/plugins/git/GitSCM.java:629:
catch (GitException ex)
Sent PR and this time I added a test as well. So I won't happen again hopefully.
SCM checkout retry count isn’t working. I set it to 3 after upgrading the GIT client plugins.
Following are the details:
Jenkins core is at 1.547
Git server plugin is at 1.3
Jenkins GIT client plugin is at 1.8.0
Jenkins GIT plugin is at 2.2.1
Just to note that it isn't working for me either - using Jenkins 1.558, Git Client Plugin 1.9.1, Git Plugin 2.2.1.
Sadly, we do get the occasional ssh connect failure, and we can't seem to find a way to fix it by itself, so retrying the checkout is the only way for us to get more reliable builds.
We still run into this issue - we're using git plugin v3.8.0
e.g. while loading a shared pipeline library on a multibranch-pipeline
We got error "unable to access [shared pipeline repo git repo url]: The requested URL returned error: 504
WorkflowScript: Loading libraries failed
No retry is done to load the shared pipeline library, while SCM retry count is set to 5
Same issue here. Here's my code snippet:
stage('checkout') { steps { retry(env.TRY_COUNT) { timeout(time: 45, unit: 'SECONDS') { checkout scm } } } }
Environment:
- Linux
- Jenkins 2.150.1
- Pipeline plugin 2.5
- Pipeline: Declarative plugin 1.3.2
- git repository
- Git plugin 3.9.1
- Git Client plugin 2.7.3
- Git Pipeline for Blue Ocean plugin 1.5.4
What happens is:
- when the checkout scm fails due to a time-out, it just falls through the retry
- in other words: the retry is not even attempted
We have intermittent connection issues between our Jenkins instance on one server and our Bitbucket/Git server which is in a different region. We've been unable to rectify this issue, but the retry should work fine as we've tested this with a manual retry script outside of Jenkins. The SCM retry count doesn't work for us, but would be a good containment method if it did.
- Git plugin 4.6.0
- Jenkins 2.263.4 running in Docker
Hi all, I'm also facing the same issue. In fact, my issue is exactly the same as this one: JENKINS-39194. In other words, the "SCM checkout retry count" works fine for freestyle jobs, but not for pipeline jobs that use the "Pipeline script from SCM" option. I'm using workflow-cps plugin version 2.74 and Git plugin version 3.12.0.
According to the linked ticket, the fix should be in for the version I'm using. Perhaps there has been a regression since then? Has anyone managed to implement a workaround to implement retries on pipeline jobs?
I'm also facing the same issue with the Jenkins multi branch pipelines. The below workaround works for me.
for (int retryCount = 1; retryCount < 6; retryCount++) { try { checkout([$class: 'GitSCM', branches: [[name: branch ]], doGenerateSubmoduleConfigurations: false, extensions: [ [$class: 'CloneOption', noTags: false, reference: referenceRepo, shallow: shallow], [$class: 'UserIdentity', email: 'jenkins@local.demo.com', name: 'Jenkins' ] ], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'scm-ssh-key', name: repo.name, refspec: '+' + refspec + ':refs/remotes/' + repo.name + '/' + branch, url: repo.getURL() ]] ]) break; } catch (err) { if (retryCount == 5){ error "${err}" } else { echo "code checkout failed with exception ${err}. retrying..." // Retry after 15 seconds sleep (time: 15, unit: 'SECONDS') } } }
I tried testing the "Retry Count" feature with Jenkins 1.499 on a Windows Server 2008 R2 Enterprise server using Subversion and it never retried the checkout with the retry count set to 1. The Quiet Period value was set to 10 seconds. I tried it again with the Retry Count set to 2 and it still failed to try the checkout again.