Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-28551

Git fetch fails randomly with status code 128

XMLWordPrintable

      We have configured a multi phase job where any changes in Git SCM will trigger two jobs. Among the two jobs any one of the job getting failed randomly while fetching the git changes. And, during the next build it's getting passed. Here is my error log.

      11:41:52 Started by upstream project "1_CPU_Xtreme_Feedback" build number 3092
11:41:52 originally caused by:
11:41:52  Started by an SCM change
11:41:52 [EnvInject] - Loading node environment variables.
11:41:52 Building remotely on kcebuilder2 in workspace c:\jenkins_keivmcsi\workspace\CPU_XF_ARM9
11:41:55 Fetching changes from the remote Git repository
11:41:55 Fetching upstream changes from git:lift
11:42:14 FATAL: Failed to fetch from git:lift
11:42:14 hudson.plugins.git.GitException: Failed to fetch from git:lift
11:42:14 	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:612)
11:42:14 	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:836)
11:42:14 	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:861)
11:42:14 	at hudson.model.AbstractProject.checkout(AbstractProject.java:1411)
11:42:14 	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:652)
11:42:14 	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
11:42:14 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:561)
11:42:14 	at hudson.model.Run.execute(Run.java:1665)
11:42:14 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
11:42:14 	at hudson.model.ResourceController.execute(ResourceController.java:88)
11:42:14 	at hudson.model.Executor.run(Executor.java:246)
11:42:14 Caused by: hudson.plugins.git.GitException: Command "fetch -t git:lift +refs/heads/*:refs/remotes/origin/*" returned status code 128:
11:42:14 stdout: 
11:42:14 stderr: ssh: Could not resolve hostname git: no address associated with name
11:42:14 fatal: Could not read from remote repository.
11:42:14 
11:42:14 Please make sure you have the correct access rights
11:42:14 and the repository exists.
11:42:14 
11:42:14 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:928)
11:42:14 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:902)
11:42:14 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.fetch(CliGitAPIImpl.java:163)
11:42:14 	at hudson.plugins.git.GitAPI.fetch(GitAPI.java:229)
11:42:14 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
11:42:14 	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
11:42:14 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
11:42:14 	at java.lang.reflect.Method.invoke(Unknown Source)
11:42:14 	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:299)
11:42:14 	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:280)
11:42:14 	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:239)
11:42:14 	at hudson.remoting.UserRequest.perform(UserRequest.java:118)
11:42:14 	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
11:42:14 	at hudson.remoting.Request$2.run(Request.java:328)
11:42:14 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
11:42:14 	at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
11:42:14 	at java.util.concurrent.FutureTask.run(Unknown Source)
11:42:14 	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
11:42:14 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
11:42:14 	at java.lang.Thread.run(Unknown Source)

          [JENKINS-28551] Git fetch fails randomly with status code 128

          Karthic Sivaswamy created issue -

          Karthic Sivaswamy added a comment -

          If the upstream job is started in "node 1" and it triggers two jobs, if both the job starts running in "node 2", at least one job fails with this message. When triggered manually both the jobs are running successfully next time.

          Karthic Sivaswamy added a comment - If the upstream job is started in "node 1" and it triggers two jobs, if both the job starts running in "node 2", at least one job fails with this message. When triggered manually both the jobs are running successfully next time.

          Mark Waite added a comment -

          The message "could not resolve hostname" seems to hint that your DNS server is sometimes not responding to the request to convert a hostname to an IP address. It seems unlikely that the git plugin or the git client plugin can help with unreliable DNS service.

          If you need to confirm the DNS service reliability, you could write a batch command (Windows) or shell script (Linux) to check that your DNS server consistently and reliably returns addresses throughout the day. If your DNS server is outside your control, you could consider installing a caching DNS server on each slave so that they will remember the successfully resolved names and return them from the cache, rather than asking your central DNS server.

          Mark Waite added a comment - The message "could not resolve hostname" seems to hint that your DNS server is sometimes not responding to the request to convert a hostname to an IP address. It seems unlikely that the git plugin or the git client plugin can help with unreliable DNS service. If you need to confirm the DNS service reliability, you could write a batch command (Windows) or shell script (Linux) to check that your DNS server consistently and reliably returns addresses throughout the day. If your DNS server is outside your control, you could consider installing a caching DNS server on each slave so that they will remember the successfully resolved names and return them from the cache, rather than asking your central DNS server.
          Mark Waite made changes -
          Resolution New: Not A Defect [ 7 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]

          Karthic Sivaswamy added a comment -

          But, out of two jobs which starts simultaneously only one job is getting this error and the other job is getting executed successfully. Both are started in the same node. Why did both the jobs didn't get the same message "could not resolve hostname"? And, If I start the upstream job also in the same node I am not getting this error.

          Karthic Sivaswamy added a comment - But, out of two jobs which starts simultaneously only one job is getting this error and the other job is getting executed successfully. Both are started in the same node. Why did both the jobs didn't get the same message "could not resolve hostname"? And, If I start the upstream job also in the same node I am not getting this error.

          Mark Waite added a comment -

          If the DNS server is not responding reliably, then difference could happen at any time, either within the same job or between different jobs when the DNS server is asked to convert a hostname to an IP address.

          Have you attempted to change the git source code URL from a hostname based URL to an IP address based URL in order to confirm that the DNS server is the problem?

          Since the message that is being reported is coming from inside the command line git program, there is not much the git plugin can do for you except report the message.

          You could also try JGit by adding it as a git implementation in the system configuration, then switch to use that implementation in the job definition. If the problem is intermittent misbehavior of the DNS server, then both git and JGit should be affected, since they both require hostname to IP address conversion.

          Mark Waite added a comment - If the DNS server is not responding reliably, then difference could happen at any time, either within the same job or between different jobs when the DNS server is asked to convert a hostname to an IP address. Have you attempted to change the git source code URL from a hostname based URL to an IP address based URL in order to confirm that the DNS server is the problem? Since the message that is being reported is coming from inside the command line git program, there is not much the git plugin can do for you except report the message. You could also try JGit by adding it as a git implementation in the system configuration, then switch to use that implementation in the job definition. If the problem is intermittent misbehavior of the DNS server, then both git and JGit should be affected, since they both require hostname to IP address conversion.

          Karthic Sivaswamy added a comment - - edited

          Now I configured all the jobs to be run on a single "Node", I am still getting these errors. Now three out of one jobs fails with this error.

          Actually this is our git config file, we are using IP address only

          {{Host git
          HostName 10.8.152.132
          User git
          identityFile ~/.ssh/my_git_key}}

          I will try your suggestions and let you know. Thanks.

          Karthic Sivaswamy added a comment - - edited Now I configured all the jobs to be run on a single "Node", I am still getting these errors. Now three out of one jobs fails with this error. Actually this is our git config file, we are using IP address only {{Host git HostName 10.8.152.132 User git identityFile ~/.ssh/my_git_key}} I will try your suggestions and let you know. Thanks.
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 163396 ] New: JNJira + In-Review [ 197189 ]
          Mark Waite made changes -
          Status Original: Resolved [ 5 ] New: Closed [ 6 ]

            ndeloof Nicolas De Loof
            karthic_s Karthic Sivaswamy
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: