Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67466

Hangs and never recovers when TCP error on remote

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • git-client-plugin
    • None
    • 3.11.0

      Today on ci.jenkins.io we've experienced multiple 'hangs' where jobs never proceed after hours.

      e.g. https://ci.jenkins.io/blue/organizations/jenkins/Tools%2Fbom/detail/PR-795/4/pipeline
      Logs: (7 hours ago at this point)

      [2021-12-27T14:48:00.386Z] Pull request #795 updated
[2021-12-27T14:48:00.453Z] Connecting to https://api.github.com using ci.jenkins.io GitHub App (jenkinsci org)
[2021-12-27T14:48:01.205Z] Obtained Jenkinsfile from 7ce33ff966f9150c1826dff1ccada5a73fcc481e+2432c3eefe69ba91c698754de319685125be30a0 (e9c0a8de9981992d798bb50a0fe397f7bd9aacde)
[2021-12-27T14:48:01.229Z] Loading library pipeline-library@master
[2021-12-27T14:48:01.229Z] Attempting to resolve master from remote references...
[2021-12-27T14:48:01.491Z] Found match: refs/heads/master revision ebad8ebd6007bd97e909e16e14b9ebf1d3893ddd
[2021-12-27T14:48:01.493Z] using credential app-ci.jenkins.io-jenkinsinfra
[2021-12-27T14:48:01.508Z] Fetching changes from the remote Git repository
[2021-12-27T14:48:01.508Z] Fetching without tags

      Stack trace from thread:

       Executor #-1 for Built-In Node : executing Infra/infra-reports/masterrunning , holding [ 0x000000065f5007a0 0x000000065f530468 0x00000006675d3780 ] Executor #-1 for Built-In Node : executing Tools/bom/PR-795running , holding [ 0x0000000653100370 0x000000065313d488 0x000000065e91cdd0 ]at java.net.SocketInputStream.socketRead0(java.base@11.0.13/Native Method)
at java.net.SocketInputStream.socketRead(java.base@11.0.13/SocketInputStream.java:115)
at java.net.SocketInputStream.read(java.base@11.0.13/SocketInputStream.java:168)
at java.net.SocketInputStream.read(java.base@11.0.13/SocketInputStream.java:140)
at sun.security.ssl.SSLSocketInputRecord.read(java.base@11.0.13/SSLSocketInputRecord.java:478)
at sun.security.ssl.SSLSocketInputRecord.readHeader(java.base@11.0.13/SSLSocketInputRecord.java:472)
at sun.security.ssl.SSLSocketInputRecord.decode(java.base@11.0.13/SSLSocketInputRecord.java:160)
at sun.security.ssl.SSLTransport.decode(java.base@11.0.13/SSLTransport.java:111)
at sun.security.ssl.SSLSocketImpl.decode(java.base@11.0.13/SSLSocketImpl.java:1426)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(java.base@11.0.13/SSLSocketImpl.java:1336)
at sun.security.ssl.SSLSocketImpl.startHandshake(java.base@11.0.13/SSLSocketImpl.java:450)
at sun.security.ssl.SSLSocketImpl.startHandshake(java.base@11.0.13/SSLSocketImpl.java:421)
at sun.net.www.protocol.https.HttpsClient.afterConnect(java.base@11.0.13/HttpsClient.java:572)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(java.base@11.0.13/AbstractDelegateHttpsURLConnection.java:197)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(java.base@11.0.13/HttpURLConnection.java:1592)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(java.base@11.0.13/HttpURLConnection.java:1520)
at java.net.HttpURLConnection.getResponseCode(java.base@11.0.13/HttpURLConnection.java:527)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(java.base@11.0.13/HttpsURLConnectionImpl.java:334)
at org.eclipse.jgit.transport.http.JDKHttpConnection.getResponseCode(JDKHttpConnection.java:85)
at org.eclipse.jgit.util.HttpSupport.response(HttpSupport.java:205)
at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:654)
at org.eclipse.jgit.transport.TransportHttp.openFetch(TransportHttp.java:465)
at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:142)
at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:94)
at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1309)
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:213)
at org.jenkinsci.plugins.gitclient.JGitAPIImpl$2.execute(JGitAPIImpl.java:631)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:999)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1242)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1302)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$doRetrieve$1(SCMSourceRetriever.java:198)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever$$Lambda$1533/0x0000000801de4840.call(Unknown Source)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:148)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:197)
at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:137)
at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:196)
at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:571)
at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:523)
at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:334)
at hudson.model.ResourceController.execute(ResourceController.java:99)
at hudson.model.Executor.run(Executor.java:432)

      markewaite and dduportal investigated at the time and reported connectivity issues to the GitHub API at the time.

      The job never completes or times out.
      I'm not sure what a normal timeout should be, some repositories can take awhile, but it basically turns Jenkins into a zombie full of stuck threads in the current state

            markewaite Mark Waite
            timja Tim Jacomb
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: