Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35694

Make customizable timeouts to fail when GitHub peer is unreachable

      The following stacktrace below happens making Jenkins UI to be stuck.

      "Executor #-1 for master : executing BranchIndexing[<XXXXXXXX>]" #37436 daemon prio=5 os_prio=0 tid=0x00007fc756abb800 nid=0x3177 runnable [0x00007fc7f019a000]
         java.lang.Thread.State: RUNNABLE
          at java.net.SocketInputStream.socketRead0(Native Method)
          at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
          at java.net.SocketInputStream.read(SocketInputStream.java:170)
          at java.net.SocketInputStream.read(SocketInputStream.java:141)
          at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
          at sun.security.ssl.InputRecord.read(InputRecord.java:503)
          at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
          - locked <0x000000076c602138> (a java.lang.Object)
          at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
          at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
          - locked <0x000000076c636868> (a sun.security.ssl.AppInputStream)
          at okio.Okio$2.read(Okio.java:139)
          at okio.AsyncTimeout$2.read(AsyncTimeout.java:211)
          at okio.RealBufferedSource.indexOf(RealBufferedSource.java:306)
          at okio.RealBufferedSource.indexOf(RealBufferedSource.java:300)
          at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196)
          at com.squareup.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:191)
          at com.squareup.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:80)
          at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:906)
          at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:782)
          at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:444)
          at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:389)
          at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:502)
          at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105)
          at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:25)
          at org.kohsuke.github.Requester.parse(Requester.java:486)
          at org.kohsuke.github.Requester._to(Requester.java:243)
          at org.kohsuke.github.Requester._to(Requester.java:251)
          at org.kohsuke.github.Requester._to(Requester.java:251)
          at org.kohsuke.github.Requester._to(Requester.java:251)
          at org.kohsuke.github.Requester.to(Requester.java:205)
          at org.kohsuke.github.GHRepository.getBranches(GHRepository.java:1074)
          at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.doRetrieve(GitHubSCMSource.java:268)
          at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:256)
          at jenkins.scm.api.SCMSource.fetch(SCMSource.java:146)
          at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:296)
          at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:157)
          at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:122)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:410)
      

      As you can see the github_branch_source plugin is stack trying to get the branches from GitHub Enterprise.

      org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.doRetrieve(GitHubSCMSource.java:268)
      

      which you can see calls GHRepository#getBranches

      at org.kohsuke.github.GHRepository.getBranches(GHRepository.java:1074)
      

      On the top of the stacktrace you can see {{ at java.net.SocketInputStream.socketRead0(Native Method)}} which basically means the JVM is waiting on native method to get the branches from GitHub Enterprise.

      java.lang.Thread.State: RUNNABLE
          at java.net.SocketInputStream.socketRead0(Native Method)
      

      I think the issue might be resolved with just a customized Timeout

      https://github.com/square/okhttp/wiki/Recipes#timeouts

          [JENKINS-35694] Make customizable timeouts to fail when GitHub peer is unreachable

          Félix Belzunce Arcos created issue -
          Félix Belzunce Arcos made changes -
          Description Original: The following stacktrace below happens making Jenkins UI to be stuck.

          {code:java}
          "Executor #-1 for master : executing BranchIndexing[<XXXXXXXX>]" #37436 daemon prio=5 os_prio=0 tid=0x00007fc756abb800 nid=0x3177 runnable [0x00007fc7f019a000]
             java.lang.Thread.State: RUNNABLE
              at java.net.SocketInputStream.socketRead0(Native Method)
              at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
              at java.net.SocketInputStream.read(SocketInputStream.java:170)
              at java.net.SocketInputStream.read(SocketInputStream.java:141)
              at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
              at sun.security.ssl.InputRecord.read(InputRecord.java:503)
              at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
              - locked <0x000000076c602138> (a java.lang.Object)
              at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
              at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
              - locked <0x000000076c636868> (a sun.security.ssl.AppInputStream)
              at okio.Okio$2.read(Okio.java:139)
              at okio.AsyncTimeout$2.read(AsyncTimeout.java:211)
              at okio.RealBufferedSource.indexOf(RealBufferedSource.java:306)
              at okio.RealBufferedSource.indexOf(RealBufferedSource.java:300)
              at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196)
              at com.squareup.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:191)
              at com.squareup.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:80)
              at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:906)
              at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:782)
              at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:444)
              at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:389)
              at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:502)
              at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105)
              at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:25)
              at org.kohsuke.github.Requester.parse(Requester.java:486)
              at org.kohsuke.github.Requester._to(Requester.java:243)
              at org.kohsuke.github.Requester._to(Requester.java:251)
              at org.kohsuke.github.Requester._to(Requester.java:251)
              at org.kohsuke.github.Requester._to(Requester.java:251)
              at org.kohsuke.github.Requester.to(Requester.java:205)
              at org.kohsuke.github.GHRepository.getBranches(GHRepository.java:1074)
              at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.doRetrieve(GitHubSCMSource.java:268)
              at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:256)
              at jenkins.scm.api.SCMSource.fetch(SCMSource.java:146)
              at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:296)
              at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:157)
              at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:122)
              at hudson.model.ResourceController.execute(ResourceController.java:98)
              at hudson.model.Executor.run(Executor.java:410)
          {code}

          As you can see the github_branch_source plugin[ is stack trying to get the branches from GitHub Enterprise|https://github.com/jenkinsci/github-branch-source-plugin/blob/38a28e64e3d000f1599645ab10d484edc26df13f/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java#L268].

          {code:java}
          org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.doRetrieve(GitHubSCMSource.java:268)
          {code}

          which you can see calls [GHRepository#getBranches|https://github.com/kohsuke/github-api/blob/github-api-1.75/src/main/java/org/kohsuke/github/GHRepository.java#L1074]

          {code:java}
          at org.kohsuke.github.GHRepository.getBranches(GHRepository.java:1074)
          {code}

          On the top of the stacktrace you can see {{ at java.net.SocketInputStream.socketRead0(Native Method)}} which basically means the JVM is waiting on native method to get the branches from GitHub Enterprise.

          {code:java}

          java.lang.Thread.State: RUNNABLE
              at java.net.SocketInputStream.socketRead0(Native Method)
          {code}

          I think the issue might be resolved with just a customized Timeout

          https://github.com/square/okhttp/wiki/Recipes#timeouts
          New: The following stacktrace below happens making Jenkins UI to be stuck.

          {code:java}
          "Executor #-1 for master : executing BranchIndexing[<XXXXXXXX>]" #37436 daemon prio=5 os_prio=0 tid=0x00007fc756abb800 nid=0x3177 runnable [0x00007fc7f019a000]
             java.lang.Thread.State: RUNNABLE
              at java.net.SocketInputStream.socketRead0(Native Method)
              at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
              at java.net.SocketInputStream.read(SocketInputStream.java:170)
              at java.net.SocketInputStream.read(SocketInputStream.java:141)
              at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
              at sun.security.ssl.InputRecord.read(InputRecord.java:503)
              at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
              - locked <0x000000076c602138> (a java.lang.Object)
              at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
              at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
              - locked <0x000000076c636868> (a sun.security.ssl.AppInputStream)
              at okio.Okio$2.read(Okio.java:139)
              at okio.AsyncTimeout$2.read(AsyncTimeout.java:211)
              at okio.RealBufferedSource.indexOf(RealBufferedSource.java:306)
              at okio.RealBufferedSource.indexOf(RealBufferedSource.java:300)
              at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196)
              at com.squareup.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:191)
              at com.squareup.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:80)
              at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:906)
              at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:782)
              at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:444)
              at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:389)
              at com.squareup.okhttp.internal.huc.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:502)
              at com.squareup.okhttp.internal.huc.DelegatingHttpsURLConnection.getResponseCode(DelegatingHttpsURLConnection.java:105)
              at com.squareup.okhttp.internal.huc.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:25)
              at org.kohsuke.github.Requester.parse(Requester.java:486)
              at org.kohsuke.github.Requester._to(Requester.java:243)
              at org.kohsuke.github.Requester._to(Requester.java:251)
              at org.kohsuke.github.Requester._to(Requester.java:251)
              at org.kohsuke.github.Requester._to(Requester.java:251)
              at org.kohsuke.github.Requester.to(Requester.java:205)
              at org.kohsuke.github.GHRepository.getBranches(GHRepository.java:1074)
              at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.doRetrieve(GitHubSCMSource.java:268)
              at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.retrieve(GitHubSCMSource.java:256)
              at jenkins.scm.api.SCMSource.fetch(SCMSource.java:146)
              at jenkins.branch.MultiBranchProject.computeChildren(MultiBranchProject.java:296)
              at com.cloudbees.hudson.plugins.folder.computed.ComputedFolder.updateChildren(ComputedFolder.java:157)
              at com.cloudbees.hudson.plugins.folder.computed.FolderComputation.run(FolderComputation.java:122)
              at hudson.model.ResourceController.execute(ResourceController.java:98)
              at hudson.model.Executor.run(Executor.java:410)
          {code}

          As you can see the github_branch_source plugin [is stack trying to get the branches from GitHub Enterprise|https://github.com/jenkinsci/github-branch-source-plugin/blob/38a28e64e3d000f1599645ab10d484edc26df13f/src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java#L268].

          {code:java}
          org.jenkinsci.plugins.github_branch_source.GitHubSCMSource.doRetrieve(GitHubSCMSource.java:268)
          {code}

          which you can see calls [GHRepository#getBranches|https://github.com/kohsuke/github-api/blob/github-api-1.75/src/main/java/org/kohsuke/github/GHRepository.java#L1074]

          {code:java}
          at org.kohsuke.github.GHRepository.getBranches(GHRepository.java:1074)
          {code}

          On the top of the stacktrace you can see {{ at java.net.SocketInputStream.socketRead0(Native Method)}} which basically means the JVM is waiting on native method to get the branches from GitHub Enterprise.

          {code:java}

          java.lang.Thread.State: RUNNABLE
              at java.net.SocketInputStream.socketRead0(Native Method)
          {code}

          I think the issue might be resolved with just a customized Timeout

          https://github.com/square/okhttp/wiki/Recipes#timeouts
          Félix Belzunce Arcos made changes -
          Priority Original: Blocker [ 1 ] New: Major [ 3 ]
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 172120 ] New: JNJira + In-Review [ 184570 ]

            fbelzunc Félix Belzunce Arcos
            fbelzunc Félix Belzunce Arcos
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: