Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39452

build blocked , looks related to com.splunk.splunkjenkins.utils.LogConsumer

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Cannot Reproduce
    • Component/s: splunk-devops-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 1.658 , Splunk plugin 1.3
    • Similar Issues:

      Description

      We've sporadically been getting builds hanging, I just noticed that we simultaneously see log events for the LogConsumer. Could this be blocking the build?

      2016-11-02 12:51:58.499-0500 [id=203] WARNING c.s.s.utils.LogConsumer#run: content length:314java.net.SocketException: Connection reset
      at java.net.SocketInputStream.read(SocketInputStream.java:209)
      at java.net.SocketInputStream.read(SocketInputStream.java:141)
      at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
      at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
      at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
      at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
      at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
      at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
      at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
      at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
      at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
      at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
      at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
      at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
      at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:220)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:164)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:139)
      at com.splunk.splunkjenkins.utils.LogConsumer.run(LogConsumer.java:84)
      2016-11-02 12:51:58.499-0500 [id=203] SEVERE c.s.s.utils.LogConsumer#run: will resend the message:{raw data

      {event_tag=audit_trail, message=aborted job job/jobby/job/buildy/job/buildy-Build/208/, user=Bar, Foo (...}
      2016-11-02 12:51:58.499-0500 [id=202] WARNING c.s.s.utils.LogConsumer#run: content length:60531java.net.SocketException: Connection reset
      at java.net.SocketInputStream.read(SocketInputStream.java:209)
      at java.net.SocketInputStream.read(SocketInputStream.java:141)
      at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
      at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
      at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
      at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
      at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
      at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
      at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
      at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
      at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
      at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
      at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
      at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
      at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
      at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
      at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:220)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:164)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:139)
      at com.splunk.splunkjenkins.utils.LogConsumer.run(LogConsumer.java:84)
      2016-11-02 12:51:58.500-0500 [id=202] SEVERE c.s.s.utils.LogConsumer#run: will resend the message:{length:60531 2016-11-02T12:42:58.057-0500 line:2407 [main] INFO org.apache.maven.cli.event.ExecutionEventLogger - Scanning for projects...
      2016-11-02 12:51:58.600-0500 [id=203] SEVERE c.s.s.utils.LogConsumer#run: failed to send {raw data{num_executors=4, event_tag=slave, connect_time=2016-10-26T21:58:01Z, node_name=Jenkins-Slave1, running_builds=[job/jobby/j...}
      2016-11-02 12:51:58.600-0500 [id=203] SEVERE c.s.s.utils.LogConsumer#run: failed to send {raw data{event_tag=audit_trail, message=aborted job job/jobby/job/buildy/job/buildy-Build/208/, user=Bar, Foo (...}

      2016-11-02 12:51:58.600-0500 [id=203] SEVERE c.s.s.utils.LogConsumer#run: failed to send {length:60531 2016-11-02T12:42:58.057-0500 line:2407 [main] INFO org.apache.maven.cli.event.ExecutionEventLogger - Scanning for projects...

        Attachments

          Activity

          Hide
          fengxx Ted Xiao added a comment - - edited

          LogConsumer is running on non-blocking mode, the log shows the network between Jenkins master and Splunk server is not stable, and connection was reset (the plugin will resend the data, not a issue).

          I don't have detail background about Issue, just my 2 cents based on my Jenkins admin experience:
          Maybe the connection between the Jenkins master and Slave(job was running on) was also got reset, the job will hang for 30 minutes until socket timeout.
          refer to https://github.com/jenkinsci/remoting/blob/remoting-2.56/src/main/java/hudson/remoting/Engine.java#L391 (jenkins 1.658 shipped with slave.jar v2.56)

          Show
          fengxx Ted Xiao added a comment - - edited LogConsumer is running on non-blocking mode, the log shows the network between Jenkins master and Splunk server is not stable, and connection was reset (the plugin will resend the data, not a issue). I don't have detail background about Issue, just my 2 cents based on my Jenkins admin experience: Maybe the connection between the Jenkins master and Slave(job was running on) was also got reset, the job will hang for 30 minutes until socket timeout. refer to https://github.com/jenkinsci/remoting/blob/remoting-2.56/src/main/java/hudson/remoting/Engine.java#L391 (jenkins 1.658 shipped with slave.jar v2.56)
          Hide
          davebo Dave Bottger added a comment -

          I don't have a good way to reliably recreate this, so I will trust your instincts and experience and chalk it up to a flaky connection between master and slave.

          Show
          davebo Dave Bottger added a comment - I don't have a good way to reliably recreate this, so I will trust your instincts and experience and chalk it up to a flaky connection between master and slave.

            People

            Assignee:
            fengxx Ted Xiao
            Reporter:
            davebo Dave Bottger
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: