Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18879

Collecting finbugs analysis results randomly fails with exception

    XMLWordPrintable

Details

    Description

      It started some weeks ago that jobs with long running findbugs tasks started to fail while collecting the results. So every morning I have to restart a job or two to get everything back to blue.

      This exception seems to be random. It only ever occurs while the findbugs plugin is running and not during any other static code analysis. Any plugin that runs afterwards fails with a similar message that the connection is closed.

      Another strange thing is that afterwards I can't see the slave log anymore. All I get is the wait image on the page but no log ever loads again until a restart of the master. A simple restart of the slave node does not change anything although jobs are running fine.

      ERROR: Publisher hudson.plugins.findbugs.FindBugsPublisher aborted due to exception
hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Sorry, this connection is closed.
	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41)
	at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34)
	at hudson.remoting.Request.call(Request.java:174)
	at hudson.remoting.Channel.call(Channel.java:713)
	at hudson.FilePath.act(FilePath.java:895)
	at hudson.FilePath.act(FilePath.java:879)
	at hudson.plugins.findbugs.FindBugsPublisher.perform(FindBugsPublisher.java:161)
	at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:144)
	at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:334)
	at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:776)
	at hudson.model.Build$BuildExecution.post2(Build.java:183)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726)
	at hudson.model.Run.execute(Run.java:1618)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:247)
Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Sorry, this connection is closed.
	at hudson.remoting.Request.abort(Request.java:299)
	at hudson.remoting.Channel.terminate(Channel.java:773)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
Caused by: java.io.IOException: Sorry, this connection is closed.
	at com.trilead.ssh2.transport.TransportManager.sendMessage(TransportManager.java:642)
	at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:378)
	at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:953)
	at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58)
	at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2308)
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2321)
	at java.io.ObjectInputStream$BlockDataInputStream.readUnsignedShort(ObjectInputStream.java:2804)
	at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:2862)
	at java.io.ObjectInputStream.readString(ObjectInputStream.java:1636)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1339)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
	at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1704)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1342)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
	at java.io.ObjectInputStream.defaultReadObject(ObjectInputStream.java:499)
	at java.lang.Throwable.readObject(Throwable.java:913)
	at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1891)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1989)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1913)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1796)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1348)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
	at hudson.remoting.Command.readFrom(Command.java:92)
	at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:72)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
Caused by: java.io.IOException: Assertion error: sendMessage may never be invoked by the receiver thread!
	at com.trilead.ssh2.transport.TransportManager.sendMessage(TransportManager.java:634)
	at com.trilead.ssh2.channel.Channel.freeupWindow(Channel.java:378)
	at com.trilead.ssh2.channel.Channel$Output.write(Channel.java:97)
	at com.trilead.ssh2.channel.ChannelManager.msgChannelExtendedData(ChannelManager.java:858)
	at com.trilead.ssh2.channel.ChannelManager.handleMessage(ChannelManager.java:1517)
	at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:780)
	at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:475)
	at java.lang.Thread.run(Thread.java:724)

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-19619 Collecting findbugs analysis results occasionally causes ssh slave to go offline causing job to abort

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-19619 Collecting findbugs analysis results occasionally causes ssh slave to go offline causing job to abort

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-18781 Configurable channel timeout for slaves

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-18836 java.io.IOException: Unexpected termination of the channel

          • Major - Major loss of function.
          • Resolved

          Activity

            Ascending order - Click to sort in descending order
            View 22 older comments
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #2938
            [FIXED JENKINS-18836][FIXED JENKINS-18879][FIXED JENKINS-19619] Upgrade trilead-ssh to version with the fix (Revision bb265c5e95b0fe39128720b903914236962db41b)

            Result = UNSTABLE
            Stephen Connolly : bb265c5e95b0fe39128720b903914236962db41b
            Files :

            • changelog.html
            • core/pom.xml
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #2938 [FIXED JENKINS-18836] [FIXED JENKINS-18879] [FIXED JENKINS-19619] Upgrade trilead-ssh to version with the fix (Revision bb265c5e95b0fe39128720b903914236962db41b) Result = UNSTABLE Stephen Connolly : bb265c5e95b0fe39128720b903914236962db41b Files : changelog.html core/pom.xml
            rbaradari Ramin Baradari added a comment -

            I upgraded our installation to Jenkins ver. 1.536. I will report back in a few days if it is fixed.

            rbaradari Ramin Baradari added a comment - I upgraded our installation to Jenkins ver. 1.536. I will report back in a few days if it is fixed.
            rbaradari Ramin Baradari added a comment -

            The builds have been stable during the last week.

            rbaradari Ramin Baradari added a comment - The builds have been stable during the last week.
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            core/pom.xml
            http://jenkins-ci.org/commit/jenkins/1bb06ada301496ebed6d212188d1b7c9d006317b
            Log:
            [FIXED JENKINS-18836][FIXED JENKINS-18879][FIXED JENKINS-19619] Upgrade trilead-ssh to version with the fix

            (cherry picked from commit bb265c5e95b0fe39128720b903914236962db41b)

            Conflicts:
            changelog.html

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/1bb06ada301496ebed6d212188d1b7c9d006317b Log: [FIXED JENKINS-18836] [FIXED JENKINS-18879] [FIXED JENKINS-19619] Upgrade trilead-ssh to version with the fix (cherry picked from commit bb265c5e95b0fe39128720b903914236962db41b) Conflicts: changelog.html
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            src/com/trilead/ssh2/channel/Channel.java
            http://jenkins-ci.org/commit/trilead-ssh2/5811ddd7ae15670a4f9ad345352613b3f2f2db97
            Log:
            JENKINS-22938 SSH slave connections die after the slave outputs 4MB of stderr, usually during findbugs analysis

            The fix for JENKINS-18836, JENKINS-18879, JENKINS-19619 was incorrect in its analysis.

            • There is no call to getChannelData() on the new code path, so thus you cannot have two calls of freeupWindow()
            • The problem with the original call to freeupWindow() is that it is on the receiver thread. You should not mix the responsibilities. Blocking the receiver thread to send a message will negatively impact performance and connection stability.
            • The correct solution is to push the freeupWindow onto the async queue thus the ACK gets sent and the purity of the receiving thread can be maintained.
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/com/trilead/ssh2/channel/Channel.java http://jenkins-ci.org/commit/trilead-ssh2/5811ddd7ae15670a4f9ad345352613b3f2f2db97 Log: JENKINS-22938 SSH slave connections die after the slave outputs 4MB of stderr, usually during findbugs analysis The fix for JENKINS-18836 , JENKINS-18879 , JENKINS-19619 was incorrect in its analysis. There is no call to getChannelData() on the new code path, so thus you cannot have two calls of freeupWindow() The problem with the original call to freeupWindow() is that it is on the receiver thread. You should not mix the responsibilities. Blocking the receiver thread to send a message will negatively impact performance and connection stability. The correct solution is to push the freeupWindow onto the async queue thus the ACK gets sent and the purity of the receiving thread can be maintained.

            People

              kohsuke Kohsuke Kawaguchi
              rbaradari Ramin Baradari
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: