[JENKINS-23471] Channel reader thread consumes all CPUs - Jenkins Jira

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Major
Resolution: Fixed
Component/s: core
Labels:
- 1.580.1-fixed
- remoting

Similar Issues:

Show

Description

On ci.jenkins-ci.org I noticed that there are a lot of channel reader threads claiming to be connected to remote-slave-6.

ps output indicates that these threads are eating up CPUs, even though the method names suggest they should be blocking.

"Channel reader thread: remote-slave-6" prio=10 tid=0x0000000004be5800 nid=0x1f59 runnable [0x00007fb6722e1000]
   java.lang.Thread.State: RUNNABLE
        at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:77)
        at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
        at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:68)
        at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:93)
        at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

"Channel reader thread: remote-slave-6" prio=10 tid=0x0000000004be4000 nid=0x71d8 runnable [0x00007fb673ffe000]
   java.lang.Thread.State: RUNNABLE
        at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79)
        at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
        at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:68)
        at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:93)
        at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

"Channel reader thread: remote-slave-6" prio=10 tid=0x0000000004e48800 nid=0x4edc runnable [0x00007fb6719d8000]
   java.lang.Thread.State: RUNNABLE
        at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79)
        at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
        at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:67)
        at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:93)
        at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

"Channel reader thread: remote-slave-6" prio=10 tid=0x0000000004df2000 nid=0x406d runnable [0x00007fb67b432000]
   java.lang.Thread.State: RUNNABLE
        at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:79)
        at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
        at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:68)
        at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:93)
        at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:33)
        at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
        at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

threads.txt
305 kB
2014-06-17 16:54

Activity

Ascending order - Click to sort in descending order

View 5 older comments

SCM/JIRA link daemon added a comment - 2014-09-05 21:37

Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
src/main/java/hudson/remoting/ChunkedInputStream.java
http://jenkins-ci.org/commit/remoting/83da718ab06b2f881aa08434684a02ea95ff5135
Log:
[FIXED JENKINS-23471]

if readUntilBreak hits stream EOF, it would end up hanging in a busy loop. EOF is a break boundary,
so it should cause the method to return gracefully.

SCM/JIRA link daemon added a comment - 2014-09-05 21:37 Code changed in jenkins User: Kohsuke Kawaguchi Path: src/main/java/hudson/remoting/ChunkedInputStream.java http://jenkins-ci.org/commit/remoting/83da718ab06b2f881aa08434684a02ea95ff5135 Log: [FIXED JENKINS-23471] if readUntilBreak hits stream EOF, it would end up hanging in a busy loop. EOF is a break boundary, so it should cause the method to return gracefully.

SCM/JIRA link daemon added a comment - 2014-09-15 18:16

Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
changelog.html
pom.xml
http://jenkins-ci.org/commit/jenkins/8fc609fe0952b285d5b26a59fd5ff4c29704d33d
Log:
[JENKINS-23471 JENKINS-24050]

Integrated the fix in remoting to Jenkins 1.580.

SCM/JIRA link daemon added a comment - 2014-09-15 18:16 Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/8fc609fe0952b285d5b26a59fd5ff4c29704d33d Log: [JENKINS-23471 JENKINS-24050] Integrated the fix in remoting to Jenkins 1.580.

Olaf Lenz added a comment - 2014-09-16 09:04

Will the patch also be included into the LTS?

Olaf Lenz added a comment - 2014-09-16 09:04 Will the patch also be included into the LTS?

SCM/JIRA link daemon added a comment - 2014-10-12 15:15

Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
changelog.html
pom.xml
http://jenkins-ci.org/commit/jenkins/9c82fc42eb08b89047c544aaa586291ad1485472
Log:
[JENKINS-23471 JENKINS-24050]

Integrated the fix in remoting to Jenkins 1.580.

(cherry picked from commit 8fc609fe0952b285d5b26a59fd5ff4c29704d33d)

SCM/JIRA link daemon added a comment - 2014-10-12 15:15 Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/9c82fc42eb08b89047c544aaa586291ad1485472 Log: [JENKINS-23471 JENKINS-24050] Integrated the fix in remoting to Jenkins 1.580. (cherry picked from commit 8fc609fe0952b285d5b26a59fd5ff4c29704d33d)

SCM/JIRA link daemon added a comment - 2016-08-04 18:22

Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
changelog.html
pom.xml
http://jenkins-ci.org/commit/jenkins/91c5551d4c7682d4adba28fe591fa7772eee62e0
Log:
[JENKINS-23471 JENKINS-24050]

Integrated the fix in remoting to Jenkins 1.580.

(cherry picked from commit 8fc609fe0952b285d5b26a59fd5ff4c29704d33d)

SCM/JIRA link daemon added a comment - 2016-08-04 18:22 Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/91c5551d4c7682d4adba28fe591fa7772eee62e0 Log: [JENKINS-23471 JENKINS-24050] Integrated the fix in remoting to Jenkins 1.580. (cherry picked from commit 8fc609fe0952b285d5b26a59fd5ff4c29704d33d)

Jenkins

Channel reader thread consumes all CPUs

Details

Description

Attachments

Attachments

Activity

People

Dates