Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22320

High CPU consumption because of SSH communication

XMLWordPrintable

      Our Jenkins instance runs 1.553 (since two weeks, was an three year old version before) on Linux with 10 slaves: 6 Linux connected through SSH and 4 windows with WebStart agent started as services.

      When no job is running, CPU is almost 0. Correct.

      When only two or three jobs are running, Jenkins CPU raises between 70% to 160%. atop declares 96% of CPU time spent in IRQ (even with "no" disk access), most CPU consumption is considered as system time. On average since boot, the master node consumes 100% of one CPU. Even if it has 4 CPUs, job execution time is between 2x and 3x compared to older version.

      I configured JMX and did a quick CPU profiling. Top consumer threads are unnamed and are all related to SSH communication

      "Thread-13" - Thread t@90
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41)
	at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52)
	at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79)
	at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108)
	at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232)
	at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:682)
	at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:480)
	at java.lang.Thread.run(Thread.java:744)

      So there is chance the "SSH agent plugin" is concerned.

      I am ready to do deeper analysis on my system if required, and of course to test patches.

        1. Selection_145.png
          25 kB
          Yves Martin
        2. Selection_146.png
          31 kB
          Yves Martin
        3. trilead-ssh2-64K-buffer.patch
          0.7 kB
          Thomas Herrlin

            kohsuke Kohsuke Kawaguchi
            ymartin1040 Yves Martin
            Votes:
            3 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: