Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-35982

EC2 Plugin: High CPU in hudson.plugins.ec2.win.winrm.WindowsProcess

      A Jenkins master running the EC2 Plugin will exhibit high CPU usage. High CPU Analysis will show threads with a stack trace similar to:

      "input copy: java -jar C:\Windows\Temp\slave.jar" #53906 daemon prio=5 os_prio=0 tid=0x00007fab61963800 nid=0x1860 runnable [0x00007fab1288f000]
      java.lang.Thread.State: RUNNABLE
      at java.lang.Throwable.fillInStackTrace(Native Method)
      at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
      - locked <0x00000006b9e757f8> (a java.io.IOException)
      at java.lang.Throwable.<init>(Throwable.java:265)
      at java.lang.Exception.<init>(Exception.java:66)
      at java.io.IOException.<init>(IOException.java:58)
      at java.io.PipedInputStream.read(PipedInputStream.java:310)
      - locked <0x00000005583223c0> (a java.io.PipedInputStream)
      at java.io.PipedInputStream.read(PipedInputStream.java:377)
      - locked <0x00000005583223c0> (a java.io.PipedInputStream)
      at java.io.InputStream.read(InputStream.java:101)
      at hudson.plugins.ec2.win.winrm.WindowsProcess$2.run(WindowsProcess.java:124)
      

      Which appears to be related to this code in the EC2 plugin.

          [JENKINS-35982] EC2 Plugin: High CPU in hudson.plugins.ec2.win.winrm.WindowsProcess

          James Sweet created issue -
          Jesse Glick made changes -
          Labels Original: plugin New: performance windows
          Jesse Glick made changes -
          Description Original: A Jenkins master running the EC2 Plugin will exhibit high CPU usage. High CPU Analysis will show threads with a stack trace similar to:

          "input copy: java -jar C:\Windows\Temp\slave.jar" #53906 daemon prio=5 os_prio=0 tid=0x00007fab61963800 nid=0x1860 runnable [0x00007fab1288f000]
          java.lang.Thread.State: RUNNABLE
          at java.lang.Throwable.fillInStackTrace(Native Method)
          at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
          - locked <0x00000006b9e757f8> (a java.io.IOException)
          at java.lang.Throwable.<init>(Throwable.java:265)
          at java.lang.Exception.<init>(Exception.java:66)
          at java.io.IOException.<init>(IOException.java:58)
          at java.io.PipedInputStream.read(PipedInputStream.java:310)
          - locked <0x00000005583223c0> (a java.io.PipedInputStream)
          at java.io.PipedInputStream.read(PipedInputStream.java:377)
          - locked <0x00000005583223c0> (a java.io.PipedInputStream)
          at java.io.InputStream.read(InputStream.java:101)
          at hudson.plugins.ec2.win.winrm.WindowsProcess$2.run(WindowsProcess.java:124)

          Which appears to be related to this code--https://github.com/jenkinsci/ec2-plugin/blob/0dc221934cbd087b2819b5660e9b778208c9f2dc/src/main/java/hudson/plugins/ec2/win/winrm/WindowsProcess.java#L125-L133 in the EC2 plugin.
          New: A Jenkins master running the EC2 Plugin will exhibit high CPU usage. High CPU Analysis will show threads with a stack trace similar to:

          {code:none}
          "input copy: java -jar C:\Windows\Temp\slave.jar" #53906 daemon prio=5 os_prio=0 tid=0x00007fab61963800 nid=0x1860 runnable [0x00007fab1288f000]
          java.lang.Thread.State: RUNNABLE
          at java.lang.Throwable.fillInStackTrace(Native Method)
          at java.lang.Throwable.fillInStackTrace(Throwable.java:783)
          - locked <0x00000006b9e757f8> (a java.io.IOException)
          at java.lang.Throwable.<init>(Throwable.java:265)
          at java.lang.Exception.<init>(Exception.java:66)
          at java.io.IOException.<init>(IOException.java:58)
          at java.io.PipedInputStream.read(PipedInputStream.java:310)
          - locked <0x00000005583223c0> (a java.io.PipedInputStream)
          at java.io.PipedInputStream.read(PipedInputStream.java:377)
          - locked <0x00000005583223c0> (a java.io.PipedInputStream)
          at java.io.InputStream.read(InputStream.java:101)
          at hudson.plugins.ec2.win.winrm.WindowsProcess$2.run(WindowsProcess.java:124)
          {code}

          Which appears to be related to [this code|https://github.com/jenkinsci/ec2-plugin/blob/0dc221934cbd087b2819b5660e9b778208c9f2dc/src/main/java/hudson/plugins/ec2/win/winrm/WindowsProcess.java#L125-L133] in the EC2 plugin.
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 172617 ] New: JNJira + In-Review [ 184694 ]

          Mika Karjalainen added a comment - - edited

          Running jvmtop with --profile option gave me this analysis:

          Profiling PID 1524: /usr/share/jenkins/jenkins.war --webroot
          
            99.22% (    76.60s) hudson.plugins.ec2.win.winrm.WindowsProcess$2.run()
             0.39% (     0.30s) ....sonyericsson.jenkins.plugins.bfa.model.FailureReader()
             0.16% (     0.12s) org.kohsuke.stapler.export.NotExportableException.<init>()
             0.04% (     0.03s) hudson.remoting.ObjectInputStreamEx.resolveClass()
             0.03% (     0.02s) ....amazonaws.http.protocol.SdkHttpRequestExecutor.doRec()
             0.02% (     0.02s) hudson.remoting.ChunkedOutputStream.sendFrame()
             0.02% (     0.02s) hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest()
             0.01% (     0.01s) hudson.util.IOUtils.copy()
             0.01% (     0.01s) hudson.remoting.Command.readFrom()
             0.01% (     0.01s) winstone.BoundedExecutorService$1.run()
             0.01% (     0.01s) hudson.remoting.FlightRecorderInputStream.read()
             0.01% (     0.01s) ....google.common.util.concurrent.ForwardingExecutorServ()
             0.00% (     0.00s) ....thoughtworks.xstream.io.xml.XmlFriendlyNameCoder.dec()
             0.00% (     0.00s) okio.Okio$2.read()
             0.00% (     0.00s) hudson.Util.tryOnceDeleteFile()
             0.00% (     0.00s) ....kohsuke.stapler.Function$InstanceFunction.getParamet()
             0.00% (     0.00s) com.jcraft.jzlib.InfCodes.inflate_fast()
             0.00% (     0.00s) hudson.PluginWrapper.getVersionOf()
             0.00% (     0.00s) com.thoughtworks.xstream.io.path.PathTracker.pushElement()
             0.00% (     0.00s) jenkins.util.VirtualFile$FileVF.isIllegalSymlink()
          

          During the analysis I did not have any ec2 windows agents running so this seems a bit weird. Running Jenkins 2.32.1 and EC2 plugin 1.36.

          Mika Karjalainen added a comment - - edited Running jvmtop with --profile option gave me this analysis: Profiling PID 1524: /usr/share/jenkins/jenkins.war --webroot 99.22% ( 76.60s) hudson.plugins.ec2.win.winrm.WindowsProcess$2.run() 0.39% ( 0.30s) ....sonyericsson.jenkins.plugins.bfa.model.FailureReader() 0.16% ( 0.12s) org.kohsuke.stapler.export.NotExportableException.<init>() 0.04% ( 0.03s) hudson.remoting.ObjectInputStreamEx.resolveClass() 0.03% ( 0.02s) ....amazonaws.http.protocol.SdkHttpRequestExecutor.doRec() 0.02% ( 0.02s) hudson.remoting.ChunkedOutputStream.sendFrame() 0.02% ( 0.02s) hudson.plugins.ec2.win.winrm.WinRMClient.sendRequest() 0.01% ( 0.01s) hudson.util.IOUtils.copy() 0.01% ( 0.01s) hudson.remoting.Command.readFrom() 0.01% ( 0.01s) winstone.BoundedExecutorService$1.run() 0.01% ( 0.01s) hudson.remoting.FlightRecorderInputStream.read() 0.01% ( 0.01s) ....google.common.util.concurrent.ForwardingExecutorServ() 0.00% ( 0.00s) ....thoughtworks.xstream.io.xml.XmlFriendlyNameCoder.dec() 0.00% ( 0.00s) okio.Okio$2.read() 0.00% ( 0.00s) hudson.Util.tryOnceDeleteFile() 0.00% ( 0.00s) ....kohsuke.stapler.Function$InstanceFunction.getParamet() 0.00% ( 0.00s) com.jcraft.jzlib.InfCodes.inflate_fast() 0.00% ( 0.00s) hudson.PluginWrapper.getVersionOf() 0.00% ( 0.00s) com.thoughtworks.xstream.io.path.PathTracker.pushElement() 0.00% ( 0.00s) jenkins.util.VirtualFile$FileVF.isIllegalSymlink() During the analysis I did not have any ec2 windows agents running so this seems a bit weird. Running Jenkins 2.32.1 and EC2 plugin 1.36.

          Johannes Ebke added a comment -

          I observed this issue after the disconnect of Windows agents.

          Looking at the code; it seems obvious that the Jenkins thread is stuck in a busy loop at https://github.com/jenkinsci/ec2-plugin/blob/0dc221934cbd087b2819b5660e9b778208c9f2dc/src/main/java/hudson/plugins/ec2/win/winrm/WindowsProcess.java#L126 since there is no escape for broken pipes. If spurious IOExceptions are really a problem there, as the comment claims, the retries should probably be limited to a sensible number.

          Johannes Ebke added a comment - I observed this issue after the disconnect of Windows agents. Looking at the code; it seems obvious that the Jenkins thread is stuck in a busy loop at https://github.com/jenkinsci/ec2-plugin/blob/0dc221934cbd087b2819b5660e9b778208c9f2dc/src/main/java/hudson/plugins/ec2/win/winrm/WindowsProcess.java#L126 since there is no escape for broken pipes. If spurious IOExceptions are really a problem there, as the comment claims, the retries should probably be limited to a sensible number.

          Johannes Ebke added a comment - - edited

          Looking at the github forks of this plugin; I found
          https://github.com/jenkinsci/ec2-plugin/compare/master...karliszigurs:fix-intermittent-pipe-failure-in-windows
          which would probably fix this issue.

          Edit: The PR https://github.com/jenkinsci/ec2-plugin/pull/263 containing this change was already merged. Thanks!

          Johannes Ebke added a comment - - edited Looking at the github forks of this plugin; I found https://github.com/jenkinsci/ec2-plugin/compare/master...karliszigurs:fix-intermittent-pipe-failure-in-windows which would probably fix this issue. Edit: The PR https://github.com/jenkinsci/ec2-plugin/pull/263 containing this change was already merged. Thanks!

          We are now running Jenkins 2.121.2 and ec2-plugin 1.39 and still see this issue. Two days after Jenkins restart CPU load has climbed up to 60% and top 5 threads sorted by cpu time are all 'input copy: java -jar C:\Windows\Temp\slave.jar'

          Mika Karjalainen added a comment - We are now running Jenkins 2.121.2 and ec2-plugin 1.39 and still see this issue. Two days after Jenkins restart CPU load has climbed up to 60% and top 5 threads sorted by cpu time are all 'input copy: java -jar C:\Windows\Temp\slave.jar'
          Shubham Dhoka made changes -
          Labels Original: performance windows New: ec2-plugin performance windows

          Raihaan Shouhell added a comment - xmiklis can you try with the latest snapshot? https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/ec2/1.45-rc1024.cafa05a12c08/

            raihaan Raihaan Shouhell
            jtsweet James Sweet
            Votes:
            7 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated: