Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62158

Bad performance on EC2 instance for first build

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I have a pipeline project which should run at a EC2 instance node.

      I have configured an EC2 connection and starting EC2 t3.medium Windows 10 instances automatically. This all works fine.

      But, the first build at an EC2 instance always performs very bad (slow!!). The next build at the same instance (without rebott etc) is much more faster.

       

      @Library('BMS-Libraries')
      import static bms.mail.Email.*
      import static bms.nexus.Nexus.*
      import static bms.utils.Utils.*
      
      node('AWS_VS2017') {
      		stage('Cleanup Build Machine'){
      			//deleting current workspace directory
      			deleteDir()
      		}
      		
      		stage('Preparing Build machine...'){
      	retrieveAndExtractBuildTools(this)
      		}
      
      //Do some more .......
      }
      

      I attached a screenshot of the runtime of the different pipeline steps.

       

      I connected via RDP to the instance during first build nad task-manager didn't display a high CPU or Memory consumption

        Attachments

          Activity

          Hide
          raihaan Raihaan Shouhell added a comment -

          Are you using WinRM by any chance?

          Show
          raihaan Raihaan Shouhell added a comment - Are you using WinRM by any chance?
          Hide
          dhoerner Daniel Hoerner added a comment -

          The EC2 Plugin always use WinRM for Windows Slaves at AWS.

          Show
          dhoerner Daniel Hoerner added a comment - The EC2 Plugin always use WinRM for Windows Slaves at AWS.
          Hide
          raihaan Raihaan Shouhell added a comment -

          By default yes, there is a way to use ssh.

          Have you tried the latest of the plugin by any chance?

          Show
          raihaan Raihaan Shouhell added a comment - By default yes, there is a way to use ssh. Have you tried the latest of the plugin by any chance?
          Hide
          dhoerner Daniel Hoerner added a comment -

          I just tried the latest version (1.54 of the plugin) with Jenkins 2.249.3 and nothing has changed.

          I also have some issues with a broken connection during the build with following output:

          Node Log:

          Agent successfully connected and online
          ERROR: Connection terminated
          java.io.EOFException
          	at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2763)
          	at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3258)
          	at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:873)
          	at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:350)
          	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
          	at hudson.remoting.Command.readFrom(Command.java:142)
          	at hudson.remoting.Command.readFrom(Command.java:128)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
          Caused: java.io.IOException: Unexpected termination of the channel
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)

          Build Log:

          EC2 (AWS Cloud) - AWS Windows 10 with VS 2017 (i-xxxxxxxxxxxxxxxxxxxxxxxx) was marked offline: Connection was broken: java.io.EOFExceptionEC2 (AWS Cloud) - AWS Windows 10 with VS 2017 (i-xxxxxxxxxxxxxxxxxxxxxxxx) was marked offline: Connection was broken: java.io.EOFException at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2763) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3258) at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:873) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:350) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
          [Pipeline] }[Pipeline] // node[Pipeline] End of Pipelinejava.io.IOException: Unable to create live FilePath for EC2 (AWS Cloud) - AWS Windows 10 with VS 2017 (i-xxxxxxxxxxxxxxxxxxxxxxxxxx) at org.jenkinsci.plugins.workflow.support.steps.FilePathDynamicContext.get(FilePathDynamicContext.java:64) at org.jenkinsci.plugins.workflow.support.steps.FilePathDynamicContext.get(FilePathDynamicContext.java:47) at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:94) at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:138) at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:74) at hudson.plugins.emailext.EmailExtStep$EmailExtStepExecution.run(EmailExtStep.java:231) at hudson.plugins.emailext.EmailExtStep$EmailExtStepExecution.run(EmailExtStep.java:174) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:367) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)Finished: FAILURE
          
          Show
          dhoerner Daniel Hoerner added a comment - I just tried the latest version (1.54 of the plugin) with Jenkins 2.249.3 and nothing has changed. I also have some issues with a broken connection during the build with following output: Node Log: Agent successfully connected and online ERROR: Connection terminated java.io.EOFException at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2763) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3258) at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:873) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:350) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) Build Log: EC2 (AWS Cloud) - AWS Windows 10 with VS 2017 (i-xxxxxxxxxxxxxxxxxxxxxxxx) was marked offline: Connection was broken: java.io.EOFExceptionEC2 (AWS Cloud) - AWS Windows 10 with VS 2017 (i-xxxxxxxxxxxxxxxxxxxxxxxx) was marked offline: Connection was broken: java.io.EOFException at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2763) at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3258) at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:873) at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:350) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:142) at hudson.remoting.Command.readFrom(Command.java:128) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) [Pipeline] }[Pipeline] // node[Pipeline] End of Pipelinejava.io.IOException: Unable to create live FilePath for EC2 (AWS Cloud) - AWS Windows 10 with VS 2017 (i-xxxxxxxxxxxxxxxxxxxxxxxxxx) at org.jenkinsci.plugins.workflow.support.steps.FilePathDynamicContext.get(FilePathDynamicContext.java:64) at org.jenkinsci.plugins.workflow.support.steps.FilePathDynamicContext.get(FilePathDynamicContext.java:47) at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:94) at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:138) at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:74) at hudson.plugins.emailext.EmailExtStep$EmailExtStepExecution.run(EmailExtStep.java:231) at hudson.plugins.emailext.EmailExtStep$EmailExtStepExecution.run(EmailExtStep.java:174) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47) at hudson.security.ACL.impersonate(ACL.java:367) at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang. Thread .run( Thread .java:834)Finished: FAILURE
          Hide
          stevenfoster Steven Foster added a comment -

          Raihaan Shouhell

          > By default yes, there is a way to use ssh.

          can you elaborate on this? looking at the code I only see a WinRM connection method, and an SSH connection method that relies on unix tools

          Show
          stevenfoster Steven Foster added a comment - Raihaan Shouhell > By default yes, there is a way to use ssh. can you elaborate on this? looking at the code I only see a WinRM connection method, and an SSH connection method that relies on unix tools

            People

            Assignee:
            mramonleon Ramon Leon
            Reporter:
            dhoerner Daniel Hoerner
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Dates

              Created:
              Updated: