Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-36914

stash step is excessively slow on ARM

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Not A Defect
    • Component/s: workflow-api-plugin
    • Labels:
    • Environment:
    • Similar Issues:

      Description

      When using the stash build step on a Tegra X1 build slave, it is orders of magnitude slower than it should be: a single file of 20MB takes 29 seconds to archive, and larger files take proportionally longer. The device is connected by 1Gb/s ethernet and the file is stored on an SSD, so it should take well under 1s. While this is happening, the java process on the slave is at 100% CPU usage.

      Sample pipeline script:

      stage 'prepare'
      node('tegra-cuda') {
          deleteDir()
          sh 'dd if=/dev/zero of=dummy bs=1M count=20'
          sh 'date'
          stash name: 'source', includes: 'dummy'
          sh 'date'
      }
      

      Output:

      [Pipeline] stage (prepare)
      Entering stage prepare
      Proceeding
      [Pipeline] node
      Running on e5f011df5aa1-e069b02f in /var/lib/jenkins/workspace/ARM stash test
      [Pipeline] {
      [Pipeline] deleteDir
      [Pipeline] sh
      [ARM stash test] Running shell script
      + dd if=/dev/zero of=dummy bs=1M count=20
      20+0 records in
      20+0 records out
      20971520 bytes (21 MB) copied, 0.0732756 s, 286 MB/s
      [Pipeline] sh
      [ARM stash test] Running shell script
      + date
      Mon Jul 25 11:02:05 UTC 2016
      [Pipeline] stash
      Stashed 1 file(s)
      [Pipeline] sh
      [ARM stash test] Running shell script
      + date
      Mon Jul 25 11:02:34 UTC 2016
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      Finished: SUCCESS
      

      Thread dump for the relevant slave:

      Channel reader thread: channel
      
      "Channel reader thread: channel" Id=11 Group=main RUNNABLE (in native)
      	at java.net.SocketInputStream.socketRead0(Native Method)
      	at java.net.SocketInputStream.read(SocketInputStream.java:152)
      	at java.net.SocketInputStream.read(SocketInputStream.java:122)
      	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
      	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
      	-  locked java.io.BufferedInputStream@b24124
      	at hudson.remoting.FlightRecorderInputStream.read(FlightRecorderInputStream.java:82)
      	at hudson.remoting.ChunkedInputStream.readHeader(ChunkedInputStream.java:72)
      	at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:103)
      	at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39)
      	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      
      main
      
      "main" Id=1 Group=main WAITING on hudson.remoting.Engine@13108c4
      	at java.lang.Object.wait(Native Method)
      	-  waiting on hudson.remoting.Engine@13108c4
      	at java.lang.Thread.join(Thread.java:1281)
      	at java.lang.Thread.join(Thread.java:1355)
      	at hudson.remoting.jnlp.Main.main(Main.java:137)
      	at hudson.remoting.jnlp.Main._main(Main.java:130)
      	at hudson.remoting.jnlp.Main.main(Main.java:96)
      	at hudson.plugins.swarm.SwarmClient.connect(SwarmClient.java:239)
      	at hudson.plugins.swarm.Client.run(Client.java:107)
      	at hudson.plugins.swarm.Client.main(Client.java:68)
      
      Ping thread for channel hudson.remoting.Channel@195804b:channel
      
      "Ping thread for channel hudson.remoting.Channel@195804b:channel" Id=16 Group=main TIMED_WAITING
      	at java.lang.Thread.sleep(Native Method)
      	at hudson.remoting.PingThread.run(PingThread.java:90)
      
      pool-1-thread-274 for channel
      
      "pool-1-thread-274 for channel" Id=1122 Group=main RUNNABLE
      	at com.jcraft.jzlib.Deflate.fill_window(Deflate.java:966)
      	at com.jcraft.jzlib.Deflate.deflate_slow(Deflate.java:1125)
      	at com.jcraft.jzlib.Deflate.deflate(Deflate.java:1587)
      	at com.jcraft.jzlib.Deflater.deflate(Deflater.java:140)
      	at com.jcraft.jzlib.DeflaterOutputStream.deflate(DeflaterOutputStream.java:129)
      	at com.jcraft.jzlib.DeflaterOutputStream.write(DeflaterOutputStream.java:102)
      	at org.apache.commons.compress.utils.CountingOutputStream.write(CountingOutputStream.java:48)
      	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.writeRecord(TarArchiveOutputStream.java:571)
      	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.write(TarArchiveOutputStream.java:435)
      	at hudson.util.io.TarArchiver.visit(TarArchiver.java:100)
      	at hudson.util.DirScanner.scanSingle(DirScanner.java:49)
      	at hudson.util.DirScanner$Glob.scan(DirScanner.java:131)
      	at hudson.FilePath$1.invoke(FilePath.java:463)
      	at hudson.FilePath$1.invoke(FilePath.java:459)
      	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:120)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      	at hudson.remoting.Request$2.run(Request.java:326)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at hudson.remoting.Engine$1$1.run(Engine.java:62)
      	at java.lang.Thread.run(Thread.java:745)
      
      	Number of locked synchronizers = 1
      	- java.util.concurrent.ThreadPoolExecutor$Worker@17d329e
      
      pool-1-thread-275 for channel
      
      "pool-1-thread-275 for channel" Id=1126 Group=main RUNNABLE
      	at sun.management.ThreadImpl.dumpThreads0(Native Method)
      	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
      	at hudson.Functions.getThreadInfos(Functions.java:1196)
      	at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:98)
      	at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:95)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:120)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
      	at hudson.remoting.Request$2.run(Request.java:326)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at hudson.remoting.Engine$1$1.run(Engine.java:62)
      	at java.lang.Thread.run(Thread.java:745)
      
      	Number of locked synchronizers = 1
      	- java.util.concurrent.ThreadPoolExecutor$Worker@a0b835
      
      RemoteInvocationHandler [#1]
      
      "RemoteInvocationHandler [#1]" Id=10 Group=main TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@d2df5d
      	at java.lang.Object.wait(Native Method)
      	-  waiting on java.lang.ref.ReferenceQueue$Lock@d2df5d
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
      	at hudson.remoting.RemoteInvocationHandler$Unexporter.run(RemoteInvocationHandler.java:415)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
      	at java.lang.Thread.run(Thread.java:745)
      
      Thread-1
      
      "Thread-1" Id=9 Group=main TIMED_WAITING on hudson.remoting.Channel@195804b
      	at java.lang.Object.wait(Native Method)
      	-  waiting on hudson.remoting.Channel@195804b
      	at hudson.remoting.Channel.join(Channel.java:948)
      	at hudson.remoting.Engine.run(Engine.java:267)
      
      Finalizer
      
      "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@466bb5
      	at java.lang.Object.wait(Native Method)
      	-  waiting on java.lang.ref.ReferenceQueue$Lock@466bb5
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
      	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
      	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
      
      process reaper
      
      "process reaper" Id=1123 Group=system TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@6e9009
      	at sun.misc.Unsafe.park(Native Method)
      	-  waiting on java.util.concurrent.SynchronousQueue$TransferStack@6e9009
      	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
      	at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
      	at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
      	at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
      	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      
      Reference Handler
      
      "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@1496b81
      	at java.lang.Object.wait(Native Method)
      	-  waiting on java.lang.ref.Reference$Lock@1496b81
      	at java.lang.Object.wait(Object.java:503)
      	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
      
      Signal Dispatcher
      
      "Signal Dispatcher" Id=4 Group=system RUNNABLE
      

      I've taken a guess at the appropriate component, but I'm not sure which component corresponds to "workflow-basic-steps" (https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/)

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              bmerry Bruce Merry
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: