Stash failed with IOException during scalability testing

This issue is archived. You can view it, but you can't modify it. Learn more

XMLWordPrintable

      During an scalability test, one of the executions failed on stash while executing this pipeline script in a high concurrency environment (15 nodes and 4 executors and a start queue of about 700 jobs)

      def branch(name) {
        return {
          node {
            sh "sleep 30 && head -c 52428800 /dev/urandom > ${name}.bin"
            stash includes: "${name}.bin", name: "${name}"
          }
        }
      }
      
      
      stage "Thinking"
      
      for (i = 0; i < 5; i++) {
        sleep time: 250, unit: 'MILLISECONDS'
        echo "Thinking $i"
      }
      
      stage "Working"
      
      def branches = [:]
      branches["b1"] = branch("b1")
      branches["b2"] = branch("b2")
      branches["b3"] = branch("b3")
      
      parallel branches
      
      stage "Resting"
      
      for (i = 0; i < 5; i++) {
        sleep time: 150, unit: 'MILLISECONDS'
        echo "Resting $i"
      }
      

      This was the result of the execution:

      Started
      [Pipeline] stage (Thinking)
      Entering stage Thinking
      Proceeding
      [Pipeline] sleep
      Sleeping for 0.25 sec
      [Pipeline] echo
      Thinking 0
      [Pipeline] sleep
      Sleeping for 0.25 sec
      [Pipeline] echo
      Thinking 1
      [Pipeline] sleep
      Sleeping for 0.25 sec
      [Pipeline] echo
      Thinking 2
      [Pipeline] sleep
      Sleeping for 0.25 sec
      [Pipeline] echo
      Thinking 3
      [Pipeline] sleep
      Sleeping for 0.25 sec
      [Pipeline] echo
      Thinking 4
      [Pipeline] stage (Working)
      Entering stage Working
      Proceeding
      [Pipeline] parallel
      [Pipeline] [b1] { (Branch: b1)
      [Pipeline] [b2] { (Branch: b2)
      [Pipeline] [b3] { (Branch: b3)
      [Pipeline] [b1] node
      [Pipeline] [b2] node
      [Pipeline] [b3] node
      [b1] Still waiting to schedule task
      [b1] Waiting for next available executor
      [b2] Still waiting to schedule task
      [b2] Waiting for next available executor
      [b3] Still waiting to schedule task
      [b3] Waiting for next available executor
      [b1] Running on node-10.0.0.58 in /home/ec2-user/workspace/Pipeline@4
      [Pipeline] [b1] {
      [Pipeline] [b1] sh
      [b1] [Pipeline@4] Running shell script
      [b1] + sleep 30
      [b2] Running on node-10.0.0.61 in /home/ec2-user/workspace/Pipeline@3
      [Pipeline] [b2] {
      [Pipeline] [b2] sh
      [b2] [Pipeline@3] Running shell script
      [b2] + sleep 30
      [b1] + head -c 52428800 /dev/urandom
      [Pipeline] [b1] stash
      [b2] + head -c 52428800 /dev/urandom
      [Pipeline] [b2] stash
      [b1] Stashed 1 file(s)
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [b3] Running on EC2_CJOC (i-0695a7fdd8011c959) in /home/ec2-user/workspace/Pipeline@2
      [Pipeline] [b3] {
      [Pipeline] [b3] sh
      [b3] [Pipeline@2] Running shell script
      [b3] + sleep 30
      [b2] Stashed 1 file(s)
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [b3] + head -c 52428800 /dev/urandom
      [Pipeline] [b3] stash
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      Failed in branch b3
      [Pipeline] // parallel
      [Pipeline] End of Pipeline
      java.io.IOException: remote file operation failed: /home/ec2-user/workspace/Pipeline@2 at hudson.remoting.Channel@5f30d359:EC2_CJOC (i-0695a7fdd8011c959): java.io.IOException: This archives contains unclosed entries.
          at hudson.FilePath.act(FilePath.java:1014)
          at hudson.FilePath.act(FilePath.java:996)
          at hudson.FilePath.archive(FilePath.java:459)
          at org.jenkinsci.plugins.workflow.flow.StashManager.stash(StashManager.java:90)
          at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:95)
          at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:85)
          at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:52)
          at hudson.security.ACL.impersonate(ACL.java:213)
          at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:49)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: This archives contains unclosed entries.
          at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:225)
          at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:241)
          at hudson.util.io.TarArchiver.close(TarArchiver.java:111)
          at hudson.FilePath$1.invoke(FilePath.java:465)
          at hudson.FilePath$1.invoke(FilePath.java:459)
          at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
          at hudson.remoting.UserRequest.perform(UserRequest.java:153)
          at hudson.remoting.UserRequest.perform(UserRequest.java:50)
          at hudson.remoting.Request$2.run(Request.java:332)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
          at ......remote call to EC2_CJOC (i-0695a7fdd8011c959)(Native Method)
          at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
          at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
          at hudson.remoting.Channel.call(Channel.java:781)
          at hudson.FilePath.act(FilePath.java:1007)
          ... 13 more
      Finished: FAILURE
      

      This was tested on

      • O.S.: Linux on AWS
      • Jenkins 2.7.3
      • JDK version: JDK8

            Assignee:
            Unassigned
            Reporter:
            Alvaro Lobato
            Archiver:
            Jenkins Service Account

              Created:
              Updated:
              Resolved:
              Archived: