Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38723

Stash failed with IOException during scalability testing

XMLWordPrintable

      During an scalability test, one of the executions failed on stash while executing this pipeline script in a high concurrency environment (15 nodes and 4 executors and a start queue of about 700 jobs)

      def branch(name) {
        return {
          node {
            sh "sleep 30 && head -c 52428800 /dev/urandom > ${name}.bin"
            stash includes: "${name}.bin", name: "${name}"
          }
        }
      }
      
      
      stage "Thinking"
      
      for (i = 0; i < 5; i++) {
        sleep time: 250, unit: 'MILLISECONDS'
        echo "Thinking $i"
      }
      
      stage "Working"
      
      def branches = [:]
      branches["b1"] = branch("b1")
      branches["b2"] = branch("b2")
      branches["b3"] = branch("b3")
      
      parallel branches
      
      stage "Resting"
      
      for (i = 0; i < 5; i++) {
        sleep time: 150, unit: 'MILLISECONDS'
        echo "Resting $i"
      }
      

      This was the result of the execution:

      Started
      [Pipeline] stage (Thinking)
      Entering stage Thinking
      Proceeding
      [Pipeline] sleep
      Sleeping for 0.25 sec
      [Pipeline] echo
      Thinking 0
      [Pipeline] sleep
      Sleeping for 0.25 sec
      [Pipeline] echo
      Thinking 1
      [Pipeline] sleep
      Sleeping for 0.25 sec
      [Pipeline] echo
      Thinking 2
      [Pipeline] sleep
      Sleeping for 0.25 sec
      [Pipeline] echo
      Thinking 3
      [Pipeline] sleep
      Sleeping for 0.25 sec
      [Pipeline] echo
      Thinking 4
      [Pipeline] stage (Working)
      Entering stage Working
      Proceeding
      [Pipeline] parallel
      [Pipeline] [b1] { (Branch: b1)
      [Pipeline] [b2] { (Branch: b2)
      [Pipeline] [b3] { (Branch: b3)
      [Pipeline] [b1] node
      [Pipeline] [b2] node
      [Pipeline] [b3] node
      [b1] Still waiting to schedule task
      [b1] Waiting for next available executor
      [b2] Still waiting to schedule task
      [b2] Waiting for next available executor
      [b3] Still waiting to schedule task
      [b3] Waiting for next available executor
      [b1] Running on node-10.0.0.58 in /home/ec2-user/workspace/Pipeline@4
      [Pipeline] [b1] {
      [Pipeline] [b1] sh
      [b1] [Pipeline@4] Running shell script
      [b1] + sleep 30
      [b2] Running on node-10.0.0.61 in /home/ec2-user/workspace/Pipeline@3
      [Pipeline] [b2] {
      [Pipeline] [b2] sh
      [b2] [Pipeline@3] Running shell script
      [b2] + sleep 30
      [b1] + head -c 52428800 /dev/urandom
      [Pipeline] [b1] stash
      [b2] + head -c 52428800 /dev/urandom
      [Pipeline] [b2] stash
      [b1] Stashed 1 file(s)
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [b3] Running on EC2_CJOC (i-0695a7fdd8011c959) in /home/ec2-user/workspace/Pipeline@2
      [Pipeline] [b3] {
      [Pipeline] [b3] sh
      [b3] [Pipeline@2] Running shell script
      [b3] + sleep 30
      [b2] Stashed 1 file(s)
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      [b3] + head -c 52428800 /dev/urandom
      [Pipeline] [b3] stash
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] }
      Failed in branch b3
      [Pipeline] // parallel
      [Pipeline] End of Pipeline
      java.io.IOException: remote file operation failed: /home/ec2-user/workspace/Pipeline@2 at hudson.remoting.Channel@5f30d359:EC2_CJOC (i-0695a7fdd8011c959): java.io.IOException: This archives contains unclosed entries.
          at hudson.FilePath.act(FilePath.java:1014)
          at hudson.FilePath.act(FilePath.java:996)
          at hudson.FilePath.archive(FilePath.java:459)
          at org.jenkinsci.plugins.workflow.flow.StashManager.stash(StashManager.java:90)
          at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:95)
          at org.jenkinsci.plugins.workflow.support.steps.stash.StashStep$Execution.run(StashStep.java:85)
          at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:52)
          at hudson.security.ACL.impersonate(ACL.java:213)
          at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:49)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
      Caused by: java.io.IOException: This archives contains unclosed entries.
          at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:225)
          at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:241)
          at hudson.util.io.TarArchiver.close(TarArchiver.java:111)
          at hudson.FilePath$1.invoke(FilePath.java:465)
          at hudson.FilePath$1.invoke(FilePath.java:459)
          at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
          at hudson.remoting.UserRequest.perform(UserRequest.java:153)
          at hudson.remoting.UserRequest.perform(UserRequest.java:50)
          at hudson.remoting.Request$2.run(Request.java:332)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at java.util.concurrent.FutureTask.run(FutureTask.java:262)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:745)
          at ......remote call to EC2_CJOC (i-0695a7fdd8011c959)(Native Method)
          at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
          at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
          at hudson.remoting.Channel.call(Channel.java:781)
          at hudson.FilePath.act(FilePath.java:1007)
          ... 13 more
      Finished: FAILURE
      

      This was tested on

      • O.S.: Linux on AWS
      • Jenkins 2.7.3
      • JDK version: JDK8

            Unassigned Unassigned
            alobato Alvaro Lobato
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: