Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38640

Unstash painfully slow for large artifacts (due to compression?)

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Minor
    • Resolution: Fixed
    • workflow-api-plugin

    Description

      When sending a stash across the network, the data transfer can be significantly slower than the underlying network. This seems to be due to some slow on-the-fly compression happening during the transfer. Using the following script, the Unstash phase takes 1-2 seconds for 10MB of /dev/zero, and 49 seconds for 10MB of /dev/urandom. The Stash phase consistently takes 1-2 seconds. This makes me think there's a compression step, since /dev/zero should compress much faster than /dev/urandom.

      node {
          deleteDir()
          stage "Stash"
          sh "dd if=/dev/zero of=data bs=1M count=10"
          stash name: 'build_outputs', includes: 'data'
          sh "date"
          node('lab') {
              deleteDir()
              stage "Unstash"
              unstash 'build_outputs'
              sh "ls -al"
          }
      }
      

      This looks similar to JENKINS-36914, in particular this comment:
      https://issues.jenkins-ci.org/browse/JENKINS-36914?focusedCommentId=268472&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-268472

      I'm not calling this a dupe of that because the original text refers to stash only, and this bug seems isolated to unstash. Also this is on x86, not ARM.

      Attachments

        Issue Links

          Activity

            oleg_nenashev Oleg Nenashev added a comment -

            External Workspace Manager (https://github.com/jenkinsci/external-workspace-manager-plugin/blob/master/README.md) could be a workaround. With this plugin you can share the entire workspace on filer instead of stashing particular files and getting overhead on the Jenkins side.

            oleg_nenashev Oleg Nenashev added a comment - External Workspace Manager ( https://github.com/jenkinsci/external-workspace-manager-plugin/blob/master/README.md ) could be a workaround. With this plugin you can share the entire workspace on filer instead of stashing particular files and getting overhead on the Jenkins side.

            Code changed in jenkins
            User: Andres Rodriguez
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/flow/StashManager.java
            http://jenkins-ci.org/commit/workflow-api-plugin/5b4c7552511fe503cf4a5278b3de3c117cb25e0e
            Log:
            JENKINS-38640 Use a GREEDY RemoteInputStream

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andres Rodriguez Path: src/main/java/org/jenkinsci/plugins/workflow/flow/StashManager.java http://jenkins-ci.org/commit/workflow-api-plugin/5b4c7552511fe503cf4a5278b3de3c117cb25e0e Log: JENKINS-38640 Use a GREEDY RemoteInputStream

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/flow/StashManager.java
            http://jenkins-ci.org/commit/workflow-api-plugin/ca8f6dd3537dcb866e0aa02957ff80e702593884
            Log:
            Merge pull request #19 from andresrc/JENKINS-38640

            JENKINS-38640 Use a GREEDY RemoteInputStream

            Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/42a296e511f8...ca8f6dd3537d

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/flow/StashManager.java http://jenkins-ci.org/commit/workflow-api-plugin/ca8f6dd3537dcb866e0aa02957ff80e702593884 Log: Merge pull request #19 from andresrc/ JENKINS-38640 JENKINS-38640 Use a GREEDY RemoteInputStream Compare: https://github.com/jenkinsci/workflow-api-plugin/compare/42a296e511f8...ca8f6dd3537d

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/resources/org/jenkinsci/plugins/workflow/support/steps/stash/StashStep/help.html
            http://jenkins-ci.org/commit/workflow-basic-steps-plugin/413df48bdcb832261e8fb110150eeb8069e77c33
            Log:
            JENKINS-38640 JENKINS-36914 Warn users to avoid stash/unstash of large files

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/resources/org/jenkinsci/plugins/workflow/support/steps/stash/StashStep/help.html http://jenkins-ci.org/commit/workflow-basic-steps-plugin/413df48bdcb832261e8fb110150eeb8069e77c33 Log: JENKINS-38640 JENKINS-36914 Warn users to avoid stash/unstash of large files
            jkoleszar John Koleszar added a comment -

            Thanks for the workaround. It won't help in my application, as the master can only initiate connections to the slaves via ssh, and no connections can be initiated in the other direction. A shared filesystem via NFS or similar isn't really practical, unfortunately. Stash does what I want, and I'm willing to take some performance hit, but it should be able to do much better than 200KB/s on this hardware. Looking forward to trying out the greedy change to see how that does.

            jkoleszar John Koleszar added a comment - Thanks for the workaround. It won't help in my application, as the master can only initiate connections to the slaves via ssh, and no connections can be initiated in the other direction. A shared filesystem via NFS or similar isn't really practical, unfortunately. Stash does what I want, and I'm willing to take some performance hit, but it should be able to do much better than 200KB/s on this hardware. Looking forward to trying out the greedy change to see how that does.

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/resources/org/jenkinsci/plugins/workflow/support/steps/stash/StashStep/help.html
            http://jenkins-ci.org/commit/workflow-basic-steps-plugin/f541cd2cda5f316042d38af556f4160f7e470ccf
            Log:
            Merge pull request #23 from jenkinsci/jglick-stash-docs

            JENKINS-38640 JENKINS-36914 Warn users to avoid stash/unstash of large files

            Compare: https://github.com/jenkinsci/workflow-basic-steps-plugin/compare/95e202bec553...f541cd2cda5f

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/resources/org/jenkinsci/plugins/workflow/support/steps/stash/StashStep/help.html http://jenkins-ci.org/commit/workflow-basic-steps-plugin/f541cd2cda5f316042d38af556f4160f7e470ccf Log: Merge pull request #23 from jenkinsci/jglick-stash-docs JENKINS-38640 JENKINS-36914 Warn users to avoid stash/unstash of large files Compare: https://github.com/jenkinsci/workflow-basic-steps-plugin/compare/95e202bec553...f541cd2cda5f
            seanf Sean Flanigan added a comment -

            The External Workspace Manager plugin seems to require sharing drives between master and slaves, which rules it out in a lot of situations.

            If, instead of stashing, the official recommendation is to use an external repository manager such as Nexus or Artifactory, it would be good to have some pipeline examples. For instance, how to stash the majority of workspace files, whilst letting the repository manager handle the larger files, and still ensuring that the right files are used together throughout the pipeline.

            seanf Sean Flanigan added a comment - The External Workspace Manager plugin seems to require sharing drives between master and slaves, which rules it out in a lot of situations. If, instead of stashing, the official recommendation is to use an external repository manager such as Nexus or Artifactory, it would be good to have some pipeline examples. For instance, how to stash the majority of workspace files, whilst letting the repository manager handle the larger files, and still ensuring that the right files are used together throughout the pipeline.
            jkoleszar John Koleszar added a comment -

            The admonition not to use stash is too strong, in my opinion. I can see plenty of situations where having a repository manager or setting up shared storage are completely unnecessary complexity. The latest release of this plugin performs as I'd expect.

            jkoleszar John Koleszar added a comment - The admonition not to use stash is too strong, in my opinion. I can see plenty of situations where having a repository manager or setting up shared storage are completely unnecessary complexity. The latest release of this plugin performs as I'd expect.
            oleg_nenashev Oleg Nenashev added a comment -

            There is NO official recommendation to avoid stash/unstash. I've just mentioned it as a possible workaround

            oleg_nenashev Oleg Nenashev added a comment - There is NO official recommendation to avoid stash/unstash. I've just mentioned it as a possible workaround
            jkoleszar John Koleszar added a comment -

            Thanks, Oleg. This is the documentation I was referring to about stash only being suitable for small files:
            https://github.com/jenkinsci/workflow-basic-steps-plugin/commit/413df48bdcb832261e8fb110150eeb8069e77c33

            Maybe there's an upper limit where it falls over, but I haven't hit it yet. In my opinion, if there were one, it'd be a bug worth fixing, rather than something to document away as WAI.

            jkoleszar John Koleszar added a comment - Thanks, Oleg. This is the documentation I was referring to about stash only being suitable for small files: https://github.com/jenkinsci/workflow-basic-steps-plugin/commit/413df48bdcb832261e8fb110150eeb8069e77c33 Maybe there's an upper limit where it falls over, but I haven't hit it yet. In my opinion, if there were one, it'd be a bug worth fixing, rather than something to document away as WAI.
            oleg_nenashev Oleg Nenashev added a comment -

            The fix has been backported to 2.19.3

            oleg_nenashev Oleg Nenashev added a comment - The fix has been backported to 2.19.3

            People

              andresrc Andres Rodriguez
              jkoleszar John Koleszar
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: