Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40912

Add verbose option to stash/unstash to get list of files

    • Pipeline - October, Pipeline - April 2018

      It would be great if we could get more info on stash/unstash so I could see a list of files included. I was recently hit by an apparent unstashing hiccup where 1 file that should have been unstashed was not actually there and it made the subsequent errors rather confusing.

      Had there been a list of files in the log that were successfully unstashed then it would have been more clear what the problem was. Perhaps there could even be a validation step to go with this where if the number of files unstashed doesnt match number of stashed files then it logs an error.

          [JENKINS-40912] Add verbose option to stash/unstash to get list of files

          Andrew Bayer added a comment -

          Blech, still can't reproduce. If you can get me what versions of core, workflow-cps, and workflow-basic-steps you have installed, that'd be helpful. And if you by any chance can reproduce this live now and could get me the Jenkins thread dump (from http://jenkins/threadDump) when it's hanging, that'd be even more helpful. =) Thanks!

          Andrew Bayer added a comment - Blech, still can't reproduce. If you can get me what versions of core, workflow-cps , and workflow-basic-steps you have installed, that'd be helpful. And if you by any chance can reproduce this live now and could get me the Jenkins thread dump (from http://jenkins/threadDump ) when it's hanging, that'd be even more helpful. =) Thanks!

          I was using the following up-to-date versions:

          • Jenkins core 2.86
          • Jenkins plugins:
            • workflow-cps 2.41
            • (workflow-cps-global-lib 2.9)
            • workflow-basic-steps 2.6

          Unfortunately I don't have a thread dump anymore, but from my rememberings when using jstack I have to confirm jspiewak's sad and/or surprising statement:

          I was not able to see anything in either the agent or master thread dumps.

          Reinhold Füreder added a comment - I was using the following up-to-date versions: Jenkins core 2.86 Jenkins plugins: workflow-cps 2.41 (workflow-cps-global-lib 2.9) workflow-basic-steps 2.6 Unfortunately I don't have a thread dump anymore, but from my rememberings when using jstack I have to confirm jspiewak 's sad and/or surprising statement: I was not able to see anything in either the agent or master thread dumps.

          FWIW we are running Jenkins 2.73.1 (LTS) with

          • workflow-cps 2.4.1
          • workflow-cps-global-lib 2.9
          • workflow-basic-steps 2.6

          One of the builds performed stash like this:

           

          {{stage('Create package') {
          docker.image('python:2.7.13').inside {
          sh 'make sdist bdist' // produce source and binary distributions
          archiveArtifacts artifacts: '*/dist/'
          stash includes: 'dist/*', name: 'dist'
          }
          }}}

          And the other build:

          precise.inside {
            sh 'make dependencies-timestamper'
            stash includes: 'ext/timestamper', name: 'dependencies-timestamper'
          }
          

          So at least in ours cases, running the build inside of a Docker container is in play.

           

           

           

          Joshua Spiewak added a comment - FWIW we are running Jenkins 2.73.1 (LTS) with workflow-cps 2.4.1 workflow-cps-global-lib 2.9 workflow-basic-steps 2.6 One of the builds performed stash like this:   {{stage('Create package') { docker.image('python:2.7.13').inside { sh 'make sdist bdist' // produce source and binary distributions archiveArtifacts artifacts: '* /dist/ ' stash includes: 'dist/*', name: 'dist' } }}} And the other build: precise.inside { sh 'make dependencies-timestamper' stash includes: 'ext/timestamper' , name: 'dependencies-timestamper' } So at least in ours cases, running the build inside of a Docker container is in play.      

          Andrew Bayer added a comment -

          Got a new attempt up at https://github.com/jenkinsci/workflow-api-plugin/pull/62 and https://github.com/jenkinsci/workflow-basic-steps-plugin/pull/54 which should avoid the problems we hit with the first one. It adds a new StashManager.stashFileList(Run,String) method and uses that in StashStep and UnstashStep, rather than changing the signatures of StashManager.stash and StashManager.unstash.

          Andrew Bayer added a comment - Got a new attempt up at https://github.com/jenkinsci/workflow-api-plugin/pull/62 and https://github.com/jenkinsci/workflow-basic-steps-plugin/pull/54 which should avoid the problems we hit with the first one. It adds a new StashManager.stashFileList(Run,String) method and uses that in StashStep and UnstashStep , rather than changing the signatures of StashManager.stash and StashManager.unstash .

          Andrew Bayer added a comment -

          Ok, this sucker is back in the "I dunno what to do" state. There's continued uncertainty as to what exactly sparklepony and everyone else would actually like - my PRs switch stash and unstash to return a list of the files affected - does that meet your requirements? Or would you prefer logging the files with a verbose option?

          Andrew Bayer added a comment - Ok, this sucker is back in the "I dunno what to do" state. There's continued uncertainty as to what exactly sparklepony and everyone else would actually like - my PRs switch stash and unstash to return a list of the files affected - does that meet your requirements? Or would you prefer logging the files with a verbose option?

          Andrew Bayer added a comment -

          jglick - so with the changes over on https://github.com/jenkinsci/workflow-api-plugin/pull/67, it feels like what would make the most sense is maybe for StashAwareArtifactManager to have a listStashFiles method - with StandardArtifactManager doing something like https://github.com/jenkinsci/workflow-api-plugin/pull/62/files#diff-4ad53e530bca0f047ecfd41828c4e0eeR182, but other StashAwareArtifactManager implementations doing something else. Does that make any sense?

          Andrew Bayer added a comment - jglick - so with the changes over on https://github.com/jenkinsci/workflow-api-plugin/pull/67 , it feels like what would make the most sense is maybe for StashAwareArtifactManager to have a listStashFiles method - with StandardArtifactManager doing something like https://github.com/jenkinsci/workflow-api-plugin/pull/62/files#diff-4ad53e530bca0f047ecfd41828c4e0eeR182 , but other StashAwareArtifactManager implementations doing something else. Does that make any sense?

          Jesse Glick added a comment -

          abayer I am still guessing that the basic user requirement would be satisfied by a verbose: true flag to the stash step. Once StashAwareArtifactManager lands, there could be multiple implementations, but at any rate the obvious way would be for FileVisitor to have a convenience method

          public FileVisitor verbose(TaskListener listener) {…}
          

          with the standard implementation being

          int count = workspace.archive(ArchiverFactory.TARGZ.verbose(listener), os, new DirScanner.Glob(/* as before… */));
          

          If you wish to avoid a new core dep (but see JEP-301!), the verbose method could temporarily be hosted as static in StashManager.

          My comment in PR 67 was just to the effect that if you agree with that design and want this RFE now, the easiest way to avoid clashes in introduced APIs would be to extract the new StashManager method overloads from that PR into a simple PR with no core dep change or StashAwareArtifactManager, including the added TaskListener parameters and the boolean verbose flag. Similarly, the StashStep and UnstashStep patches from workflow-basic-steps PR 60 would be extracted into a small PR that also adds the verbose step parameter. All easy for me to do (~30m) when I have this stuff in flight anyway, but I am not going to bother unless you are prepared to review and merge it.

          All that said, this whole RFE has a trivial workaround (I would hesitate to even call it that; more a diagnostic idiom): in your Jenkinsfile after an apparently misconfigured stash, when you unstash just

          sh 'ls -R'
          

          to see what you got and whether it matches your expectations. Optionally do the same in the original workspace prior to the stash and compare. Short of reimplementing the Ant patternset matching code from scratch to offer fine-grained explanations of why each individual candidate relative path did or did not match each component of each pattern, no feature we offer is really going to change the diagnostic experience much: you just experiment with whether a given patternset works or not, accompanied if necessary with a bit of stackoverflow.com searching, documentation review, and, well, thinking.

          Jesse Glick added a comment - abayer I am still guessing that the basic user requirement would be satisfied by a verbose: true flag to the stash step. Once StashAwareArtifactManager lands, there could be multiple implementations, but at any rate the obvious way would be for FileVisitor to have a convenience method public FileVisitor verbose(TaskListener listener) {…} with the standard implementation being int count = workspace.archive(ArchiverFactory.TARGZ.verbose(listener), os, new DirScanner.Glob( /* as before… */ )); If you wish to avoid a new core dep (but see JEP-301!), the verbose method could temporarily be hosted as static in StashManager . My comment in PR 67 was just to the effect that if you agree with that design and want this RFE now , the easiest way to avoid clashes in introduced APIs would be to extract the new StashManager method overloads from that PR into a simple PR with no core dep change or StashAwareArtifactManager , including the added TaskListener parameters and the boolean verbose flag. Similarly, the StashStep and UnstashStep patches from workflow-basic-steps PR 60 would be extracted into a small PR that also adds the verbose step parameter. All easy for me to do (~30m) when I have this stuff in flight anyway, but I am not going to bother unless you are prepared to review and merge it. All that said, this whole RFE has a trivial workaround (I would hesitate to even call it that; more a diagnostic idiom): in your Jenkinsfile after an apparently misconfigured stash , when you unstash just sh 'ls -R' to see what you got and whether it matches your expectations. Optionally do the same in the original workspace prior to the stash and compare. Short of reimplementing the Ant patternset matching code from scratch to offer fine-grained explanations of why each individual candidate relative path did or did not match each component of each pattern, no feature we offer is really going to change the diagnostic experience much: you just experiment with whether a given patternset works or not, accompanied if necessary with a bit of stackoverflow.com searching, documentation review, and, well, thinking.

          Jesse Glick added a comment -

          Actually verbose would need to be on ArchiverFactory I suppose.

          An alternative impl API would be for FilePath.archive to get an overload returning a List<String> paths, which could be used by callers to print diagnostics (though not in real time, and without the possibility of distinguishing files vs. directories vs. symlinks in output).

          Jesse Glick added a comment - Actually verbose would need to be on ArchiverFactory I suppose. An alternative impl API would be for FilePath.archive to get an overload returning a List<String> paths, which could be used by callers to print diagnostics (though not in real time, and without the possibility of distinguishing files vs. directories vs. symlinks in output).

          My preference would be for a verbose flag on both stash and unstash. Even just having the unstash print out just the number of files (to match it up with the output stash currently provides) would be an improvement

          Justin Georgeson added a comment - My preference would be for a verbose flag on both stash and unstash. Even just having the unstash print out just the number of files (to match it up with the output stash currently provides) would be an improvement

          Jesse Glick added a comment -

          Even printing a count would require a core API change, I think. Not positive.

          Jesse Glick added a comment - Even printing a count would require a core API change, I think. Not positive.

            jtaboada Jose Blas Camacho Taboada
            sparklepony Sparkle Pony
            Votes:
            14 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated: