Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38517

“Local module directory” not supported for external libraries

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • subversion-plugin
    • LTS 2.7.4

      scm-api 1.3
      subversion 2.6
      worfklow-cps-global-lib 2.3

      Zulu OpenJDK 8.17.0.3-win64 (1.8.0_102-b14)
      Win 7 Pro master--error does not involve agents

      Multiple shared libs throw "ERROR: Library <lib name> expected to contain at least one of src or vars directories" when multiple external SVN Legacy SCM libs are loaded by the Jenkinsfile and the Local module directory indicates preservation/overriding of the parent directory. (Ie, when the dot default is not used and a working copy root directory for each lib is created under the @libs directory.

      Configuration of global and folder specific libraries as shown in the attachments. The Jenkinsfile script begins

      @Library('pipeline_global_helpers') _
      
      @Library('pipeline_branch_build')
      import com.foo.bar.Application
      

      Running the pipeline makes it as far as checking out or updating the local working copies in the @libs/pipeline_global_helpers and @libs/pipeline_branch_build in the pipeline workspace on the master, but execution aborts thereafter with the stack trace

      ERROR: Library pipeline_branch_build expected to contain at least one of src or vars directories
      org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
      WorkflowScript: Loading libraries failed
      
      1 error
      
      	at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
      	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1073)
      	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:591)
      	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569)
      	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546)
      	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
      	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
      	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
      	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:67)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:410)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:373)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:213)
      	at hudson.model.ResourceController.execute(ResourceController.java:98)
      	at hudson.model.Executor.run(Executor.java:410)
      Finished: FAILURE
      

      The libraries both have src and vars subdirectories and cause no error if used individually if checked out with Local module directory set to the default dot (no working copy root directory). I have also avoided the error once or twice when overriding/preserving the working copy root directory when loading a single external library, but not reliably. Ultimately the above error recurs.

      Checking out straight into the @libs directory is good for the single-library use case but if I attempt to use multiple libraries, the first checkout's working copy is blown away by the second library's checkout.

      For now, my workaround is to use a single external library.

          [JENKINS-38517] “Local module directory” not supported for external libraries

          Brian Ray added a comment - - edited

          The clouds have parted. I refactored the SVN directories. The most recent screenshot is a successful pipeline run importing two global libs and exercising a variable from one and a class from the other.

          The thing that was not clear to me until now, and the impetus for trying to set the Local module directory to begin with, is that the external lib functionality seems to load the libraries between "recycling" the working copy in @libs. At the outset of this troubleshooting I thought it would need to preserve each checked out lib in a separate directory.

          Back to productive effort now ...

          Brian Ray added a comment - - edited The clouds have parted. I refactored the SVN directories. The most recent screenshot is a successful pipeline run importing two global libs and exercising a variable from one and a class from the other. The thing that was not clear to me until now, and the impetus for trying to set the Local module directory to begin with, is that the external lib functionality seems to load the libraries between "recycling" the working copy in @libs . At the outset of this troubleshooting I thought it would need to preserve each checked out lib in a separate directory. Back to productive effort now ...

          Jesse Glick added a comment -

          Not sure I see how the difference would be visible to a user. Is there something that could be clarified in documentation?

          Jesse Glick added a comment - Not sure I see how the difference would be visible to a user. Is there something that could be clarified in documentation?

          Brian Ray added a comment -

          If it's an easy change, I think just hiding the field entirely would be best in this context. There's no reason to use the field for external libraries--except maybe one.

          If the field were suppressed from the UI, nobody'd make the mistake of trying to use it, and no doc would be needed.

          The only reason it might be useful is if the external loader library were optimizing checkouts between builds when multiple libraries were involved. The Check-out Strategy of using svn update as much as possible is short circuited when the local directory is recycled for the checkout of another library. If each library were checked out in its own directory, svn update would work.

          Brian Ray added a comment - If it's an easy change, I think just hiding the field entirely would be best in this context. There's no reason to use the field for external libraries--except maybe one . If the field were suppressed from the UI, nobody'd make the mistake of trying to use it, and no doc would be needed. The only reason it might be useful is if the external loader library were optimizing checkouts between builds when multiple libraries were involved. The Check-out Strategy of using svn update as much as possible is short circuited when the local directory is recycled for the checkout of another library. If each library were checked out in its own directory, svn update would work.

          Jesse Glick added a comment -

          when the local directory is recycled for the checkout of another library

          Jenkins does reuse a workspace between builds for checking out/updating library repositories, but a distinct workspace per library, so I think this is not a concern.

          Jesse Glick added a comment - when the local directory is recycled for the checkout of another library Jenkins does reuse a workspace between builds for checking out/updating library repositories, but a distinct workspace per library, so I think this is not a concern.

          Brian Ray added a comment -

          I think I'm seeing something slightly different than what you are describing or maybe misinterpreting what you are describing. Definitely no concerns about functionality, and only a (very) small concern about performance ...

          When a pipeline uses multiple external libraries, it seems to re-use the @libs directory within the workspace on the master and check out the libraries fresh each build, despite no commits to those libraries between builds and despite using the svn update checkout strategy. E.g.,

          This workspace @libs dir on the master doesn't segregate the libraries into separate subdirectories, and contains only the last library loaded at the end of the build. Because the Local module directory is set to ..

           Directory of c:\Jenkins\workspace\Dev-Snippets\x-pipeline-2@libs
          
          2016-10-10  06:18    <DIR>          .
          2016-10-10  06:18    <DIR>          ..
          2016-10-05  12:34    <DIR>          src
          2016-10-10  06:18    <DIR>          vars
                         0 File(s)              0 bytes
                         4 Dir(s)  248,796,782,592 bytes free
          

          The build directory does have the libraries segregated into separate subdirectories, which if applied to the workspace above, would allow for checkout optimization.

           Directory of c:\Jenkins\jobs\Dev-Snippets\jobs\x-pipeline-2\builds\183\libs
          
          2016-10-10  06:18    <DIR>          .
          2016-10-10  06:18    <DIR>          ..
          2016-10-10  06:18    <DIR>          cps_build
          2016-10-10  06:18    <DIR>          cps_XXXXX
          2016-10-10  06:18    <DIR>          cps_general
                         0 File(s)              0 bytes
                         5 Dir(s)  248,788,549,632 bytes free
          

          FWIW this is a dev Jenkins instance on my workstation. Zero executors configured for the master. Since initial report I've updated the core and one of the relevant plugins.

          LTS 2.19.1

          scm-api 1.3
          subversion 2.6
          worfklow-cps-global-lib 2.4

          Brian Ray added a comment - I think I'm seeing something slightly different than what you are describing or maybe misinterpreting what you are describing. Definitely no concerns about functionality, and only a (very) small concern about performance ... When a pipeline uses multiple external libraries, it seems to re-use the @libs directory within the workspace on the master and check out the libraries fresh each build, despite no commits to those libraries between builds and despite using the svn update checkout strategy. E.g., This workspace @libs dir on the master doesn't segregate the libraries into separate subdirectories, and contains only the last library loaded at the end of the build. Because the Local module directory is set to . . Directory of c:\Jenkins\workspace\Dev-Snippets\x-pipeline-2@libs 2016-10-10 06:18 <DIR> . 2016-10-10 06:18 <DIR> .. 2016-10-05 12:34 <DIR> src 2016-10-10 06:18 <DIR> vars 0 File(s) 0 bytes 4 Dir(s) 248,796,782,592 bytes free The build directory does have the libraries segregated into separate subdirectories, which if applied to the workspace above, would allow for checkout optimization. Directory of c:\Jenkins\jobs\Dev-Snippets\jobs\x-pipeline-2\builds\183\libs 2016-10-10 06:18 <DIR> . 2016-10-10 06:18 <DIR> .. 2016-10-10 06:18 <DIR> cps_build 2016-10-10 06:18 <DIR> cps_XXXXX 2016-10-10 06:18 <DIR> cps_general 0 File(s) 0 bytes 5 Dir(s) 248,788,549,632 bytes free — FWIW this is a dev Jenkins instance on my workstation. Zero executors configured for the master. Since initial report I've updated the core and one of the relevant plugins. LTS 2.19.1 scm-api 1.3 subversion 2.6 worfklow-cps-global-lib 2.4

          Jesse Glick added a comment -

          Ah, that indeed looks like a mistake; it should be using a distinct workspace per library.

          Jesse Glick added a comment - Ah, that indeed looks like a mistake; it should be using a distinct workspace per library.

          Jesse Glick added a comment -

          Submitted a proposed fix for that.

          Jesse Glick added a comment - Submitted a proposed fix for that.

          Brian Ray added a comment -

          Thank you. I'm still for suppressing Local module directory from the UI in this context, too.

          Brian Ray added a comment - Thank you. I'm still for suppressing Local module directory from the UI in this context, too.

          Brian Ray added a comment -

          The patch in worflow-cps-global-lib 2.5 speeds up non-initial library checkouts considerably. Thanks.

          Brian Ray added a comment - The patch in worflow-cps-global-lib 2.5 speeds up non-initial library checkouts considerably. Thanks.

          Jon Sten added a comment -

          Just a short note, I created JENKINS-40408 which is a bug introduced due to workflow-cps-global-lib PR 18 which was made to fix this issue.

          Jon Sten added a comment - Just a short note, I created JENKINS-40408 which is a bug introduced due to workflow-cps-global-lib PR 18 which was made to fix this issue.

            Unassigned Unassigned
            brianeray Brian Ray
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: