Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63470

Multibranch pipeline shared library permission error

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Component/s: pipeline
    • Labels:
      None
    • Environment:
      Jenkins 2.235.2
      pipeline: multibranch 2.22
    • Similar Issues:

      Description

      I'm running Jenkins in a docker container, jobs run in separate containers. I'm currently getting an error on multi branch pipeline using shared library. the job clones the shared library the first time, however from there on it gives a permission error. I can't seem to find similar questions or solutions online.

       

      Loading library <SHARED_LIBRARY>@develop
      Attempting to resolve master from remote references...
       > git --version # timeout=10
      using GIT_SSH to set credentials <CREDENTIAL_NAME>
       > git ls-remote -h -- <GIT_LINK> # timeout=10
      Found match: refs/heads/develop revision <COMMIT_HASH>
      ERROR: Checkout failed
      java.nio.file.AccessDeniedException: /${ITEM_ROOTDIR}
          at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
          at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
          at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
          at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
          at java.nio.file.Files.createDirectory(Files.java:674)
          at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
          at java.nio.file.Files.createDirectories(Files.java:767)
          at hudson.FilePath.mkdirs(FilePath.java:3260)
          at hudson.FilePath.access$1300(FilePath.java:211)
          at hudson.FilePath$Mkdirs.invoke(FilePath.java:1251)
          at hudson.FilePath$Mkdirs.invoke(FilePath.java:1247)
          at hudson.FilePath.act(FilePath.java:1075)
          at hudson.FilePath.act(FilePath.java:1058)
          at hudson.FilePath.mkdirs(FilePath.java:1243)
          at hudson.plugins.git.GitSCM.createClient(GitSCM.java:828)
          at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1166)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
          at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$doRetrieve$1(SCMSourceRetriever.java:157)
          at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:107)
          at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:156)
          at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:96)
          at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:157)
          at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:138)
          at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
          at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1065)
          at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
          at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
          at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
          at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
          at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
          at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
          at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
          at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
          at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
          at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
          at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522)
          at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:337)
          at hudson.model.ResourceController.execute(ResourceController.java:97)
          at hudson.model.Executor.run(Executor.java:428)
      ERROR: Maximum checkout retry attempts reached, aborting
      org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
      WorkflowScript: Loading libraries failed
      
      1 error
      
      	at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
      	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
      	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
      	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
      	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
      	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
      	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
      	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
      	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:337)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:428)
      Finished: FAILURE
      

      The shared pipeline is cloned on master when the job first runs, it's cloned on the path

      <JENKINS_HOME>/jobs/<JOB_NAME>/branches/<BRANCH_NAME>/workspace/<GIT_REPO_NAME>/<SHARED_LIBRARY_BRANCH_NAME@libs/<SHARED_LIBRARY_GIT_REPO_NAME>
      

      {{}}

      I can't see it on the slave container. I believe Jenkins agent has the privileges because it's running as admin and the docker container has permissions on the path. Here's logs from the first run, as you can see it can clone the shared pipeline and start the slave container, however if you look in the post (2nd run of the same job). It fails to clone the shared pipeline.

      Branch indexing
      Running as admin
       > git rev-parse --is-inside-work-tree # timeout=10
      Setting origin to git@github.com:<GIT_REPO>.git
       > git config remote.origin.url git@github.com:<GIT_REPO>.git # timeout=10
      Fetching origin...
      Fetching upstream changes from origin
       > git --version # timeout=10
       > git config --get remote.origin.url # timeout=10
      using GIT_SSH to set credentials <GITHUB_SSH_KEY>
       > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
      Seen branch in repository origin/develop
      Seen branch in repository origin/master
      Seen 2 remote branches
      Obtained Jenkinsfile from <GIT_HASH>
      Running in Durability level: PERFORMANCE_OPTIMIZED
      Loading library <SHARED_LIBRARY>@develop
      Attempting to resolve develop from remote references...
       > git --version # timeout=10
      using GIT_SSH to set credentials <GITHUB_SSH_KEY>
       > git ls-remote -h -- git@github.com:<SHARED_LIBRARY>.git # timeout=10
      Found match: refs/heads/develop revision <GIT_HASH>
      using credential <GITHUB_SSH_KEY>
      Cloning the remote Git repository
      Cloning with configured refspecs honoured and without tags
      Cloning repository git@github.com:<SHARED_LIBRARY>.git
       > git init /var/jenkins_home/jobs/<JOB_NAME/GIT_REPO_NAME>/branches/develop/workspace/<JOB_NAME/GIT_REPO_NAME>/develop@libs/<SHARED_LIBRARY> # timeout=10
      Fetching upstream changes from git@github.com:<SHARED_LIBRARY>.git
       > git --version # timeout=10
      using GIT_SSH to set credentials <GITHUB_SSH_KEY>
       > git fetch --no-tags --progress -- git@github.com:<SHARED_LIBRARY>.git +refs/heads/*:refs/remotes/origin/* # timeout=10
       > git config remote.origin.url git@github.com:<SHARED_LIBRARY>.git # timeout=10
       > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
       > git config remote.origin.url git@github.com:<SHARED_LIBRARY>.git # timeout=10
      Fetching without tags
      Fetching upstream changes from git@github.com:<SHARED_LIBRARY>.git
      using GIT_SSH to set credentials <GITHUB_SSH_KEY>
       > git fetch --no-tags --progress -- git@github.com:<SHARED_LIBRARY>.git +refs/heads/*:refs/remotes/origin/* # timeout=10
      Checking out Revision <GIT_HASH> (develop)
       > git config core.sparsecheckout # timeout=10
       > git checkout -f <GIT_HASH> # timeout=10
      Commit message: "<SHARED_LIBRARY_COMMIT_MESSAGE>"
      First time build. Skipping changelog.
      [Pipeline] Start of Pipeline
      [Pipeline] node
      Running on android-slave on docker in /home/jenkins/workspace/<JOB_NAME>
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Declarative: Checkout SCM)
      [Pipeline] checkout
      

      Jenkinsfile

      #!groovy
      @Library("<SHARED_LIBRARY>")
      
      pipeline {
          agent {
              label 'android-slave'
          }
          stages {
              stage("Stage1") {
                  steps {
                      script {
                          println "Testing multi branch pipeline"
                      }
                  }
              }
          }
      }
      

       

       

       

       

        Attachments

          Activity

          Hide
          rjmpsmith Robert Smith added a comment -

          I'm seeing the same thing and thought that I would share my experiences in case they help track down the root cause.

          First, it's interesting to note that the issue only seems to affect multi-branch pipelines. Standard pipelines work fine with shared libraries.

          Also of note, each time you restart Jenkins you get one more good build before it fails again in the same way.

          For us, the issue seems to have begun when we upgraded the Branch API plugin from version 2.5.5 to 2.5.8. We don't see the issue with version 2.5.5.

          We have discovered that, when we see this error, Jenkins is using the LITERAL path "/${ITEM_ROOTDIR}' as if the variable is not being resolved properly. When we modified our Jenkins Dockerfile to create this directory and change the owner to the user as which Jenkins runs, the pipeline began working again.

          I can confirm that it is actually using this path to store the shared library, so that's not a very good work around:

          root# docker exec -ti 65285767be6a /bin/bash
          
          bash-4.4$ cd /\$\{ITEM_ROOTDIR\}/
          bash-4.4$ ls
          
          workspace
          
          bash-4.4$ ls workspace/
          shared-lib-hello-world_master@libs  workspaces.txt 

          I had originally suspected that this had to do with SELinux and labels, so spent quite a bit of time examining Docker security settings, permissions and ACLs. Along the way, I discovered that if you gave the user running the Jenkins container explicit ACLs on /var/lib/docker, that also seemed to work around the issue, though I'm not entirely certain why that is. Note that this only works if run after the pipeline is created in the UI.

          setfacl -R -d -m u:jenkins:rwx /var/lib/docker && setfacl -R -m u:jenkins:rwx /var/lib/docker
          Show
          rjmpsmith Robert Smith added a comment - I'm seeing the same thing and thought that I would share my experiences in case they help track down the root cause. First, it's interesting to note that the issue only seems to affect multi-branch pipelines. Standard pipelines work fine with shared libraries. Also of note, each time you restart Jenkins you get one more good build before it fails again in the same way. For us, the issue seems to have begun when we upgraded the Branch API plugin from version 2.5.5 to 2.5.8. We don't see the issue with version 2.5.5. We have discovered that, when we see this error, Jenkins is using the LITERAL path "/${ITEM_ROOTDIR}' as if the variable is not being resolved properly. When we modified our Jenkins Dockerfile to create this directory and change the owner to the user as which Jenkins runs, the pipeline began working again. I can confirm that it is actually using this path to store the shared library, so that's not a very good work around: root# docker exec -ti 65285767be6a /bin/bash bash-4.4$ cd /\$\{ITEM_ROOTDIR\}/ bash-4.4$ ls workspace bash-4.4$ ls workspace/ shared-lib-hello-world_master@libs workspaces.txt I had originally suspected that this had to do with SELinux and labels, so spent quite a bit of time examining Docker security settings, permissions and ACLs. Along the way, I discovered that if you gave the user running the Jenkins container explicit ACLs on /var/lib/docker, that also seemed to work around the issue, though I'm not entirely certain why that is. Note that this only works if run after the pipeline is created in the UI. setfacl -R -d -m u:jenkins:rwx / var /lib/docker && setfacl -R -m u:jenkins:rwx / var /lib/docker
          Hide
          kilese Kathryn Royall added a comment -

          There is an underlying issue in the versions of branch-api-plugin newer than 2.5.5 that cannot properly handle legacy paths with ITEM_ROOTDIR. The simplest fix, is updating the workspaceDir in your Jenkins config.xml to use JENKINS_HOME instead such as:
          <workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}</workspaceDir>
          <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
          For a more in depth analysis continue reading below.

          The branch-api-plugin 2.5.5 and earlier would still have a bad resolution on legacy paths, but would return null to indicate that it failed to resolve the path. This would cause Jenkins to perform the resolution itself here: https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/model/Jenkins.java#L2481

          In the versions of branch-api-plugin newer than 2.5.5, a cache was introduced with this PR: https://github.com/jenkinsci/branch-api-plugin/pull/188 that would cache an erroneous root directory causing the first run to fail to resolve returning null to Jenkins, but future runs would return the erroneous root directory with ${ITEM_ROOTDIR}.

          The REGEX processing of the root directory should probably be updated in the branch-api-plugin:

          https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/WorkspaceLocatorImpl.java#L330

          It can be seen here that it only performs substitutions on ${JENKINS_HOME}, but the warning below indicates that legacy is not supported. However, the if statement will not properly process this test case:

          workspaceDir.set(r.jenkins, "${ITEM_ROOTDIR}/workspace/${ITEM_FULLNAME}");
          assertEquals("old default and also deprecated ${ITEM_FULLNAME}", null, WorkspaceLocatorImpl.getWorkspaceRoot(r.jenkins)); 

          By updating the pattern to below, legacy paths can be better identified (with a warning and return null).

          private static final Pattern GOOD_RAW_WORKSPACE_DIR = Pattern.compile("(((?![$][{]ITEM_ROOTDIR[}]).)+)[/\\\\][$][{]ITEM_FULL_?NAME[}][/\\\\]?");
          

           

          Show
          kilese Kathryn Royall added a comment - There is an underlying issue in the versions of branch-api-plugin newer than 2.5.5 that cannot properly handle legacy paths with ITEM_ROOTDIR. The simplest fix, is updating the workspaceDir in your Jenkins config.xml to use JENKINS_HOME instead such as: <workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULL_NAME}</workspaceDir> <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir> For a more in depth analysis continue reading below. The branch-api-plugin 2.5.5 and earlier would still have a bad resolution on legacy paths, but would return null to indicate that it failed to resolve the path. This would cause Jenkins to perform the resolution itself here:  https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/jenkins/model/Jenkins.java#L2481 In the versions of branch-api-plugin newer than 2.5.5, a cache was introduced with this PR:  https://github.com/jenkinsci/branch-api-plugin/pull/188  that would cache an erroneous root directory causing the first run to fail to resolve returning null to Jenkins, but future runs would return the erroneous root directory with ${ITEM_ROOTDIR}. The REGEX processing of the root directory should probably be updated in the branch-api-plugin: https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/WorkspaceLocatorImpl.java#L330 It can be seen here that it only performs substitutions on ${JENKINS_HOME}, but the warning below indicates that legacy is not supported. However, the if statement will not properly process this test case: workspaceDir.set(r.jenkins, "${ITEM_ROOTDIR}/workspace/${ITEM_FULLNAME}" ); assertEquals( "old default and also deprecated ${ITEM_FULLNAME}" , null , WorkspaceLocatorImpl.getWorkspaceRoot(r.jenkins)); By updating the pattern to below, legacy paths can be better identified (with a warning and return null). private static final Pattern GOOD_RAW_WORKSPACE_DIR = Pattern.compile( "(((?![$][{]ITEM_ROOTDIR[}]).)+)[/\\\\][$][{]ITEM_FULL_?NAME[}][/\\\\]?" );  

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            sliide_jay jay Rawal
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated: