Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69573

Cache of Global Pipeline Libraries May Fail to Be Created Until Next Expiration

XMLWordPrintable

    • 710.v4b_94b_077a_808

      Environments:

      • Jenkins Version: 2.346.3
      • Plugin Version: 612.v84da_9c54906d

      Steps:

      1. Turn on cache for 'Global Pipeline Libraries' in 'Configure System' and specify a refresh interval greater than 0.
      2. Cache expiration detected, the plugin deletes the existing cache folder and creates a new empty cache folder with the name generated from some hash function.
      3. The plugin fails to fetch library from GitHub repository (e.g. the GitHub server is under maintenance).
      4. All subsequent pipelines trying to load the library from the cache folder fail with following errors.  
        Loading library some_library_name@some_tag_version
Library some_library_name@some_tag_version is cached. Copying from home.
ERROR: Library shared-lib expected to contain at least one of src or vars directories
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: Loading libraries failed

      By navigating to '$JENKINS_HOME/global-libraries-cache', the cache folder is there but the contents are corrupted.

      Only after next expiration, the plugin will attempt another try to refresh the cache. The situation is even worse when the refresh interval is relatively long.

      The root cause is that the plugin determines the existence of a library cache by checking the existence of the cache folder and uses the modification time of the cache folder to determine if the cache has expired, regardless of the content of the folder.

      The improvement I'd suggest is that removing the newly created cache folder if the plugin fails to fetch libraries, to let the next pipeline have the chance to re-create the cache, rather than to wait for next cache expiration.

          [JENKINS-69573] Cache of Global Pipeline Libraries May Fail to Be Created Until Next Expiration

          Grégory Boué added a comment -

          Hi,

          I had exactly the same issue with Jenkins Jenkins 2.361.2 and plugin 613.v9c41a_160233f.

          The behaviour that led to this error was the following.

          • I created a multibranch pipeline, with 8 branches that uses a global library.
          • After the creation, the job look for branches and create 8 jobs simultaneously
          • Then in the first job has the following error : 
          •  > git fetch --no-tags --force --progress -- [https://gitlab-repo-gpf.apps.eul.sncf.fr/dosn/2608/pic/shared-libs/common-libs/common_libs.git]
 +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url [https://gitlab-repo-gpf.apps.eul.sncf.fr/dosn/2608/pic/shared-libs/common-libs/common_libs.git]
 # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision d9f937fd4c7bff2f0b5b30222358d31a89353d12 (master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f d9f937fd4c7bff2f0b5b30222358d31a89353d12 # timeout=10
ERROR: Checkout failed
java.lang.NullPointerException
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2741)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$1100(CliGitAPIImpl.java:87)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:3058)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1362)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$doRetrieve$1(SCMSourceRetriever.java:201)
	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:148)
	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:200)
	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:137)
	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:243)
	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:150)
	at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125)
	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1087)
	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624)
	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:562)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:514)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:336)
	at hudson.model.ResourceController.execute(ResourceController.java:107)
	at hudson.model.Executor.run(Executor.java:449)
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
          • The seven other pipeline had the same error than describe here 

          Grégory Boué added a comment - Hi, I had exactly the same issue with Jenkins Jenkins 2.361.2 and plugin 613.v9c41a_160233f. The behaviour that led to this error was the following. I created a multibranch pipeline, with 8 branches that uses a global library. After the creation, the job look for branches and create 8 jobs simultaneously Then in the first job has the following error :  > git fetch --no-tags --force --progress -- [https: //gitlab-repo-gpf.apps.eul.sncf.fr/dosn/2608/pic/shared-libs/common-libs/common_libs.git] +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url [https: //gitlab-repo-gpf.apps.eul.sncf.fr/dosn/2608/pic/shared-libs/common-libs/common_libs.git] # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision d9f937fd4c7bff2f0b5b30222358d31a89353d12 (master) > git config core.sparsecheckout # timeout=10 > git checkout -f d9f937fd4c7bff2f0b5b30222358d31a89353d12 # timeout=10 ERROR: Checkout failed java.lang.NullPointerException at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2741) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$1100(CliGitAPIImpl.java:87) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:3058) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1362) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.lambda$doRetrieve$1(SCMSourceRetriever.java:201) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrySCMOperation(SCMSourceRetriever.java:148) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.doRetrieve(SCMSourceRetriever.java:200) at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:137) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:243) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:150) at org.jenkinsci.plugins.workflow.libs.LibraryDecorator$1.call(LibraryDecorator.java:125) at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1087) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:624) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142) at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:562) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:514) at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:336) at hudson.model.ResourceController.execute(ResourceController.java:107) at hudson.model.Executor.run(Executor.java:449) java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread The seven other pipeline had the same error than describe here 

          Steve Boardwell added a comment -

          Is there any chance of this being moved forward? Would a pull request be welcome? I'm not familiar with the plugin but I could have a look.

          Steve Boardwell added a comment - Is there any chance of this being moved forward? Would a pull request be welcome? I'm not familiar with the plugin but I could have a look.

          Steve Boardwell added a comment -

          I have added a PR to fix this issue https://github.com/jenkinsci/pipeline-groovy-lib-plugin/pull/125

          Steve Boardwell added a comment - I have added a PR to fix this issue https://github.com/jenkinsci/pipeline-groovy-lib-plugin/pull/125

          Steve Boardwell added a comment -

          The PR has been merged and released in https://github.com/jenkinsci/pipeline-groovy-lib-plugin/releases/tag/710.v4b_94b_077a_808 

          We can close this ticket.

          Steve Boardwell added a comment - The PR has been merged and released in https://github.com/jenkinsci/pipeline-groovy-lib-plugin/releases/tag/710.v4b_94b_077a_808   We can close this ticket.

          Valentin Delaye added a comment -

          Thanks for fixing this. I was facing this issue 1-2 times per year and needed sometime to rm the cache directly on the volume. I'm happy this is not needed anymore.

          Valentin Delaye added a comment - Thanks for fixing this. I was facing this issue 1-2 times per year and needed sometime to rm the cache directly on the volume. I'm happy this is not needed anymore.

          Ray Kivisto added a comment -

          Marking as resolved, as it's released here https://github.com/jenkinsci/pipeline-groovy-lib-plugin/releases/tag/710.v4b_94b_077a_808

          Ray Kivisto added a comment - Marking as resolved, as it's released here https://github.com/jenkinsci/pipeline-groovy-lib-plugin/releases/tag/710.v4b_94b_077a_808

            sboardwell Steve Boardwell
            danielcwc wangcheng cai
            Votes:
            4 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: