Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-75141

Concurrent '@Grab' May Lead to 'Error grabbing Grapes' Error Due to Lack of Lock on Ivy Cache

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Trivial Trivial
    • None
    • Jenkins: 2.462.1
      pipeline-groovy-lib-plugin: 745.vdf6077913de0
    • 749.v70084559234a_

      I tested pipeline-groovy-lib-plugin 745.vdf6077913de0 with Jenkins 2.462.1 and got following results.

      Job 'testParallel':

      def tasks = ["1": { build job: 'testGrab', parameters: [string(name: 'id', value: '1')] }
                  , "2": { build job: 'testGrab', parameters: [string(name: 'id', value: '2')] }
                  , "3": { build job: 'testGrab', parameters: [string(name: 'id', value: '3')] }
                  , "4": { build job: 'testGrab', parameters: [string(name: 'id', value: '4')] }
                  , "5": { build job: 'testGrab', parameters: [string(name: 'id', value: '5')] }
                  , "6": { build job: 'testGrab', parameters: [string(name: 'id', value: '6')] }
                  , "7": { build job: 'testGrab', parameters: [string(name: 'id', value: '7')] }
                  , "8": { build job: 'testGrab', parameters: [string(name: 'id', value: '8')] }
                  , "9": { build job: 'testGrab', parameters: [string(name: 'id', value: '9')] }
                  , "10": { build job: 'testGrab', parameters: [string(name: 'id', value: '10')] }
                  , "11": { build job: 'testGrab', parameters: [string(name: 'id', value: '11')] }
                  , "12": { build job: 'testGrab', parameters: [string(name: 'id', value: '12')] }
                  , "13": { build job: 'testGrab', parameters: [string(name: 'id', value: '13')] }
                  , "14": { build job: 'testGrab', parameters: [string(name: 'id', value: '14')] }
                  ]
      parallel tasks 

      Downstream job 'testGrab'

      @Grab(group = 'org.apache.kafka', module = 'kafka-clients', version = '2.0.0')
      import org.apache.kafka.clients.producer.KafkaProducer
      sleep 10
      echo 'hello' 

      The downstream job still failed with following logs:

      org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
      General error during conversion: Error grabbing Grapes -- [unresolved dependency: org.apache.kafka#kafka-clients;2.0.0: not found]
      
      java.lang.RuntimeException: Error grabbing Grapes -- [unresolved dependency: org.apache.kafka#kafka-clients;2.0.0: not found]
      	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
      	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
      	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
      	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:83)
      	at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:77)
      	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:250)
      	at groovy.grape.GrapeIvy.getDependencies(GrapeIvy.groovy:422)
      	at jdk.internal.reflect.GeneratedMethodAccessor129.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:169)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:59)
      	at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:569)
      	at groovy.grape.GrapeIvy$resolve$1.callCurrent(Unknown Source)
      	at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:536)
      	at groovy.grape.GrapeIvy$resolve$0.callCurrent(Unknown Source)
      	at groovy.grape.GrapeIvy.grab(GrapeIvy.groovy:254)
      	at groovy.grape.Grape.grab(Grape.java:167)
      	at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:376)
      	at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:346)
      	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:966)
      	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:626)
      	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
      	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
      	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
      	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
      	at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163)
      	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:190)
      	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:175)
      	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:652)
      	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:598)
      	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:335)
      	at hudson.model.ResourceController.execute(ResourceController.java:101)
      	at hudson.model.Executor.run(Executor.java:446)
      
      1 error
      
      	at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:309)
      	at org.codehaus.groovy.control.ErrorCollector.addException(ErrorCollector.java:155)
      	at org.codehaus.groovy.control.SourceUnit.addException(SourceUnit.java:335)
      	at groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:385)
      	at org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:346)
      	at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:966)
      	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:626)
      	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:602)
      	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:579)
      	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:323)
      	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:293)
      	at PluginClassLoader for script-security//org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox$Scope.parse(GroovySandbox.java:163)
      	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:190)
      	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:175)
      	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:652)
      	at PluginClassLoader for workflow-cps//org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:598)
      	at PluginClassLoader for workflow-job//org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:335)
      	at hudson.model.ResourceController.execute(ResourceController.java:101)
      	at hudson.model.Executor.run(Executor.java:446)
      Finished: FAILURE 

      After further investigation, I've found the default Ivy cache has no lock protection. 

      https://ant.apache.org/ivy/history/2.5.3/concept.html#cache

      ... This part of the cache can be shared if you use a well suited lock strategy. 

      https://ant.apache.org/ivy/history/2.5.3/settings/caches.html

      lockStrategy ... defaults to no-lock 

      https://ant.apache.org/ivy/history/2.5.3/settings/lock-strategies.html

      no-lock
      This lock strategy actually performs no locking at all, and thus should not be used in an environment where the cache is shared by multiple processes. 

      After customizing the ~/.groovy/grapeConfig.yaml (Check defaultGrapeConfig.yaml if no customized grapeConfig.yaml is provided) by setting 'lockStrategy' to 'artifact-lock-nio', the testParallel succeeds. Jenkins needs to be restarted to load the customized grapeConfig.yaml.

      <ivysettings>
          <settings defaultResolver="downloadGrapes"/>
          <caches lockStrategy="artifact-lock-nio" />
          ...
      </ivysettings> 

          [JENKINS-75141] Concurrent '@Grab' May Lead to 'Error grabbing Grapes' Error Due to Lack of Lock on Ivy Cache

          Basil Crow added a comment -

          danielcwc I made artifact-lock-nio the default in jenkinsci/pipeline-groovy-lib-plugin#148 (released in 749.v70084559234a_) and proposed the same change upstream in Groovy in apache/groovy#2142. Anyway, this is a duplicate of JENKINS-48974, so please leave any additional comments there.

          Basil Crow added a comment - danielcwc  I made artifact-lock-nio the default in jenkinsci/pipeline-groovy-lib-plugin#148 (released in 749.v70084559234a_ ) and proposed the same change upstream in Groovy in apache/groovy#2142 . Anyway, this is a duplicate of JENKINS-48974 , so please leave any additional comments there.

            Unassigned Unassigned
            danielcwc wangcheng cai
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: