Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48974

Pipeline Intermittent "Error grabbing Grapes" Issues

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • None
    • Jenkins: 2.73.2.1 workflow-cps-global-lib:2.9
    • 749.v70084559234a_

      Intermittently, when a pipeline uses the "@Grab" notation, it results in the following error: 
      java.lang.RuntimeException: Error grabbing Grapes – [unresolved dependency: commons-codec#commons-codec;1.9: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)] 

      If this is related to GROOVY-7407, perhaps the recommended WorkaroundGroovy7407WrappingGrapeEngine.java could be imported into workflow-cps-global-lib and called the first time a trusted library is loaded (before it can attempt to grab anything)?

       

          [JENKINS-48974] Pipeline Intermittent "Error grabbing Grapes" Issues

          Abdennour Toumi added a comment - - edited

          Yes sag47 Yes

          Jenkins Embarrassed me in this area also. 

          A shared library with more than 50 functions (vars) is no useless until fixing this issue : 

          The issue appeared previously, and i was able to fix it by : 

           

            customInitContainers:
             - name: setup-extra-configs
               imagePullPolicy: Always
               image: busybox:latest
               command:
               - sh
               - -c
               args:
               - ' echo ''Setting extra configmaps ..'' \
                 && cp /tmp/extra-config/grapConfig_v2.xml /etc/groovy_home_dir/grapConfig.xml'
               volumeMounts:
               - mountPath: /tmp/extra-config/
                 name: jenkins-extra-config
               - mountPath: /etc/groovy_home_dir
                 name: jenkins-groovy-config

          But now خلاص , we are done. we have to see alternatives

           

          Abdennour Toumi added a comment - - edited Yes sag47  Yes Jenkins Embarrassed me in this area also.  A shared library with more than 50 functions (vars) is no useless until fixing this issue :  The issue appeared previously, and i was able to fix it by :      customInitContainers:   - name: setup-extra-configs     imagePullPolicy: Always     image: busybox:latest     command:     - sh     - -c     args:     -  ' echo ' 'Setting extra configmaps ..' ' \       && cp /tmp/extra-config/grapConfig_v2.xml /etc/groovy_home_dir/grapConfig.xml'     volumeMounts:     - mountPath: /tmp/extra-config/       name: jenkins-extra-config     - mountPath: /etc/groovy_home_dir       name: jenkins-groovy-config But now خلاص , we are done. we have to see alternatives  

          Nico added a comment -

          svanoort: Is this supposed to be fixed at any time in the future? This problem unfortunately prohibits us from using Jenkins with a single managed shared library for many applications with even more branches. It is not unlikely that two branches are being built at the same time (by CI Management tool) which then leads to this behaviour. Seeing that this ticket is open for over three years without any progress worries me.

          I already tried looking into the code of Groovy, but I didn't get very far. I'd like to help but do neither know the internal workings of Groovy nor Ivy/Grapes.

          Thanks in advance.

          Nico added a comment - svanoort : Is this supposed to be fixed at any time in the future? This problem unfortunately prohibits us from using Jenkins with a single managed shared library for many applications with even more branches. It is not unlikely that two branches are being built at the same time (by CI Management tool) which then leads to this behaviour. Seeing that this ticket is open for over three years without any progress worries me. I already tried looking into the code of Groovy, but I didn't get very far. I'd like to help but do neither know the internal workings of Groovy nor Ivy/Grapes. Thanks in advance.

          Jesse Glick added a comment -

          There has been no recent activity on the Groovy issue, and anyway picking up new Groovy releases in Jenkins is a hard problem and there is no concrete plan to do so. The general advice here is to just avoid Grape, and more broadly avoid doing tricky stuff in Pipeline script (whether in libraries or not). JENKINS-26635 is designed to make it as easy as possible to extract blocks of Groovy code from Pipeline script intact and move them to script files that can be run from external processes.

          Jesse Glick added a comment - There has been no recent activity on the Groovy issue, and anyway picking up new Groovy releases in Jenkins is a hard problem and there is no concrete plan to do so. The general advice here is to just avoid Grape, and more broadly avoid doing tricky stuff in Pipeline script (whether in libraries or not). JENKINS-26635 is designed to make it as easy as possible to extract blocks of Groovy code from Pipeline script intact and move them to script files that can be run from external processes.

          John Behm added a comment -

          Any updates on this?

          Can't use any generated Java API clients in my Groovy code, even tho a different library is included just fine, but when trying to migrate to a newer version, I do get the same error as mentioned above.

          John Behm added a comment - Any updates on this? Can't use any generated Java API clients in my Groovy code, even tho a different library is included just fine, but when trying to migrate to a newer version, I do get the same error as mentioned above.

          Edgars Batna added a comment - - edited

          If the advice is "to just avoid Grape, and more broadly avoid doing tricky stuff in Pipeline script (whether in libraries or not)", then this is way more than just a "minor" issue. Things like this prevent us from using the Pipelines effectively. It should be said that while it's called "Groovy" in Jenkins, really this is no longer real Groovy...

          Edgars Batna added a comment - - edited If the advice is "to just avoid Grape, and more broadly avoid doing tricky stuff in Pipeline script (whether in libraries or not)", then this is way more than just a "minor" issue. Things like this prevent us from using the Pipelines effectively. It should be said that while it's called "Groovy" in Jenkins, really this is no longer real Groovy...

          John Behm added a comment - - edited

          We did have multiple problems with upgrading our external Java library.

          First off, Jenkins caches those external Java libraries and thus you will not see that your grape config with your Artifactory/Maven Central, etc. credentials may already be outdated. Thus check your grapeConfig.xml (iirc) and its validity.

           

          Second: When developing a wrapper around your e.g. Java Client library (that's generated from an OpenAPI Spec) You can use these annotations in order to prevent the caching and to always fetch a fresh artifact even tho the version might not have been changed.

           

          @Grab( 
              group='cs.whatever', 
              module='cs-whatever-harbor-client', 
              version='2.0-SNAPSHOT', 
              changing=false 
          ) 
          //changing=true can be used in order to update the cached Java library without the library having a new version
          

          Setting changing=true will force Jenkins to always pull a fresh Java artifact/library from your repository.

          John Behm added a comment - - edited We did have multiple problems with upgrading our external Java library. First off, Jenkins caches those external Java libraries and thus you will not see that your grape config with your Artifactory/Maven Central, etc. credentials may already be outdated. Thus check your grapeConfig.xml (iirc) and its validity.   Second: When developing a wrapper around your e.g. Java Client library (that's generated from an OpenAPI Spec) You can use these annotations in order to prevent the caching and to always fetch a fresh artifact even tho the version might not have been changed.   @Grab( group= 'cs.whatever' , module= 'cs-whatever-harbor-client' , version= '2.0-SNAPSHOT' , changing= false ) //changing= true can be used in order to update the cached Java library without the library having a new version Setting changing=true will force Jenkins to always pull a fresh Java artifact/library from your repository.

          Matt Margolin added a comment - - edited

          In our shared library, we use various @Grab annotations. At times, when multiple pipeline jobs that use this shared library are run concurrently, we encounter the same ConcurrentModificationException problem that causes some jobs to fail at compile time.

          A temporary solution we have found is to use the annotation

          @GrabConfig(autoDownload=false)
          

          alongside all @Grab annotations.  This solves the ConcurrentModificationExceptions we encounter, but has an additional requirement.  This will require that artifact/libraries specified via @Grab already be present on your jenkins servers when jobs are run. Artifacts/libraries can be downloaded onto your servers by using the grape cli tool.

          To solve this requirement, we defined our dependencies in our global shared library, and then use a multibranch job that simply runs the grape cli command via a sh step in case there are any new dependencies or version changes for our master branch. This serves as a single point of entry for downloading artifacts, in essence eliminating concurrent modifications.

          This can probably also be automated using configuration management or alternative solutions as well, but they key is using @GrabConfig and not modifying the grape cache concurrently.

          Matt Margolin added a comment - - edited In our shared library, we use various @Grab annotations. At times, when multiple pipeline jobs that use this shared library are run concurrently, we encounter the same ConcurrentModificationException problem that causes some jobs to fail at compile time. A temporary solution we have found is to use the annotation @GrabConfig(autoDownload= false ) alongside all @Grab annotations.  This solves the ConcurrentModificationExceptions we encounter, but has an additional requirement.  This will require that artifact/libraries specified via @Grab already be present on your jenkins servers when jobs are run. Artifacts/libraries can be downloaded onto your servers by using the  grape cli tool. To solve this requirement, we defined our dependencies in our global shared library, and then use a multibranch job that simply runs the grape cli command via a  sh step  in case there are any new dependencies or version changes for our master branch. This serves as a single point of entry for downloading artifacts, in essence eliminating concurrent modifications. This can probably also be automated using configuration management or alternative solutions as well, but they key is using @GrabConfig and not modifying the grape cache concurrently.

          three added a comment -

          I trace the code of Grab,and I found the grab logic is in  the dependency ivy,then I upgrade the ivy version. The problem dosen't appear thease days.

          three added a comment - I trace the code of Grab,and I found the grab logic is in  the dependency ivy,then I upgrade the ivy version. The problem dosen't appear thease days.

          Wendi added a comment - - edited

          three_zhang what ivy version are you using? 

          our jenkins info:

          master: 2.387.2

          ivy:2.4

          still have this issue

          we use @Library to load the shared library

          Wendi added a comment - - edited three_zhang what ivy version are you using?  our jenkins info: master: 2.387.2 ivy:2.4 still have this issue we use @Library to load the shared library

          Basil Crow added a comment -

          Basil Crow added a comment - Fixed in jenkinsci/pipeline-groovy-lib-plugin#148 . Released in 749.v70084559234a_ .

            basil Basil Crow
            stevendeal Steven Deal
            Votes:
            10 Vote for this issue
            Watchers:
            21 Start watching this issue

              Created:
              Updated:
              Resolved: