Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48974

Pipeline Intermittent "Error grabbing Grapes" Issues

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • None
    • Jenkins: 2.73.2.1 workflow-cps-global-lib:2.9
    • 749.v70084559234a_

      Intermittently, when a pipeline uses the "@Grab" notation, it results in the following error: 
      java.lang.RuntimeException: Error grabbing Grapes – [unresolved dependency: commons-codec#commons-codec;1.9: java.util.ConcurrentModificationException at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909)] 

      If this is related to GROOVY-7407, perhaps the recommended WorkaroundGroovy7407WrappingGrapeEngine.java could be imported into workflow-cps-global-lib and called the first time a trusted library is loaded (before it can attempt to grab anything)?

       

          [JENKINS-48974] Pipeline Intermittent "Error grabbing Grapes" Issues

          Steven Deal created issue -
          Steven Deal made changes -
          Summary Original: Pipeline Intermittent "Error grabbing Grapes " Issues New: Pipeline Intermittent "Error grabbing Grapes" Issues
          Jesse Glick made changes -
          Issue Type Original: Task [ 3 ] New: Bug [ 1 ]

          Sam Gleske added a comment -

          This is reliably reproducible if one runs a multibranch pipeline with multiple branches building concurrently attempting to Grab via a globally configured pipeline library.

          Here's an example: https://github.com/samrocketman/jenkins-bootstrap-jervis/issues/19

          Sam Gleske added a comment - This is reliably reproducible if one runs a multibranch pipeline with multiple branches building concurrently attempting to Grab via a globally configured pipeline library. Here's an example:  https://github.com/samrocketman/jenkins-bootstrap-jervis/issues/19

          Sam Van Oort added a comment -

          FWIW I consider this one to carry nontrivial risk of breaking libraries or triggering memory leaks or deadlocks... for now I'm going to wait to see if the Groovy community pulls in the fix and we can consume it downstream. 

           

          Will leave the JIRA open for now in case someone from the community is bold enough to take this up and give it a try.  It's probably pretty simple to apply the change... but testing it and debugging might be a real challenge.

          Sam Van Oort added a comment - FWIW I consider this one to carry nontrivial risk of breaking libraries or triggering memory leaks or deadlocks... for now I'm going to wait to see if the Groovy community pulls in the fix and we can consume it downstream.    Will leave the JIRA open for now in case someone from the community is bold enough to take this up and give it a try.  It's probably pretty simple to apply the change... but testing it and debugging might be a real challenge.
          CloudBees Inc. made changes -
          Remote Link New: This issue links to "CloudBees Internal CD-398 (Web Link)" [ 20147 ]

          Sam Gleske added a comment - - edited

          FTR - this is an issue I still regularly encounter.  I just regrettably tell my users that there's nothing I can do to fix it.   It's a core problem out of my control.

          • Jenkins 2.235.1
            • Embedded Groovy 2.4.12
          • workflow-cps plugin 2.86
          • workflow-cps-global-lib 2.17

          I have (unsuccessfully) tried adding @Grab to steps which can be retried on exception e.g.

          retry(3) {
              grabDependencies()
          } 

          I've even considered sleep-based workarounds in shared pipeline library steps.  I haven't attempted a random-based sleep.

          retry(3) {
              try {
                  grabDependencies()
              } catch(Throwable e) {
                  sleep(1)
                  throw e
              }
          }

          However, users and I still  encounter "Error grabbing Grapes" issue within multibranch pipelines.  I've been silent because this report has been open but I figured I'd post an update since there hasn't been any activity on it.  I didn't want it to be assumed that this isn't an issue any longer.

          Sam Gleske added a comment - - edited FTR - this is an issue I still regularly encounter.  I just regrettably tell my users that there's nothing I can do to fix it.   It's a core problem out of my control. Jenkins 2.235.1 Embedded Groovy 2.4.12 workflow-cps plugin 2.86 workflow-cps-global-lib 2.17 I have (unsuccessfully) tried adding @Grab to steps which can be retried on exception e.g. retry(3) { grabDependencies() } I've even considered sleep-based workarounds in shared pipeline library steps.  I haven't attempted a random-based sleep. retry(3) { try { grabDependencies() } catch(Throwable e) { sleep(1) throw e } } However, users and I still  encounter " Error grabbing Grapes " issue within multibranch pipelines.  I've been silent because this report has been open but I figured I'd post an update since there hasn't been any activity on it.  I didn't want it to be assumed that this isn't an issue any longer.

          Abdennour Toumi added a comment - - edited

          Yes sag47 Yes

          Jenkins Embarrassed me in this area also. 

          A shared library with more than 50 functions (vars) is no useless until fixing this issue : 

          The issue appeared previously, and i was able to fix it by : 

           

            customInitContainers:
             - name: setup-extra-configs
               imagePullPolicy: Always
               image: busybox:latest
               command:
               - sh
               - -c
               args:
               - ' echo ''Setting extra configmaps ..'' \
                 && cp /tmp/extra-config/grapConfig_v2.xml /etc/groovy_home_dir/grapConfig.xml'
               volumeMounts:
               - mountPath: /tmp/extra-config/
                 name: jenkins-extra-config
               - mountPath: /etc/groovy_home_dir
                 name: jenkins-groovy-config

          But now خلاص , we are done. we have to see alternatives

           

          Abdennour Toumi added a comment - - edited Yes sag47  Yes Jenkins Embarrassed me in this area also.  A shared library with more than 50 functions (vars) is no useless until fixing this issue :  The issue appeared previously, and i was able to fix it by :      customInitContainers:   - name: setup-extra-configs     imagePullPolicy: Always     image: busybox:latest     command:     - sh     - -c     args:     -  ' echo ' 'Setting extra configmaps ..' ' \       && cp /tmp/extra-config/grapConfig_v2.xml /etc/groovy_home_dir/grapConfig.xml'     volumeMounts:     - mountPath: /tmp/extra-config/       name: jenkins-extra-config     - mountPath: /etc/groovy_home_dir       name: jenkins-groovy-config But now خلاص , we are done. we have to see alternatives  

          Nico added a comment -

          svanoort: Is this supposed to be fixed at any time in the future? This problem unfortunately prohibits us from using Jenkins with a single managed shared library for many applications with even more branches. It is not unlikely that two branches are being built at the same time (by CI Management tool) which then leads to this behaviour. Seeing that this ticket is open for over three years without any progress worries me.

          I already tried looking into the code of Groovy, but I didn't get very far. I'd like to help but do neither know the internal workings of Groovy nor Ivy/Grapes.

          Thanks in advance.

          Nico added a comment - svanoort : Is this supposed to be fixed at any time in the future? This problem unfortunately prohibits us from using Jenkins with a single managed shared library for many applications with even more branches. It is not unlikely that two branches are being built at the same time (by CI Management tool) which then leads to this behaviour. Seeing that this ticket is open for over three years without any progress worries me. I already tried looking into the code of Groovy, but I didn't get very far. I'd like to help but do neither know the internal workings of Groovy nor Ivy/Grapes. Thanks in advance.

          Jesse Glick added a comment -

          There has been no recent activity on the Groovy issue, and anyway picking up new Groovy releases in Jenkins is a hard problem and there is no concrete plan to do so. The general advice here is to just avoid Grape, and more broadly avoid doing tricky stuff in Pipeline script (whether in libraries or not). JENKINS-26635 is designed to make it as easy as possible to extract blocks of Groovy code from Pipeline script intact and move them to script files that can be run from external processes.

          Jesse Glick added a comment - There has been no recent activity on the Groovy issue, and anyway picking up new Groovy releases in Jenkins is a hard problem and there is no concrete plan to do so. The general advice here is to just avoid Grape, and more broadly avoid doing tricky stuff in Pipeline script (whether in libraries or not). JENKINS-26635 is designed to make it as easy as possible to extract blocks of Groovy code from Pipeline script intact and move them to script files that can be run from external processes.

            basil Basil Crow
            stevendeal Steven Deal
            Votes:
            10 Vote for this issue
            Watchers:
            21 Start watching this issue

              Created:
              Updated:
              Resolved: