Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56567

Build fails when reading more than 125 environment credentials at the same time

      Builds fail with the following error when reading more than 125 credentials into the environment in the Jenkinsfile.

      This prevents us from deploying our services, hence is very blocking.
       

      org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
      WorkflowScript: -1: Map expressions can only contain up to 125 entries @ line -1, column -1.
      1 error
      
      	at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
      	at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
      	at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
      	at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
      	at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
      	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
      	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
      	at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
      	at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.lambda$doParse$0(CpsGroovyShell.java:135)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:136)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:132)
      	at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:560)
      	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:521)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:320)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:429)
      Finished: FAILURE
      

      It seems weird to use a data structure that only supports up to 125 values in this context, since performance is not ever going to be blocking there. 

      Note that this error does not happen if the credentials are registered in Jenkins and the environment variables are created from string litterals, which is why I beleive the problem comes from this plugin (credentials-binding).

          [JENKINS-56567] Build fails when reading more than 125 environment credentials at the same time

          Jesse Glick added a comment -

          There are no steps to reproduce but from the sound of it I assume this is something specific to Declarative Pipeline, as the credentials-binding plugin itself does nothing special with Groovy.

          Jesse Glick added a comment - There are no steps to reproduce but from the sound of it I assume this is something specific to Declarative Pipeline, as the credentials-binding plugin itself does nothing special with Groovy.

          Devin Nusbaum added a comment -

          The underlying issue is the same as JENKINS-47363. Here is the commit to groovy-cps that added the error message for map expressions with more than 125 entries. Maybe something could be done in Declarative to change the underlying construct used for environment blocks so that it isn't using Groovy map expressions directly.

          Devin Nusbaum added a comment - The underlying issue is the same as  JENKINS-47363 . Here  is the commit to groovy-cps that added the error message for map expressions with more than 125 entries. Maybe something could be done in Declarative to change the underlying construct used for environment blocks so that it isn't using Groovy map expressions directly.

          Thomas Bessou added a comment - - edited

          Maybe Groovy should use the fix that was proposed here since it works around the problem instead of keeping that crazy limitation?

          Or dig a little more into why there are only 125 entries allowed in that data structure and maybe use another one ?

          Thomas Bessou added a comment - - edited Maybe Groovy should use the fix that was proposed here since it works around the problem instead of keeping that crazy limitation? Or dig a little more into why there are only 125 entries allowed in that data structure and maybe use another one ?

          Pete Moore added a comment - - edited

          In case there is any problems recreating this it's present in scripted pipelines. This code will produce test.txt to reproduce the error

          bash-4.2$ python << EOF | tee test.txt
          print('node {\n parallel(\n')
          for i in range(200):
            print('   item%s : { sh "echo this is item %s" },' % (i,i))
          print(' )\n}\n')
          EOF

          copy paste the output from that to jenkins window or run load "test.txt". and you get this error

          WorkflowScript: 4: Map expressions can only contain up to 125 entries @ line 4, column 12.
          item0 : { sh "echo this is item 0" },

          Pete Moore added a comment - - edited In case there is any problems recreating this it's present in scripted pipelines. This code will produce test.txt to reproduce the error bash-4.2$ python << EOF | tee test.txt print('node {\n parallel(\n') for i in range(200):   print('   item%s : { sh "echo this is item %s" },' % (i,i)) print(' )\n}\n') EOF copy paste the output from that to jenkins window or run load "test.txt". and you get this error WorkflowScript: 4: Map expressions can only contain up to 125 entries @ line 4, column 12. item0 : { sh "echo this is item 0" },

          Pete Moore added a comment -

          I should add that this is limiting on large projects where you want to make a lot of jobs go parallel. 

          Pete Moore added a comment - I should add that this is limiting on large projects where you want to make a lot of jobs go parallel. 

          Devin Nusbaum added a comment - - edited

          pete312 This limitation only affects map and list literals. In scripted Pipeline, you should be able to adapt your script as a workaround. Instead of creating a parallel step with branches inline like this:

          parallel(branch1: { ... }, branch2: { ... })
          

          Create the branches separately like this:

          Map branches = [:]
          branches['branch1'] = { ... }
          branches['branch2'] = { ... }
          parallel(branches)
          

          Devin Nusbaum added a comment - - edited pete312 This limitation only affects map and list literals. In scripted Pipeline, you should be able to adapt your script as a workaround. Instead of creating a parallel step with branches inline like this: parallel(branch1: { ... }, branch2: { ... }) Create the branches separately like this: Map branches = [:] branches[ 'branch1' ] = { ... } branches[ 'branch2' ] = { ... } parallel(branches)

          Pete Moore added a comment -

          Thank you Devin your workaround is a lot cleaner than my workaround which was this.

          parallel(
            group_0 : { 
              parallel( 
                item0 : { sh "echo this is item 0" },
                item1 : { sh "echo this is item 1" },
                ...
              )
            },
            group_1 : { 
              parallel( 
                item200 : { sh "echo this is item 200" },
                item201 : { sh "echo this is item 201" },
                ...
              )
            },
          )
          
          

          Is the limit in Jenkins or groovy? If the limit is in Jenkins can we look forward to a fix?  New users (Ive talked to) like me are pretty stunned by such a small limit.

          Pete Moore added a comment - Thank you Devin your workaround is a lot cleaner than my workaround which was this. parallel( group_0 : { parallel( item0 : { sh "echo this is item 0" }, item1 : { sh "echo this is item 1" }, ... ) }, group_1 : { parallel( item200 : { sh "echo this is item 200" }, item201 : { sh "echo this is item 201" }, ... ) }, ) Is the limit in Jenkins or groovy? If the limit is in Jenkins can we look forward to a fix?  New users (Ive talked to) like me are pretty stunned by such a small limit.

            Unassigned Unassigned
            ten0 Thomas Bessou
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: