Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56500

Declarative pipeline restricted in code size

    XMLWordPrintable

Details

    Description

      There is partial fix for this in pipeline-model-definition-plugin v1.4.0 and later, significantly improved in v1.8.2.  Due to the extent to which it change how pipelines are executed it is turned off by default.  It can be turned on by setting a JVM property (either on the command-line or in Jenkins script console):

      org.jenkinsci.plugins.pipeline.modeldefinition.parser.RuntimeASTTransformer.SCRIPT_SPLITTING_TRANSFORMATION=true 

      As noted, this still works best with a Jenkinsfile with pipeline directive as the only root item in the file.
      Since v1.8.2 this workaround reports an informative error for pipelines using `def` variables before the pipeline directive. Add a @Field annotation to those declaration.
      This workaround generally does NOT work if the pipeline directive inside a shared library method. If this is a scenario you want, please come join the pipeline authoring SIG and we can discuss.

      Please give it a try and provide feedback.  

      When working with a declarative pipeline script, we run into an error "Method Code too large".

      This seems to happen when the code between 'pipeline{}' is more than a specific size. 

       

      I'm creating this issue for this component following the suggestion of Jesse Glick at the issue 37984

      I've attached a declarative pipeline script that reproduces the issue. 

       

       

      Attachments

        Issue Links

          Activity

            It doesn't pass params into pipeline with turned on flag when pipeline is inside shared library. Is there any solution for this case?

            oleksazhel Oleksandr Zhelnytskyi added a comment - It doesn't pass params into pipeline with turned on flag when pipeline is inside shared library. Is there any solution for this case?
            bitwiseman Liam Newman added a comment -

            oleksazhel
            Interesting. This feature is still experimental, but I'm surprised that would be an issue. Do you mean Jenkins job parameters or do you mean method parameters into the shared library method? Perhaps you could open a new issue and link it to this one?

            bitwiseman Liam Newman added a comment - oleksazhel Interesting. This feature is still experimental, but I'm surprised that would be an issue. Do you mean Jenkins job parameters or do you mean method parameters into the shared library method? Perhaps you could open a new issue and link it to this one?

            bitwiseman Yes, the issue was with passing parameters from method into method with pipeline inside. But I made workaround by code refactoring from fully declarative to hybrid where declarative steps call scripted pipeline methods.

            oleksazhel Oleksandr Zhelnytskyi added a comment - bitwiseman  Yes, the issue was with passing parameters from method into method with pipeline inside. But I made workaround by code refactoring from fully declarative to hybrid where declarative steps call scripted pipeline methods.
            marioja Mario Jauvin added a comment -

            Good day.  bitwiseman I would like to start by thanking you for the work on this problem.  I have a couple questions:

            1. Is this problem related or the same one whereby due to Jenkins use of CPS the bytecode size for a method is limited to 64K?
            2. Can you foresee this problem to ever be completely fixed?  I would be willing to help if you think that is possible.

            Thanks al lot

            marioja Mario Jauvin added a comment - Good day.  bitwiseman I would like to start by thanking you for the work on this problem.  I have a couple questions: Is this problem related or the same one whereby due to Jenkins use of CPS the bytecode size for a method is limited to 64K? Can you foresee this problem to ever be completely fixed?  I would be willing to help if you think that is possible. Thanks al lot
            reddwarf94 Cristian added a comment -

             

            I get the problem with this (simplified) pipeline

            import groovy.transform.Field
            
            def docker_arguments(boolean with_source_package) {
              if(with_source_package) {
                return "--network none -v ${env.WORKSPACE}/src:${env.WORKSPACE}/src:ro,z"
              } else {
                return "--network none -v ${env.WORKSPACE}/src:${env.WORKSPACE}/src:ro,z"
              }
            }
            
            @Field
            def base_image_version = "11"
            
            @Field
            def toolchain = [
              a: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: true],
              b: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              c: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              d: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: true, other_var: false],
              e: [version: "8", cpack_gen: "RPM", source_package: false, other_var: false],
              f: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              g: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              h: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              i: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              j: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: true, other_var: false],
              k: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: true, other_var: false],
              l: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              m: [version: "${base_image_version}.2", cpack_gen: "TGZ", source_package: false, other_var: false],
              n: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              o: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: true],
              p: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              q: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              r: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              s: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: true],
              t: [version: "${base_image_version}.2", cpack_gen: "TGZ", source_package: false, other_var: false],
              u: [version: "${base_image_version}.2", cpack_gen: "TGZ", source_package: false, other_var: false],
              v: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: true],
              w: [version: "1", cpack_gen: "DEB", source_package: false, other_var: false],
              x: [version: "6", cpack_gen: "DEB", source_package: false, other_var: false],
              y: [version: "1", cpack_gen: "DEB", source_package: false, other_var: false],
              z: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: true],
              aa: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: true],
              ab: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: true],
              ac: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: true],
              ad: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              1: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              2: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              3: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              4: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              5: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              6: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              7: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              8: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              9: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              10: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              11: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              12: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              13: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              14: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false],
              15: [version: "${base_image_version}.1", cpack_gen: "TGZ", source_package: false, other_var: false]
            ]
            @Field
            def toolchain_list = toolchain.keySet() as ArrayList
            @Field
            def toolchain_list_string = toolchain_list.join(',')
            
            pipeline {
              agent {
                label 'toolchain_docker'
              }
            
              parameters {
                extendedChoice(
                  name: 'PLATFORM_LIST',
                  description: 'Platforms to build for',
                  type: 'PT_CHECKBOX',
                  value: toolchain_list_string,
                  defaultValue: toolchain_list_string,
                  visibleItemCount: 15
                )
              }
            
              stages {
                stage('Build') {
                  matrix {
                    agent {
                      dockerfile {
                        dir 'jenkins/docker'
                        additionalBuildArgs "--build-arg BASE_IMAGE=toolchain-${PLATFORM}:${toolchain["${PLATFORM}"].version} --build-arg USER_ID=\$(id -u) --build-arg GROUP_ID=\$(id -g)"
                        reuseNode true
                        args docker_arguments(toolchain["${PLATFORM}"].source_package)
                      }
                    }
            
                    axes {
                      axis {
                        name 'PLATFORM'
                        values 'a',
                               'b',
                               'c',
                               'd',
                               'e',
                               'f',
                               'g',
                               'h',
                               'i',
                               'j',
                               'k',
                               'l',
                               'm',
                               'n',
                               'o',
                               'p',
                               'q',
                               'r',
                               's',
                               't',
                               'u',
                               'v',
                               'w',
                               'x',
                               'y',
                               'z',
                               'aa',
                               'ab',
                               'ac',
                               'ad',
                               '1',
                               '2',
                               '3',
                               '4',
                               '5',
                               '6',
                               '7',
                               '8',
                               '9',
                               '10',
                               '11',
                               '12',
                               '13',
                               '14',
                               '15'
                      }
                    }
            
                    when {
                      beforeAgent true
                      expression {
                        def platform_list = params.PLATFORM_LIST.split(',')
                        return platform_list.contains("${PLATFORM}")
                      }
                    }
            
                    stages {
                      stage('Build project') {
                        steps {
                          sh label:  'Configure',
                             script: 'a' + "${PLATFORM} ${PLATFORM}"
                          sh label:  'Build',
                             script: "${PLATFORM}"
                        }
                      }
                    }
                  }
                }
              }
            } 

             

            I would say a big part of the problem is that you only get that "Method too large" message. You don't know if it's just over 64 KiB or if it's 1 MiB, you don't know what parts of your pipeline are contributing to the size.

            A few things I found more or less surprising when simplifying this:

            • The code inside the docker_arguments() function does matter. A suggestion when people find this issue is to put stuff into functions, so I thought that since it's an independent method it would not matter what's inside... but it did. Maybe it's because it's being called from inside agent/dockerfile?
            • "${PLATFORM} ${PLATFORM}" vs "${PLATFORM}" in a step made a difference
            • 'a' + 'b' vs 'ab' made a difference
            • The number of elements in the PLATFORM axis makes a difference. I thought it would just be a "constant" called N times in parallel.

             

            In the real pipeline, there are 30 platforms, not 45. But we will sooner or later reach 45, and in any case we already have problems with 30 since the real pipeline is not just this matrix.

             

            All this is with SCRIPT_SPLITTING_TRANSFORMATION enabled. Which made things like

            def docker_arguments(boolean with_source_package) {
              if(with_source_package) {
                return "--network none -v ${env.WORKSPACE}/src:${env.WORKSPACE}/src:ro,z"
              } else {
                return "--network none -v ${env.WORKSPACE}/src:${env.WORKSPACE}/src:ro,z"
              }
            }
            
            pipeline {
              agent {
                label 'toolchain_docker'
              }
            
              stages {
                stage('Test') {
                  matrix {
                    agent {
                      dockerfile {
                        dir 'router_agent/jenkins/docker'
                        additionalBuildArgs "--build-arg BASE_IMAGE=docker.samknows.com/toolchain-base:11 --build-arg USER_ID=\$(id -u) --build-arg GROUP_ID=\$(id -g)"
                        reuseNode true
                        args docker_arguments(false)
                      }
                    }
            
                    axes {
                      axis {
                        name 'PLATFORM'
                        values '1','2','3','4','5','6','7','8','9','10','11','12','13','14','15','16','17','18','19','20','21','22','23','24','25','26','27','28','29','30'
                      }
                      axis {
                        name 'BROWSER'
                        values '1','2','3','4','5','6','7','8'
                      }
                    }
            
                    when {
                      beforeAgent true
                      expression {
                        return true
                      }
                    }
            
                    stages {
                      stage('Stage') {
                        steps {
                          echo "${PLATFORM} - ${BROWSER}"
                        }
                      }
                    }
                  }
                }
              }
            }
             

            work, when before they didn't.

             

            reddwarf94 Cristian added a comment -   I get the problem with this (simplified) pipeline import groovy.transform.Field def docker_arguments( boolean with_source_package) {   if (with_source_package) {     return "--network none -v ${env.WORKSPACE}/src:${env.WORKSPACE}/src:ro,z"   } else {     return "--network none -v ${env.WORKSPACE}/src:${env.WORKSPACE}/src:ro,z"   } } @Field def base_image_version = "11" @Field def toolchain = [   a: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: true ],   b: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   c: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   d: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: true , other_var: false ],   e: [version: "8" , cpack_gen: "RPM" , source_package: false , other_var: false ],   f: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   g: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   h: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   i: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   j: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: true , other_var: false ],   k: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: true , other_var: false ],   l: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   m: [version: "${base_image_version}.2" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   n: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   o: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: true ],   p: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   q: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   r: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   s: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: true ],   t: [version: "${base_image_version}.2" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   u: [version: "${base_image_version}.2" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   v: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: true ],   w: [version: "1" , cpack_gen: "DEB" , source_package: false , other_var: false ],   x: [version: "6" , cpack_gen: "DEB" , source_package: false , other_var: false ],   y: [version: "1" , cpack_gen: "DEB" , source_package: false , other_var: false ],   z: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: true ],   aa: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: true ],   ab: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: true ],   ac: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: true ],   ad: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   1: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   2: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   3: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   4: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   5: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   6: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   7: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   8: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   9: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   10: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   11: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   12: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   13: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   14: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ],   15: [version: "${base_image_version}.1" , cpack_gen: "TGZ" , source_package: false , other_var: false ] ] @Field def toolchain_list = toolchain.keySet() as ArrayList @Field def toolchain_list_string = toolchain_list.join( ',' ) pipeline {   agent {     label 'toolchain_docker'   } parameters {     extendedChoice(       name: 'PLATFORM_LIST' ,       description: 'Platforms to build for ' ,       type: 'PT_CHECKBOX' ,       value: toolchain_list_string,       defaultValue: toolchain_list_string,       visibleItemCount: 15     )   } stages {     stage( 'Build' ) {       matrix {         agent {           dockerfile {             dir 'jenkins/docker'             additionalBuildArgs "--build-arg BASE_IMAGE=toolchain-${PLATFORM}:${toolchain[" ${PLATFORM} "].version} --build-arg USER_ID=\$(id -u) --build-arg GROUP_ID=\$(id -g)"             reuseNode true             args docker_arguments(toolchain[ "${PLATFORM}" ].source_package)           }         }         axes {           axis {             name 'PLATFORM'             values 'a' ,                     'b' ,                     'c' ,                     'd' ,                     'e' ,                     'f' ,                     'g' ,                     'h' ,                     'i' ,                     'j' ,                     'k' ,                     'l' ,                     'm' ,                     'n' ,                     'o' ,                     'p' ,                     'q' ,                     'r' ,                     's' ,                     't' ,                     'u' ,                     'v' ,                     'w' ,                     'x' ,                     'y' ,                     'z' ,                     'aa' ,                     'ab' ,                     'ac' ,                     'ad' ,                     '1' ,                     '2' ,                     '3' ,                     '4' ,                     '5' ,                     '6' ,                     '7' ,                     '8' ,                     '9' ,                     '10' ,                     '11' ,                     '12' ,                     '13' ,                     '14' ,                     '15'           }         }         when {           beforeAgent true           expression {             def platform_list = params.PLATFORM_LIST.split( ',' )             return platform_list.contains( "${PLATFORM}" )           }         }         stages {           stage( 'Build project' ) {             steps {               sh label:   'Configure' ,                  script: 'a' + "${PLATFORM} ${PLATFORM}"               sh label:   'Build' ,                  script: "${PLATFORM}"             }           }         }       }     }   } }   I would say a big part of the problem is that you only get that "Method too large" message. You don't know if it's just over 64 KiB or if it's 1 MiB, you don't know what parts of your pipeline are contributing to the size. A few things I found more or less surprising when simplifying this: The code inside the docker_arguments() function does matter. A suggestion when people find this issue is to put stuff into functions, so I thought that since it's an independent method it would not matter what's inside... but it did. Maybe it's because it's being called from inside agent/dockerfile? "${PLATFORM} ${PLATFORM}" vs "${PLATFORM}" in a step made a difference 'a' + 'b' vs 'ab' made a difference The number of elements in the PLATFORM axis makes a difference. I thought it would just be a "constant" called N times in parallel.   In the real pipeline, there are 30 platforms, not 45. But we will sooner or later reach 45, and in any case we already have problems with 30 since the real pipeline is not just this matrix.   All this is with SCRIPT_SPLITTING_TRANSFORMATION enabled. Which made things like def docker_arguments( boolean with_source_package) {   if (with_source_package) {     return "--network none -v ${env.WORKSPACE}/src:${env.WORKSPACE}/src:ro,z"   } else {     return "--network none -v ${env.WORKSPACE}/src:${env.WORKSPACE}/src:ro,z"   } } pipeline {   agent {     label 'toolchain_docker'   }   stages {     stage( 'Test' ) {       matrix {         agent {           dockerfile {             dir 'router_agent/jenkins/docker'             additionalBuildArgs "--build-arg BASE_IMAGE=docker.samknows.com/toolchain-base:11 --build-arg USER_ID=\$(id -u) --build-arg GROUP_ID=\$(id -g)"             reuseNode true             args docker_arguments( false )           }         }       axes {           axis {             name 'PLATFORM'             values '1' , '2' , '3' , '4' , '5' , '6' , '7' , '8' , '9' , '10' , '11' , '12' , '13' , '14' , '15' , '16' , '17' , '18' , '19' , '20' , '21' , '22' , '23' , '24' , '25' , '26' , '27' , '28' , '29' , '30'           }           axis {             name 'BROWSER'             values '1' , '2' , '3' , '4' , '5' , '6' , '7' , '8'           }         }         when {           beforeAgent true           expression {             return true           }         }         stages {           stage( 'Stage' ) {             steps {               echo "${PLATFORM} - ${BROWSER}"             }           }         }       }     }   } } work, when before they didn't.  

            People

              bitwiseman Liam Newman
              wim Wim Gaethofs
              Votes:
              11 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated: