Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-70557

Setting readYaml max aliases for collections system properties via Java arguments does not work

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • None

      Expected behaviour

      If I start up the controller, and use the 'readYaml' step with a yaml containing 51 aliases/anchors:

      pipeline {
          agent any
          stages {
              stage('Hello') {
                  steps {
                      readYaml file: 'moreThan50anchors.yml'
                  }
              }
          }
      } 

       

      I get the expected error (due to SnakeYAML's default configuration of 50):

      org.yaml.snakeyaml.error.YAMLException: Number of aliases for non-scalar nodes exceeds the specified max=50
          at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:193)
      ...
          at org.yaml.snakeyaml.Yaml$1.next(Yaml.java:499)
          at org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep$Execution.doRun(ReadYamlStep.java:203)
      ...

      Thanks to the fix from https://issues.jenkins.io/browse/JENKINS-68830 I can then go to `Manage Jenkins` -> `Script console` and run:

      org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.DEFAULT_MAX_ALIASES_FOR_COLLECTIONS=51
      org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.MAX_MAX_ALIASES_FOR_COLLECTIONS=51 

      And run the same pipeline again, it is successful.

      Ideally, I should not be parsing such a large yaml file using the readYaml step, This command loads the data into the controller's memory, and uses resources on the controller, which at scale (if the yaml is very large, or it runs very often from multiple pipelines, for example using a Pipeline shared library) can cause performance issues.

      I should change my code to use `sh 'yq ,,,'` to use a shell step on an agent to run https://github.com/mikefarah/yq (as also recommended in the plugin documentation https://github.com/jenkinsci/pipeline-utility-steps-plugin/blob/5953af571306aed1ce61df12c6a86cd2f7a1d329/src/main/resources/org/jenkinsci/plugins/pipeline/utility/steps/conf/ReadYamlStep/help.groovy#L68-L77 ).

      Unexpected behaviour

      Instead of using the script console, if I add these as startup arguments:

      java -Dorg.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.DEFAULT_MAX_ALIASES_FOR_COLLECTIONS=51 -Dorg.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.MAX_MAX_ALIASES_FOR_COLLECTIONS=51 -jar jenkins.war 

      Then I run my same pipeline (without running anything in the script console first), I get the error in the Pipeline log:

      java.lang.IllegalArgumentException: 51 > 0. Reduce the required DEFAULT_MAX_ALIASES_FOR_COLLECTIONS or convince your administrator to increase the max allowed value with the system property "null"
          at org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.setDefaultMaxAliasesForCollections(ReadYamlStep.java:105)
          at org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep.<clinit>(ReadYamlStep.java:68)
          at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
          at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:330)
          at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:272)
          at org.jenkinsci.plugins.workflow.steps.StepDescriptor.newInstance(StepDescriptor.java:204)
          at org.jenkinsci.plugins.pipeline.utility.steps.AbstractFileOrTextStepDescriptorImpl.newInstance(AbstractFileOrTextStepDescriptorImpl.java:24)
          at org.jenkinsci.plugins.pipeline.utility.steps.AbstractFileOrTextStepDescriptorImpl.newInstance(AbstractFileOrTextStepDescriptorImpl.java:12)
          at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:302)
          at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
          at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.base/java.lang.reflect.Method.invoke(Method.java:566)
          at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
          at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
          at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1225)
          at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
          at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
          at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
          at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
          at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:180)
          at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
          at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:163)
          at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:178)
          at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:182)
          at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:152)
          at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
      Caused: java.lang.ExceptionInInitializerError
          at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
          at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:330)
          at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:272)
          at org.jenkinsci.plugins.workflow.steps.StepDescriptor.newInstance(StepDescriptor.java:204)
          at org.jenkinsci.plugins.pipeline.utility.steps.AbstractFileOrTextStepDescriptorImpl.newInstance(AbstractFileOrTextStepDescriptorImpl.java:24)
          at org.jenkinsci.plugins.pipeline.utility.steps.AbstractFileOrTextStepDescriptorImpl.newInstance(AbstractFileOrTextStepDescriptorImpl.java:12)
          at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:302)
      ...

      This exception is coming from:
      https://github.com/jenkinsci/pipeline-utility-steps-plugin/blob/0326b3cb497dee95348471fca1c7c62dbb6975fb/src/main/java/org/jenkinsci/plugins/pipeline/utility/steps/conf/ReadYamlStep.java#L105

       

      Every subsequent pipeline run after that one encounters the error:

      java.lang.NoClassDefFoundError: Could not initialize class org.jenkinsci.plugins.pipeline.utility.steps.conf.ReadYamlStep
          at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
          at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:330)
          at org.jenkinsci.plugins.structs.describable.DescribableModel.instantiate(DescribableModel.java:272)
          at org.jenkinsci.plugins.workflow.steps.StepDescriptor.newInstance(StepDescriptor.java:204)
          at org.jenkinsci.plugins.pipeline.utility.steps.AbstractFileOrTextStepDescriptorImpl.newInstance(AbstractFileOrTextStepDescriptorImpl.java:24)
          at org.jenkinsci.plugins.pipeline.utility.steps.AbstractFileOrTextStepDescriptorImpl.newInstance(AbstractFileOrTextStepDescriptorImpl.java:12)

            rkivisto Ray Kivisto
            rkivisto Ray Kivisto
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: