Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42753

Generate runtime model directly from AST model

    • Declarative - 1.2

      My apologies if this isn't a bug, but I've searched high and low and tried a number of things, but I can't seem to find the syntax ex. to solve what's broken.  After upgrading pipeline plugins to the latest releases (as of today), my declarative pipelines broke because it seems there might no longer be ability to use a property in the agent label.  This was working:

      pipeline{
      
          parameters {
              choice(choices: 'server-cd\server-demo', description: DeployEnvironmentHelp, name: 'Environment')
          } 
      
          environment {
              DEPLOYENVIRONMENT = "${params.Environment}"
          }
      
          agent none
      
          stages {
              stage ("Stage 1") {
                  agent {label DEPLOYENVIRONMENT}
                  echo 'Get Deployment Helper Files from SCM repo(s)...'
              }
          } 
      }

      But the pipeline fails saying that the DEPLOYENVIRONMENT is not a proper value (I no longer have the exact message )  

      As of model def. 1.1.1, is it no longer possible to use an environment variable or property in the agent label?   btw - I'm setting DEPLOYENVIRONMENT so I can do other tests and not have to keep using params.Environment, but in other pipelines I set properties so that I can define which agent based on properties that change in the pipeline.

       

       

          [JENKINS-42753] Generate runtime model directly from AST model

          Andrew Bayer added a comment -

          Grr, actually, I'm having trouble with that last example in my local tests, but I think that may just be that Declarative's dependencies aren't new enough - not a problem in the real world, just in my tests.

          Andrew Bayer added a comment - Grr, actually, I'm having trouble with that last example in my local tests, but I think that may just be that Declarative's dependencies aren't new enough - not a problem in the real world, just in my tests.

          Jason Davis added a comment -

          Bummer... OK - before reaching out with this issue, I did try both those recommendations but ran into troubles before I had to roll back the plugin updates.  I'm not at a place yet where I can re-set up some tests and report the results, but will asap.  Hopefully you hit on solution. 

          Jason Davis added a comment - Bummer... OK - before reaching out with this issue, I did try both those recommendations but ran into troubles before I had to roll back the plugin updates.  I'm not at a place yet where I can re-set up some tests and report the results, but will asap.  Hopefully you hit on solution. 

          Andrew Bayer added a comment -

          Repurposing this ticket to solve this whole category of problems, not just this one particular one.

          Andrew Bayer added a comment - Repurposing this ticket to solve this whole category of problems, not just this one particular one.

          Andrew Bayer added a comment -

          Andrew Bayer added a comment - Work-in-progress PR up at  https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/140

          Andrew Bayer added a comment -

          Specifically gonna incorporate material from JENKINS-42777 into tests here to make as sure as humanly possible that we don't cause regressions with this.

          Andrew Bayer added a comment - Specifically gonna incorporate material from JENKINS-42777 into tests here to make as sure as humanly possible that we don't cause regressions with this.

          Jeff G added a comment -

          Adding to the list of users who are blocked from upgrading this pipeline plugin since March.

          Jeff G added a comment - Adding to the list of users who are blocked from upgrading this pipeline plugin since March.

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pipeline-model-definition/pom.xml
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/AbstractBuildConditionResponder.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Agent.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Environment.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Libraries.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/MethodsToList.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Options.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Parameters.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PostBuild.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PostStage.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PropertiesToMap.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PropertyOptionContainer.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Root.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StageConditionals.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stages.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StepBlockWithOtherArgs.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StepsBlock.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Tools.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Triggers.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ASTParserUtils.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/BlockStatementMatch.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/Converter.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ModelStepLoader.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/SyntheticStageGraphListener.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/GroovyShellDecoratorImpl.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/AllOfConditional.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/AnyOfConditional.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/BranchConditional.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/EnvironmentConditional.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/ExpressionConditional.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/NotConditional.java
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ClosureModelTranslator.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/LibrariesTranslator.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/MethodsToListTranslator.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTranslator.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/PropertiesToMapTranslator.groovy
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/ExpressionConditionalScript.groovy
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTest.java
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ParametersTest.java
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ToolsTest.java
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/TriggersTest.java
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/steps/CredentialWrapperStepTest.java
          pipeline-model-definition/src/test/resources/credentialsUsedInWhenEnv.groovy
          pipeline-model-definition/src/test/resources/credentialsUsedInWhenExpression.groovy
          pipeline-model-definition/src/test/resources/envVarInOptions.groovy
          pipeline-model-definition/src/test/resources/envVarInParameters.groovy
          pipeline-model-definition/src/test/resources/envVarInTools.groovy
          pipeline-model-definition/src/test/resources/envVarInTriggers.groovy
          pipeline-model-definition/src/test/resources/envVarInWrapper.groovy
          pipeline-model-extensions/pom.xml
          pipeline-model-extensions/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/DeclarativeStageConditionalDescriptor.java
          pom.xml
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/963d58beea949d4d8eb468986b0908b075205b9c
          Log:
          [FIXED JENKINS-42753] Use AST transformation to generate runtime model

          There've been any number of problems in the past due to the closure
          translation approach for going from `pipeline

          { ... }

          ` to the `Root`
          object we need to actually run. I won't bother going into them here,
          just...yeah. There's lots. Until this change goes in, there are 4
          special CPS-transformed classes for the translation, plus truly
          bizarre special-casing logic for handling environment and when
          expressions via looping circuits of script.evaluate calls. I'm
          serious. It's sad.

          This replaces all that by instead generating Root and its children via
          an AST transformation at parsing/validation time. It gets us away from
          ever having to worry about translating or evaluating within the CPS
          code, using lazily evaluated closures to support things like declaring
          cross-referencing environment variables out of order, or environment
          variables depending on credentials variables, etc...

          Anyway, it works. It's more maintainable. It's cleaner. It's actually
          very slightly faster (since we've eliminated the closure translation
          stuff from CPS execution, and got rid of a now-redundant additional
          parse/validate run to get the execution model). And it's completely
          compatible syntax-wise (hopefully! It's not impossible that I missed
          some weird expressions in environment variable values, but we'll cross
          that bridge if we get to it).

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-definition/pom.xml pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/AbstractBuildConditionResponder.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Agent.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Environment.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Libraries.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/MethodsToList.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Options.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Parameters.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PostBuild.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PostStage.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PropertiesToMap.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/PropertyOptionContainer.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Root.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stage.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StageConditionals.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Stages.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StepBlockWithOtherArgs.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/StepsBlock.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Tools.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Triggers.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ASTParserUtils.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/BlockStatementMatch.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/Converter.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ModelStepLoader.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/SyntheticStageGraphListener.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/GroovyShellDecoratorImpl.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/AllOfConditional.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/AnyOfConditional.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/BranchConditional.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/EnvironmentConditional.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/ExpressionConditional.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/NotConditional.java pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ClosureModelTranslator.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/LibrariesTranslator.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/MethodsToListTranslator.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTranslator.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/PropertiesToMapTranslator.groovy pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/when/impl/ExpressionConditionalScript.groovy pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/OptionsTest.java pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ParametersTest.java pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/ToolsTest.java pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/TriggersTest.java pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/steps/CredentialWrapperStepTest.java pipeline-model-definition/src/test/resources/credentialsUsedInWhenEnv.groovy pipeline-model-definition/src/test/resources/credentialsUsedInWhenExpression.groovy pipeline-model-definition/src/test/resources/envVarInOptions.groovy pipeline-model-definition/src/test/resources/envVarInParameters.groovy pipeline-model-definition/src/test/resources/envVarInTools.groovy pipeline-model-definition/src/test/resources/envVarInTriggers.groovy pipeline-model-definition/src/test/resources/envVarInWrapper.groovy pipeline-model-extensions/pom.xml pipeline-model-extensions/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/when/DeclarativeStageConditionalDescriptor.java pom.xml http://jenkins-ci.org/commit/pipeline-model-definition-plugin/963d58beea949d4d8eb468986b0908b075205b9c Log: [FIXED JENKINS-42753] Use AST transformation to generate runtime model There've been any number of problems in the past due to the closure translation approach for going from `pipeline { ... } ` to the `Root` object we need to actually run. I won't bother going into them here, just...yeah. There's lots. Until this change goes in, there are 4 special CPS-transformed classes for the translation, plus truly bizarre special-casing logic for handling environment and when expressions via looping circuits of script.evaluate calls. I'm serious. It's sad. This replaces all that by instead generating Root and its children via an AST transformation at parsing/validation time. It gets us away from ever having to worry about translating or evaluating within the CPS code, using lazily evaluated closures to support things like declaring cross-referencing environment variables out of order, or environment variables depending on credentials variables, etc... Anyway, it works. It's more maintainable. It's cleaner. It's actually very slightly faster (since we've eliminated the closure translation stuff from CPS execution, and got rid of a now-redundant additional parse/validate run to get the execution model). And it's completely compatible syntax-wise (hopefully! It's not impossible that I missed some weird expressions in environment variable values, but we'll cross that bridge if we get to it).

          Jeff G added a comment -

          Is there a way to get an expected release date for Declarative 1.2?

          Jeff G added a comment - Is there a way to get an expected release date for Declarative 1.2?

          Andrew Bayer added a comment -

          Hi jgrant216 - it was gonna be released today, but we've decided to get https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/193 done and included, so I would expect the release next week.

          Andrew Bayer added a comment - Hi jgrant216 - it was gonna be released today, but we've decided to get https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/193 done and included, so I would expect the release next week.

          Liam Newman added a comment -

          Bulk closing resolved issues.

          Liam Newman added a comment - Bulk closing resolved issues.

            abayer Andrew Bayer
            jedavis Jason Davis
            Votes:
            5 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: