[JENKINS-62127] Dynamic Axis-Values for declarative Pipeline - Jenkins Jira

XML

Word

Printable

Details

Type: New Feature
Status: Open (View Workflow)
Priority: Critical
Resolution: Unresolved
Component/s: matrix-combinations-parameter-plugin, matrix-project-plugin, pipeline, pipeline-model-definition-plugin
Labels:
None
Environment:

Hide
System-Setup:
My System: Jenkins-Version: Jenkins ver. 2.190.1
Declarative-Plugin-Version 1.5.0
Declarative Agent API: 1.1.1
Declarative Extension Points API: 1.5.0
(many other plugins, I think this are the important ones)

The answer of the stackoverflow post say that it is currently not possible, even with the current version of the plugins.

Show
System-Setup: My System: Jenkins-Version: Jenkins ver. 2.190.1 Declarative-Plugin-Version 1.5.0 Declarative Agent API: 1.1.1 Declarative Extension Points API: 1.5.0 (many other plugins, I think this are the important ones) The answer of the stackoverflow post say that it is currently not possible, even with the current version of the plugins.

Similar Issues:

Show

Description

Hi there,

I was asked to open a Ticket here, because it seems like the feature is interesting for many peoples.

Stack Overflow - Declarative Pipeline with dynamic matrix axis values

It would be nice to have the possibility to set the dynamically set the values of the "axis".

Use-Case:
I want to compile multiple plugins with the same Jenkins-pipeline. To do so I wrote a "JenkinsLibrary" with an interface the plugins can use for parameters. One of this parameters is the axis-value for compiler.

e.g. I want to build Plugin "X" for gcc7 on armv8 and Plugin Y for vc15 x64.
So I have a Jenkins-Library with the logic of the build parameters.
And each plugin calls the library with the build-parameters. (e.g. compiler and arch )

Example:
a "JenkinsLibrary" with a file my_library.groovy

def call(Map i_options)
{
    def axis_1_values = "axis_1_value_A"
    // later: def axis_1_values = i_options.get("axis_1_values")
    pipeline
    {
        agent any
        stages { stage("stage A") { matrix {
            axes {
                axis {
                    name "axis_1"
                    values "${axis_1_values}" 
                }
                axis {
                    name "axis_2"
                    values "axis_2_value_A", "axis_2_value_B"
                }
            }
            stages {
                stage("another stage") { steps {
                    echo "hello world from ${axis_1} && ${axis_2}"
                } }
            }
        } } }
    }
}

and each plugin contains a Jenkinsfile like that:

@Library("JenkinsLibrary") _

def options = [
    axis_1_values: "a_axis_value"
]

my_library(options)

We want to keep the Jenkinsfiles of the plugins as short as possible to keep the maintenance as low as possible.

If you run the pipeline above you get the error:

[...] Expected string literal but got "${axis_1_values}"

Complete log:

    19: Expected string literal but got "${axis_1_values}" @ line 19, column 28.
                       values "${axis_1_values}" 
                              ^

1 error

    at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
    at groovy.lang.GroovyClassLoader.recompile(GroovyClassLoader.java:761)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:718)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
    at org.jenkinsci.plugins.workflow.cps.global.UserDefinedGlobalVariable.getValue(UserDefinedGlobalVariable.java:57)
    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:113)
    at sun.reflect.GeneratedMethodAccessor729.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:160)
    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:158)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:162)
    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
    at WorkflowScript.run(WorkflowScript:21)
    at ___cps.transform___(Native Method)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
    at sun.reflect.GeneratedMethodAccessor500.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:39)
    at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
    at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:28)
    at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
    at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:405)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:317)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:281)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

I tried many ways for of setting the variables in the pipeline:

// already tried the following alternatives ...
//    def axis_1_values = 'axis_1_value_A'
//    def axis_1_values = '''axis_1_value_A'''
//    def axis_1_values = ["axis_1_value_A", "axis_2_value_A"]
    pipeline
    {
[...]

[...]
axis {
    name "axis_1"
    // values "axis_1_value_A" // <- of course this works ...
    // but I want to read it from a variable
    values "${axis_1_values}" 
    // I already tried more variants
    // values ${axis_1_values}
    // values axis_1_values
[...]
}

It would be nice if this will be possible.

Attachments

Activity

Ascending order - Click to sort in descending order

tharilya created issue - 2020-04-30 16:06

Liam Newman made changes - 2020-05-01 18:54

Field	Original Value	New Value
Component/s		pipeline-model-definition-plugin [ 21706 ]

Liam Newman made changes - 2020-05-01 18:54

Assignee

Kohsuke Kawaguchi [ kohsuke ]

Liam Newman [ bitwiseman ]

Alexey Shein added a comment - 2020-06-22 21:21

I've got bitten by this as well. In my use case, I just want to parallelize my tests to N downstream jobs, I use the following Jenkinsfile:

// arrayList of stringified indices, like ["0", "1", "2", ...]
def ciIndices(ciNodesTotal) {
  (0..ciNodesTotal-1).toArray().collect { it.toString() }
}

def ciNodesTotal = 3
pipeline {
  agent any

  stages {
    stage('Test') {
      matrix {
        axes {
          axis {
            name 'Worker'
            values(*ciIndices(ciNodesTotal)) // doesn't work, throws Expected string literal but got ${*} @ line 32, column 20.
          } //axis
        } //axes

        stages {
          stage('Worker test') {
            steps {
              echo "Do build - ${Worker}"
            } //steps
          } //stage
        } //stages
      } //matrix
    } //stage
  }
}

If I specify values literally, then everything is working great.

Alexey Shein added a comment - 2020-06-22 21:21 I've got bitten by this as well. In my use case, I just want to parallelize my tests to N downstream jobs, I use the following Jenkinsfile: // arrayList of stringified indices, like [ "0" , "1" , "2" , ...] def ciIndices(ciNodesTotal) { (0..ciNodesTotal-1).toArray().collect { it.toString() } } def ciNodesTotal = 3 pipeline { agent any stages { stage( 'Test' ) { matrix { axes { axis { name 'Worker' values(*ciIndices(ciNodesTotal)) // doesn't work, throws Expected string literal but got ${*} @ line 32, column 20. } //axis } //axes stages { stage( 'Worker test' ) { steps { echo "Do build - ${Worker}" } //steps } //stage } //stages } //matrix } //stage } } If I specify values literally, then everything is working great.

Alexey Shein added a comment - 2020-11-11 21:42

Hey bitwiseman, could you please have a look at this issue? Thanks.

Alexey Shein added a comment - 2020-11-11 21:42 Hey bitwiseman , could you please have a look at this issue? Thanks.

Liam Newman added a comment - 2021-04-22 23:00

conf
I do not have time to work on this right now, but I would be happy to discuss how this might be done if anyone is interested in working on this, but it will not be easy.

Liam Newman added a comment - 2021-04-22 23:00 conf I do not have time to work on this right now, but I would be happy to discuss how this might be done if anyone is interested in working on this, but it will not be easy.

Caleb Werth added a comment - 2021-05-04 22:32

This is a feature I am interested in. What ideas do you have on implementation, bitwiseman?

Caleb Werth added a comment - 2021-05-04 22:32 This is a feature I am interested in. What ideas do you have on implementation, bitwiseman ?

Rahul Somasunderam added a comment - 2021-07-16 20:30

bitwiseman I'm interested in this too. I would be glad to take a stab at this if you can share your thoughts.

Rahul Somasunderam added a comment - 2021-07-16 20:30 bitwiseman I'm interested in this too. I would be glad to take a stab at this if you can share your thoughts.

Maksym added a comment - 2021-09-27 13:06 - edited

I have the same issue as the guys mentioned. bitwiseman could you please share GitHub project we will try to fix this issue?

Maksym added a comment - 2021-09-27 13:06 - edited I have the same issue as the guys mentioned. bitwiseman could you please share GitHub project we will try to fix this issue?

Cristovao Cordeiro added a comment - 2022-07-08 14:35

are there any updates on this? I'm also stuck with a lot of repeated axes due to this...

Cristovao Cordeiro added a comment - 2022-07-08 14:35 are there any updates on this? I'm also stuck with a lot of repeated axes due to this...

Rob McGee added a comment - 2022-09-21 02:30 - edited

This would be very handy for us too. If an experienced jenkins dev could point us in the right direction we might be able to figure this out. We'd like to pull an axis' values from a shared library so we don't have to duplicate it everywhere.

Rob McGee added a comment - 2022-09-21 02:30 - edited This would be very handy for us too. If an experienced jenkins dev could point us in the right direction we might be able to figure this out. We'd like to pull an axis' values from a shared library so we don't have to duplicate it everywhere.

avi shaaya added a comment - 2022-11-17 19:24

This feature request is very important! In my use case i want to run the same pipeline all all the agents with a specific label. It cant be done with declarative pipeline

avi shaaya added a comment - 2022-11-17 19:24 This feature request is very important! In my use case i want to run the same pipeline all all the agents with a specific label. It cant be done with declarative pipeline

tharilya added a comment - 2022-12-07 10:21 - edited

Hi bitwiseman,

I was just looking into this and was suprised that it's still open / not even started.
I am a software-developer for about 10 years now and just took a look into it.

I am little confused, because I interpreted your answer on stackoverflow and here that it's kina tricky and is much effort to get this done.
I assume the the code-lines which needs to be adapted are the overloads of the Axis-parameters which defines the allowed types.
These are defined in the "matrix-plugin" (https://github.com/jenkinsci/matrix-project-plugin) in the "Axis.java":

https://github.com/jenkinsci/matrix-project-plugin/blob/master/src/main/java/hudson/matrix/Axis.java#L79-L89

I've never really got in tough with java, but "isn't it just adding some overloads there" ?

please correct me if I am wrong.

kind regards

EDIT:
It seems linke the "other way around" could be to change the "AxisDirective" in the 'pipeline-model-definition-plugin' ( https://github.com/jenkinsci/pipeline-model-definition-plugin )

https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/master/pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/generator/AxisDirective.java

tharilya added a comment - 2022-12-07 10:21 - edited Hi bitwiseman , I was just looking into this and was suprised that it's still open / not even started. I am a software-developer for about 10 years now and just took a look into it. I am little confused, because I interpreted your answer on stackoverflow and here that it's kina tricky and is much effort to get this done. I assume the the code-lines which needs to be adapted are the overloads of the Axis-parameters which defines the allowed types. These are defined in the "matrix-plugin" ( https://github.com/jenkinsci/matrix-project-plugin ) in the "Axis.java": https://github.com/jenkinsci/matrix-project-plugin/blob/master/src/main/java/hudson/matrix/Axis.java#L79-L89 I've never really got in tough with java, but "isn't it just adding some overloads there" ? please correct me if I am wrong. kind regards EDIT: It seems linke the "other way around" could be to change the "AxisDirective" in the 'pipeline-model-definition-plugin' ( https://github.com/jenkinsci/pipeline-model-definition-plugin ) https://github.com/jenkinsci/pipeline-model-definition-plugin/blob/master/pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/generator/AxisDirective.java

avi shaaya added a comment - 2022-12-07 12:21

Does anyone has a workaround/ hack ? (I dont want to use scripted syntax to solve this)

avi shaaya added a comment - 2022-12-07 12:21 Does anyone has a workaround/ hack ? (I dont want to use scripted syntax to solve this)

Timo Sand added a comment - 2023-04-13 18:37

aviss7 This seems like an okay workaround https://www.jenkins.io/blog/2019/11/22/welcome-to-the-matrix/#controlling-cell-behavior-at-runtime

Timo Sand added a comment - 2023-04-13 18:37 aviss7 This seems like an okay workaround https://www.jenkins.io/blog/2019/11/22/welcome-to-the-matrix/#controlling-cell-behavior-at-runtime

avi shaaya added a comment - 2023-04-14 08:25 - edited

Thanks Timo,

Its some sort of a workaround that can work for some users but still we have these limitations:

axis values are hard coded (no way to change them from parameter only from code)
This workaround allow to select only one item from the axis (no solution for running 2 items or more)

avi shaaya added a comment - 2023-04-14 08:25 - edited Thanks Timo, Its some sort of a workaround that can work for some users but still we have these limitations: axis values are hard coded (no way to change them from parameter only from code) This workaround allow to select only one item from the axis (no solution for running 2 items or more)

avi shaaya made changes - 2023-05-16 10:20

Priority

Minor [ 4 ]

Critical [ 2 ]

People

Assignee:: Liam Newman

Reporter:: tharilya

Votes:: 43 Vote for this issue

Watchers:: 43 Start watching this issue

Dates

Created:: 2020-04-30 16:06

Updated:: 2023-05-16 10:20