Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41335

Allow variables and functions to be defined within pipeline to be used in any stage

      I have several use cases where I may operate on a list or map on different stages of a pipeline. This variable may not be known ahead of time - it could be defined within a stage. I'd like a syntax that allows me to define variables within a pipeline.

      Today, I have to define a shared variable before entering pipeline {}.

      Jenkinsfile
      #!/bin/groovy
      // Define outside of pipeline to make sure accessible in all script {} blocks
      def my_list
      
      pipeline {
          agent { label 'label' }
          stages {
              stage('stage1') {
                steps {
                 script {
                     my_list = [1,2,3]
                 }
                }
              }
          
              stage('stage2') {
                steps {
                  script {
                      for(int i=0; i<my_list.size();i++) {
                          echo "Doing something with ${my_list[i]}"
                      }
                  }
                }
              }
          }
      }
      

      A simple approach would be to allow "def" to work anywhere within pipeline {}. This avoids any overly new syntax for pipeline.

      Another option would allow the user to provide a special pipeline {} block for defining non-env/params style variables. This could be helpful in allowing the user to use items available in script {} for defining shared state.

      Jenkinsfile with possible example syntax
      #!/bin/groovy
      
      
      pipeline {
          agent { label 'label' }    
      
          define {
             def my_map = [:] //empty map
             def my_list //undefined shared variable
           }
      
          stages {
              stage('stage1') {
                steps {
                  script {
                    //If def can be anywhere in pipeline {}, drop script {}
                    my_list = [1,2,3]
                  }
                }
              }
          
              stage('stage2') {
                steps {
                  script {
                      for(int i=0; i<my_list.size();i++) {
                          echo "Doing something with ${my_list[i]}"
                      }
                  }
                }
              }
          }
      }
      

          [JENKINS-41335] Allow variables and functions to be defined within pipeline to be used in any stage

          Edgars Batna added a comment - - edited

          Language features already seep in everywhere (e.g. string manipulation), I don't see how defining local variables and functions does any harm.

          It would then have the potential of replacing any other build/whatever scripts once and for all. Frankly, I'm sick of dealing with unavoidable tools and scripts galore in any Jenkins project (not by the fault of Jenkins, ofc).

          Edgars Batna added a comment - - edited Language features already seep in everywhere (e.g. string manipulation), I don't see how defining local variables and functions does any harm. It would then have the potential of replacing any other build/whatever scripts once and for all. Frankly, I'm sick of dealing with unavoidable tools and scripts galore in any Jenkins project (not by the fault of Jenkins, ofc).

          Andrew Bayer added a comment -

          So, sorry to say this, but this isn't going to happen. It's not the direction we want to go with Declarative.

          Andrew Bayer added a comment - So, sorry to say this, but this isn't going to happen. It's not the direction we want to go with Declarative.

          Liam Newman added a comment - - edited

          abayer Could you point to the direction we do want to go?  Is there public document(s) somewhere talking about this?   

           Do we need JEPs to talk about these design choices? 

          Liam Newman added a comment - - edited abayer Could you point to the direction we do want to go?  Is there public document(s) somewhere talking about this?     Do we need JEPs to talk about these design choices? 

          Andrew Bayer added a comment -

          bitwiseman - in this case, it's more that it was never something that fit into Declarative's model, and I've decided that it's time to just say yeah, it ain't happening, and move on.

          Andrew Bayer added a comment - bitwiseman - in this case, it's more that it was never something that fit into Declarative's model, and I've decided that it's time to just say yeah, it ain't happening, and move on.

          Sam Van Oort added a comment -

          bitwiseman FWIW I consider this the bright red line that says "no you need to be using Scripted for this." This feature would convert Declarative into a more full programming language as opposed to a simple, declarative Pipeline description.

          Sam Van Oort added a comment - bitwiseman FWIW I consider this the bright red line that says "no you need to be using Scripted for this." This feature would convert Declarative into a more full programming language as opposed to a simple, declarative Pipeline description.

          svanoort I thought I had read somehwhere that the roadmap was for scripted to eventually be deprecated, and that declarative would replace it.

          In any case, I second Liam's point that it would be interesting to see documents explaining this vision. It would help, at the very least,  people to make long term decisions for their CI architecture around Jenkins.

          Francis Therien added a comment - svanoort I thought I had read somehwhere that the roadmap was for scripted to eventually be deprecated, and that declarative would replace it. In any case, I second Liam's point that it would be interesting to see documents explaining this vision. It would help, at the very least,  people to make long term decisions for their CI architecture around Jenkins.

          Sam Van Oort added a comment -

          ftherien Caveat: take anything I say here as a personal opinion only. By all means, people should default to Declarative (or Declarative + Shared Libraries as needs grow) because it's easier and shows an opinionated "lit path" to successful CI/CD. And yes, we devote more work to Declarative specifically – I think the split is roughly 30/60/10 for work that is Declarative-only / General Pipeline for both / Scripted-only. That's because Declarative is newer and intended to be the "easy" mechanism that covers 80% of needs, where Scripted lets you roll-your-own and the majority of the features are just General Pipeline Stuff + Groovy.

          But as far as actually deprecating Scripted? You see people propose it from time to time, generally without understanding what it actually means, because Declarative is joined at the hip to Scripted and runs on top of it. For Declarative to replace Scripted, it which would have to absorb all the crazy use cases people can implement in Scripted due to its flexibility. Not to mention the number of organizations who would likely drop Pipeline and Jenkins entirely if they made a major investment in Scripted and Shared Libraries and had to walk away from 'em. I'd encourage anybody who thinks it's a good idea to come talk to me personally; I'll be happy to explain the other reasons why it's both extremely technically/architecturally difficult and a very poor idea (not going to run through it here because it's practically a novella).

          Ergo why I push back on trying to insert Scripted-like features such as this into Declarative – we don't want to find ourselves maintaining a new programming language. Better to make Declarative the best config language it can be, and delegate fancy stuff to a proper programming language (Groovy).

          Hope that all makes sense!

          Sam Van Oort added a comment - ftherien Caveat: take anything I say here as a personal opinion only. By all means, people should default to Declarative (or Declarative + Shared Libraries as needs grow) because it's easier and shows an opinionated "lit path" to successful CI/CD. And yes, we devote more work to Declarative specifically – I think the split is roughly 30/60/10 for work that is Declarative-only / General Pipeline for both / Scripted-only. That's because Declarative is newer and intended to be the "easy" mechanism that covers 80% of needs, where Scripted lets you roll-your-own and the majority of the features are just General Pipeline Stuff + Groovy. But as far as actually deprecating Scripted? You see people propose it from time to time, generally without understanding what it actually means, because Declarative is joined at the hip to Scripted and runs on top of it. For Declarative to replace Scripted, it which would have to absorb all the crazy use cases people can implement in Scripted due to its flexibility. Not to mention the number of organizations who would likely drop Pipeline and Jenkins entirely if they made a major investment in Scripted and Shared Libraries and had to walk away from 'em. I'd encourage anybody who thinks it's a good idea to come talk to me personally; I'll be happy to explain the other reasons why it's both extremely technically/architecturally difficult and a very poor idea (not going to run through it here because it's practically a novella). Ergo why I push back on trying to insert Scripted-like features such as this into Declarative – we don't want to find ourselves maintaining a new programming language. Better to make Declarative the best config language it can be, and delegate fancy stuff to a proper programming language (Groovy). Hope that all makes sense!

          Liam Newman added a comment -

          abayer

          I want to be clear, I personally am willing to accept that this is not only out of scope, but also counter to core design.  My point is I'd like to make sure that core design and direction are documented and understandable.

           

          Liam Newman added a comment - abayer I want to be clear, I personally am willing to accept that this is not only out of scope, but also counter to core design.  My point is I'd like to make sure that core design and direction are documented and understandable.  

          I very agree with bitwiseman and ftherien: Having some background information about the underlying design would it make a lot easier to build Jenkins pipelines the "right" way: Luckily, they are actively developed, thus it is extremely important for users to know about things you should not use because they are getting obsoleted in near future due to changes / advances in design.
          Frankly, today it is sometimes quite challening to justify the maintenance effort for pipelines, if the answer to the question "why did the jenkins update break our pipe?" repeatedly is: "they decided to change it" ...
          IMO, in the recent past Jenkins came to a very good way of communicating and handling of security related issues (pre-announcement mails, changelog, ...)
          Having a design outline and a roadmap with milestones IMO would be a good starting point to achieve a similar level of transparency for the design of a complex, production-grade CI system that Jenkins is.

          Florian Miedniak added a comment - I very agree with bitwiseman and ftherien : Having some background information about the underlying design would it make a lot easier to build Jenkins pipelines the "right" way: Luckily, they are actively developed, thus it is extremely important for users to know about things you should not use because they are getting obsoleted in near future due to changes / advances in design. Frankly, today it is sometimes quite challening to justify the maintenance effort for pipelines, if the answer to the question "why did the jenkins update break our pipe?" repeatedly is: "they decided to change it" ... IMO, in the recent past Jenkins came to a very good way of communicating and handling of security related issues (pre-announcement mails, changelog, ...) Having a design outline and a roadmap with milestones IMO would be a good starting point to achieve a similar level of transparency for the design of a complex, production-grade CI system that Jenkins is.

          Liam Newman added a comment -

          Bulk closing resolved issues.

          Liam Newman added a comment - Bulk closing resolved issues.

            abayer Andrew Bayer
            rpocase Robby Pocase
            Votes:
            43 Vote for this issue
            Watchers:
            57 Start watching this issue

              Created:
              Updated:
              Resolved: