Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41335

Allow variables and functions to be defined within pipeline to be used in any stage

    XMLWordPrintable

Details

    Description

      I have several use cases where I may operate on a list or map on different stages of a pipeline. This variable may not be known ahead of time - it could be defined within a stage. I'd like a syntax that allows me to define variables within a pipeline.

      Today, I have to define a shared variable before entering pipeline {}.

      Jenkinsfile
      #!/bin/groovy
      // Define outside of pipeline to make sure accessible in all script {} blocks
      def my_list
      
      pipeline {
          agent { label 'label' }
          stages {
              stage('stage1') {
                steps {
                 script {
                     my_list = [1,2,3]
                 }
                }
              }
          
              stage('stage2') {
                steps {
                  script {
                      for(int i=0; i<my_list.size();i++) {
                          echo "Doing something with ${my_list[i]}"
                      }
                  }
                }
              }
          }
      }
      

      A simple approach would be to allow "def" to work anywhere within pipeline {}. This avoids any overly new syntax for pipeline.

      Another option would allow the user to provide a special pipeline {} block for defining non-env/params style variables. This could be helpful in allowing the user to use items available in script {} for defining shared state.

      Jenkinsfile with possible example syntax
      #!/bin/groovy
      
      
      pipeline {
          agent { label 'label' }    
      
          define {
             def my_map = [:] //empty map
             def my_list //undefined shared variable
           }
      
          stages {
              stage('stage1') {
                steps {
                  script {
                    //If def can be anywhere in pipeline {}, drop script {}
                    my_list = [1,2,3]
                  }
                }
              }
          
              stage('stage2') {
                steps {
                  script {
                      for(int i=0; i<my_list.size();i++) {
                          echo "Doing something with ${my_list[i]}"
                      }
                  }
                }
              }
          }
      }
      

      Attachments

        Issue Links

          Activity

            svanoort I thought I had read somehwhere that the roadmap was for scripted to eventually be deprecated, and that declarative would replace it.

            In any case, I second Liam's point that it would be interesting to see documents explaining this vision. It would help, at the very least,  people to make long term decisions for their CI architecture around Jenkins.

            ftherien Francis Therien added a comment - svanoort I thought I had read somehwhere that the roadmap was for scripted to eventually be deprecated, and that declarative would replace it. In any case, I second Liam's point that it would be interesting to see documents explaining this vision. It would help, at the very least,  people to make long term decisions for their CI architecture around Jenkins.
            svanoort Sam Van Oort added a comment -

            ftherien Caveat: take anything I say here as a personal opinion only. By all means, people should default to Declarative (or Declarative + Shared Libraries as needs grow) because it's easier and shows an opinionated "lit path" to successful CI/CD. And yes, we devote more work to Declarative specifically – I think the split is roughly 30/60/10 for work that is Declarative-only / General Pipeline for both / Scripted-only. That's because Declarative is newer and intended to be the "easy" mechanism that covers 80% of needs, where Scripted lets you roll-your-own and the majority of the features are just General Pipeline Stuff + Groovy.

            But as far as actually deprecating Scripted? You see people propose it from time to time, generally without understanding what it actually means, because Declarative is joined at the hip to Scripted and runs on top of it. For Declarative to replace Scripted, it which would have to absorb all the crazy use cases people can implement in Scripted due to its flexibility. Not to mention the number of organizations who would likely drop Pipeline and Jenkins entirely if they made a major investment in Scripted and Shared Libraries and had to walk away from 'em. I'd encourage anybody who thinks it's a good idea to come talk to me personally; I'll be happy to explain the other reasons why it's both extremely technically/architecturally difficult and a very poor idea (not going to run through it here because it's practically a novella).

            Ergo why I push back on trying to insert Scripted-like features such as this into Declarative – we don't want to find ourselves maintaining a new programming language. Better to make Declarative the best config language it can be, and delegate fancy stuff to a proper programming language (Groovy).

            Hope that all makes sense!

            svanoort Sam Van Oort added a comment - ftherien Caveat: take anything I say here as a personal opinion only. By all means, people should default to Declarative (or Declarative + Shared Libraries as needs grow) because it's easier and shows an opinionated "lit path" to successful CI/CD. And yes, we devote more work to Declarative specifically – I think the split is roughly 30/60/10 for work that is Declarative-only / General Pipeline for both / Scripted-only. That's because Declarative is newer and intended to be the "easy" mechanism that covers 80% of needs, where Scripted lets you roll-your-own and the majority of the features are just General Pipeline Stuff + Groovy. But as far as actually deprecating Scripted? You see people propose it from time to time, generally without understanding what it actually means, because Declarative is joined at the hip to Scripted and runs on top of it. For Declarative to replace Scripted, it which would have to absorb all the crazy use cases people can implement in Scripted due to its flexibility. Not to mention the number of organizations who would likely drop Pipeline and Jenkins entirely if they made a major investment in Scripted and Shared Libraries and had to walk away from 'em. I'd encourage anybody who thinks it's a good idea to come talk to me personally; I'll be happy to explain the other reasons why it's both extremely technically/architecturally difficult and a very poor idea (not going to run through it here because it's practically a novella). Ergo why I push back on trying to insert Scripted-like features such as this into Declarative – we don't want to find ourselves maintaining a new programming language. Better to make Declarative the best config language it can be, and delegate fancy stuff to a proper programming language (Groovy). Hope that all makes sense!
            bitwiseman Liam Newman added a comment -

            abayer

            I want to be clear, I personally am willing to accept that this is not only out of scope, but also counter to core design.  My point is I'd like to make sure that core design and direction are documented and understandable.

             

            bitwiseman Liam Newman added a comment - abayer I want to be clear, I personally am willing to accept that this is not only out of scope, but also counter to core design.  My point is I'd like to make sure that core design and direction are documented and understandable.  

            I very agree with bitwiseman and ftherien: Having some background information about the underlying design would it make a lot easier to build Jenkins pipelines the "right" way: Luckily, they are actively developed, thus it is extremely important for users to know about things you should not use because they are getting obsoleted in near future due to changes / advances in design.
            Frankly, today it is sometimes quite challening to justify the maintenance effort for pipelines, if the answer to the question "why did the jenkins update break our pipe?" repeatedly is: "they decided to change it" ...
            IMO, in the recent past Jenkins came to a very good way of communicating and handling of security related issues (pre-announcement mails, changelog, ...)
            Having a design outline and a roadmap with milestones IMO would be a good starting point to achieve a similar level of transparency for the design of a complex, production-grade CI system that Jenkins is.

            fmiedniak Florian Miedniak added a comment - I very agree with bitwiseman and ftherien : Having some background information about the underlying design would it make a lot easier to build Jenkins pipelines the "right" way: Luckily, they are actively developed, thus it is extremely important for users to know about things you should not use because they are getting obsoleted in near future due to changes / advances in design. Frankly, today it is sometimes quite challening to justify the maintenance effort for pipelines, if the answer to the question "why did the jenkins update break our pipe?" repeatedly is: "they decided to change it" ... IMO, in the recent past Jenkins came to a very good way of communicating and handling of security related issues (pre-announcement mails, changelog, ...) Having a design outline and a roadmap with milestones IMO would be a good starting point to achieve a similar level of transparency for the design of a complex, production-grade CI system that Jenkins is.
            bitwiseman Liam Newman added a comment -

            Bulk closing resolved issues.

            bitwiseman Liam Newman added a comment - Bulk closing resolved issues.

            People

              abayer Andrew Bayer
              rpocase Robby Pocase
              Votes:
              43 Vote for this issue
              Watchers:
              57 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: