Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42971

Pipeline script from SCM does not expand build parameters/env variables for lightweight checkouts

    XMLWordPrintable

Details

    • Bug
    • Status: Reopened (View Workflow)
    • Major
    • Resolution: Unresolved
    • scm-api-plugin
    • None
    • Jenkins LTS 2.32.3 + All plugins up to date.

      Pipeline: Build Step : 2.4
      Git client plugin : 2.3

    Description

      Pipeline from SCM does not expand parameters or environment variables 

      Steps to reproduce:

      • Create a new Pipeline with a String Parameter  "PIPELINE_BRANCH"
      • In the pipeline definition ; Select "Pipeline Script from SCM"
      • Enter the repository URL
      • In branch to build, enter the parameter  : ${PIPELINE_BRANCH}
      • Enter your script path
      • Run the job with a valid branch

       An error in thrown :
      hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/${PIPELINE_BRANCH}:refs/remotes/origin/${PIPELINE_BRANCH} --prune" returned status code 128:
      stdout:
       

      ${PIPELINE_BRANCH} is not  replaced by its value.

      Same issue with an environment variable.

      Expanding parameters/variable is working fine for a freestyle or matrix job.

       

       

       

       

       

      Attachments

        Issue Links

          Activity

            jglick Jesse Glick added a comment -

            Wrong component.

            Works if you have workflow-job 2.7+.

            jglick Jesse Glick added a comment - Wrong component. Works if you have workflow-job 2.7+.
            liyatikal liyatikal added a comment -

            How is this bug resolved? 

            I'm using Jenkins 2.73.2, pipeline job and it still doesn't work.

            Is there a workaround for this?

            liyatikal liyatikal added a comment - How is this bug resolved?  I'm using Jenkins 2.73.2, pipeline job and it still doesn't work. Is there a workaround for this?
            liyatikal liyatikal added a comment -

            The workaround that worked was to uncheck  the 'lightweight checkout'

            liyatikal liyatikal added a comment - The workaround that worked was to uncheck  the 'lightweight checkout'
            rodrigc Craig Rodrigues added a comment - See comment here: https://issues.jenkins-ci.org/browse/JENKINS-28447?focusedCommentId=308351&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-308351   And also JENKINS-48431
            tdensmore Todd Densmore added a comment -

            I am still seeing this as well. 

            tdensmore Todd Densmore added a comment - I am still seeing this as well. 

            I am also seeing this with Jenkins version 2.176.1 (from the Debian package repo provided by the Jenkins project).

            oschneider_precitec Oliver Schneider added a comment - I am also seeing this with Jenkins version 2.176.1 (from the Debian package repo provided by the Jenkins project).
            medianick Nick Jones added a comment -

            We're seeing this same behavior today running Pipeline: Job (workflow-job) 2.32 (on Jenkins 2.176.1 LTS), with the "Lightweight checkout" checkbox unchecked. Is there a possibility that the system is still choosing a lightweight checkout essentially "on the fly", and if so, is there a way to force a heavyweight checkout?

            medianick Nick Jones added a comment - We're seeing this same behavior today running Pipeline: Job ( workflow-job ) 2.32 (on Jenkins 2.176.1 LTS), with the "Lightweight checkout" checkbox unchecked. Is there a possibility that the system is still choosing a lightweight checkout essentially "on the fly", and if so, is there a way to force a heavyweight checkout?
            medianick Nick Jones added a comment -

            Aha – for us the issue was that we were reading a file from the workspace before the start of the the pipeline block, using a command like `def common = evaluate readTrusted('./Common.groovy')`. This seems to have bypassed the variable resolution. Removing this line caused the expected behavior to start again.

            medianick Nick Jones added a comment - Aha – for us the issue was that we were reading a file from the workspace before the start of the the pipeline block, using a command like `def common = evaluate readTrusted('./Common.groovy')` . This seems to have bypassed the variable resolution. Removing this line caused the expected behavior to start again.
            docwarems Michael S. added a comment - - edited

            We have still a problem with Pipeline: Job 2.32 (even after upgrade to 2.36) on Jenkins 2.179. "Lightweight checkout" checkbox unchecked. Parameter Expansion does not work if parameter comes from "Properties File Path", It works, however, with parameter from "Properties Content". Too bad.

            BTW, our properties file is correct because it previously worked with a freestyle job.

            I created new issue JENKINS-60250.

            docwarems Michael S. added a comment - - edited We have still a problem with Pipeline: Job 2.32 (even after upgrade to 2.36) on Jenkins 2.179. "Lightweight checkout" checkbox unchecked. Parameter Expansion does not work if parameter comes from "Properties File Path", It works, however, with parameter from "Properties Content". Too bad. BTW, our properties file is correct because it previously worked with a freestyle job. I created new issue JENKINS-60250 .
            tamerlaha ipleten added a comment -

            I could confirm that using `readTrusted` causes such error.

            tamerlaha ipleten added a comment - I could confirm that using `readTrusted` causes such error.
            kirk_gm Max Golionko added a comment -

            We have same problem, but in our case we don't have readTrusted
            But we are using kubernetes with kubernetes-plugin, and we were using yaml file for pod configuration

              agent {
                kubernetes {
                  yamlFile "pod.yaml"
                }
              }  
            

            seems that underhood it calls readTrusted
            we tried to use

              agent {
                kubernetes {
                  yaml readFile("./pod.yaml")
                }
              }
            

            but this also didn't work
            only inline pod yaml helped to solve this issue

            agent {
                kubernetes {
                  yaml """
            apiVersion: v1
            kind: Pod
            spec:
              containers:
              - name: jnlp
                image: imagename
                imagePullPolicy: Always
            
            """
                }
              }    
            
            kirk_gm Max Golionko added a comment - We have same problem, but in our case we don't have readTrusted But we are using kubernetes with kubernetes-plugin, and we were using yaml file for pod configuration agent { kubernetes { yamlFile "pod.yaml" } } seems that underhood it calls readTrusted we tried to use agent { kubernetes { yaml readFile( "./pod.yaml" ) } } but this also didn't work only inline pod yaml helped to solve this issue agent { kubernetes { yaml """ apiVersion: v1 kind: Pod spec: containers: - name: jnlp image: imagename imagePullPolicy: Always """ } }
            martink89 Martin Kosicky added a comment - - edited

            The issue still persists. I identified 3 modules that need to be fixed. The 

            scm-api plugin does not forward the Run<?,?> to SCMFileSystem builder

            git plugin would need to implement a new builder that would use the Run<?,?>

            workflow-cps plugin needs to send the Run<?,?> to the scm-api in the CpsScmFlowDefinition.java to SCMFileSystem.of (which needs to be extended)

             

            I have implemented a fix on my local machine, I have a draft opened on github although I would need some feedback, if this seems OK for the devs https://github.com/jenkinsci/scm-api-plugin/pull/160 if I am on right track
            Also I got on my PC fix on the other 2 repos so It was working, although this will require some magic to make it production quality

            martink89 Martin Kosicky added a comment - - edited The issue still persists. I identified 3 modules that need to be fixed. The  scm-api plugin does not forward the Run<?,?> to SCMFileSystem builder git plugin would need to implement a new builder that would use the Run<?,?> workflow-cps plugin needs to send the Run<?,?> to the scm-api in the CpsScmFlowDefinition.java to SCMFileSystem.of (which needs to be extended)   I have implemented a fix on my local machine, I have a draft opened on github although I would need some feedback, if this seems OK for the devs https://github.com/jenkinsci/scm-api-plugin/pull/160 if I am on right track Also I got on my PC fix on the other 2 repos so It was working, although this will require some magic to make it production quality
            kon Kalle Niemitalo added a comment - - edited

            I wondered whether I should file a request to make BitbucketSCMFileSystem in the Bitbucket Server Integration plugin implement the method that was added to SCM API. However, it seems this bug is not currently reproducible with Bitbucket Server Integration because it does not yet support lightweight checkout for this scenario. Therefore, I'm not filing such a request.

            I tried to reproduce this issue with Jenkins 2.346.2, Bitbucket Server Integration plugin 3.2.2, Git plugin 4.11.4, SCM API plugin 620.v0a_5b_1f8054c0, and Pipeline: Groovy plugin 2759.v87459c4eea_ca_. Jenkins logged "Lightweight checkout support not available, falling back to full checkout", created a Git repository on the Jenkins controller, fetched refs/heads/* from the remote, resolved the commit ID of ${PIPELINE_BRANCH}, checked out that commit, and read the pipeline from the worktree. If I understand correctly, a lightweight checkout using the Bitbucket Server Integration plugin would have requested the file content from the Bitbucket Server REST API instead of creating a Git repository on the Jenkins controller, and a lightweight checkout using the Git plugin would have streamed the blob from the Git repository instead of checking out all the files.

            kon Kalle Niemitalo added a comment - - edited I wondered whether I should file a request to make BitbucketSCMFileSystem in the Bitbucket Server Integration plugin implement the method that was added to SCM API. However, it seems this bug is not currently reproducible with Bitbucket Server Integration because it does not yet support lightweight checkout for this scenario. Therefore, I'm not filing such a request. I tried to reproduce this issue with Jenkins 2.346.2, Bitbucket Server Integration plugin 3.2.2, Git plugin 4.11.4, SCM API plugin 620.v0a_5b_1f8054c0, and Pipeline: Groovy plugin 2759.v87459c4eea_ca_. Jenkins logged "Lightweight checkout support not available, falling back to full checkout", created a Git repository on the Jenkins controller, fetched refs/heads/* from the remote, resolved the commit ID of ${PIPELINE_BRANCH}, checked out that commit, and read the pipeline from the worktree. If I understand correctly, a lightweight checkout using the Bitbucket Server Integration plugin would have requested the file content from the Bitbucket Server REST API instead of creating a Git repository on the Jenkins controller, and a lightweight checkout using the Git plugin would have streamed the blob from the Git repository instead of checking out all the files.

            People

              martink89 Martin Kosicky
              blatinville Bertrand Latinville
              Votes:
              19 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

                Created:
                Updated: