Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42971

Pipeline script from SCM does not expand build parameters/env variables for lightweight checkouts

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • None
    • Jenkins LTS 2.32.3 + All plugins up to date.

      Pipeline: Build Step : 2.4
      Git client plugin : 2.3
    • git plugin 4.14.0, workflow-cps (pipeline-groovy) 3601.v9b_36a_d99e1cc

      Pipeline from SCM does not expand parameters or environment variables 

      Steps to reproduce:

      • Create a new Pipeline with a String Parameter  "PIPELINE_BRANCH"
      • In the pipeline definition ; Select "Pipeline Script from SCM"
      • Enter the repository URL
      • In branch to build, enter the parameter  : ${PIPELINE_BRANCH}
      • Enter your script path
      • Run the job with a valid branch

       An error in thrown :
      hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/${PIPELINE_BRANCH}:refs/remotes/origin/${PIPELINE_BRANCH} --prune" returned status code 128:
      stdout:
       

      ${PIPELINE_BRANCH} is not  replaced by its value.

      Same issue with an environment variable.

      Expanding parameters/variable is working fine for a freestyle or matrix job.

       

       

       

       

       

          [JENKINS-42971] Pipeline script from SCM does not expand build parameters/env variables for lightweight checkouts

          Pay Bas added a comment - - edited

          martink89 first of all: fantastic job on getting this fixed. Works perfectly for me for regular pipeline jobs after updating to:

          1. SCM API 621.vda_a_b_055e58f7
          2. Pipeline: Groovy Plugin 3601.v9b_36a_d99e1cc
          3. Git Plugin 4.14.1

          Your efforts are greatly appreciated!

          I had hoped that this would also automatically fix the same issue for the Kubernetes plugin https://plugins.jenkins.io/kubernetes/ but no such luck unfortunately.

          I have a simple pipeline job with "Pipeline script from SCM" (Git), the contents of which looks something like this:

          #!/usr/bin/env groovy
          pipeline {
            agent {
              kubernetes {
                yamlFile 'pod-spec-of-generic-jnlp-agent-nothing-special.yml'
              }
            }
            parameters {
              string(name: 'GIT_REF', description: 'A git tag, branch or commit to check-out', trim: true)
            }
            stages {
              stage('Build') {
                steps {
                  echo "Hi there!"
                }
              }
            }
          }

          Under the "Branches to build" settings, I have: "Branch Specifier (blank for 'any')": ${GIT_REF}

          Unfortunately, when building with any GIT_REF build-parameter value, the result is:

          hudson.plugins.git.GitException: Command "git fetch --tags --force --progress --prune -- origin +refs/heads/${GIT_REF}:refs/remotes/origin/${GIT_REF}" returned status code 128:
          stderr: fatal: couldn't find remote ref refs/heads/${GIT_REF} 

          It doesn't matter if I choose "Lightweight checkout" or not. The result is the same. The all too familiar lack of variable substitution.

           

          When using a regular (non-kubernetes) agent, it all works perfectly. Including lightweight checkout thanks to you.

          Using a regular branch expression instead of a param/env var also works fine with the kubernetes agent obviously.

          I'm not sure if you are familiar with the Kubernetes plugin or not. Should I inform the authors of your work on this issue so that they might implement it as well?

          Pay Bas added a comment - - edited martink89 first of all: fantastic job on getting this fixed. Works perfectly for me for regular pipeline jobs after updating to: SCM API 621.vda_a_b_055e58f7 Pipeline: Groovy Plugin 3601.v9b_36a_d99e1cc Git Plugin 4.14.1 Your efforts are greatly appreciated! I had hoped that this would also automatically fix the same issue for the Kubernetes plugin https://plugins.jenkins.io/kubernetes/  but no such luck unfortunately. I have a simple pipeline job with "Pipeline script from SCM" (Git), the contents of which looks something like this: #!/usr/bin/env groovy pipeline { agent { kubernetes { yamlFile 'pod-spec-of- generic -jnlp-agent-nothing-special.yml' } } parameters { string(name: 'GIT_REF' , description: 'A git tag, branch or commit to check-out' , trim: true ) } stages { stage( 'Build' ) { steps { echo "Hi there!" } } } } Under the "Branches to build" settings, I have: "Branch Specifier (blank for 'any')": ${GIT_REF } Unfortunately, when building with any GIT_REF build-parameter value, the result is: hudson.plugins.git.GitException: Command "git fetch --tags --force --progress --prune -- origin +refs/heads/${GIT_REF}:refs/remotes/origin/${GIT_REF}" returned status code 128: stderr: fatal: couldn't find remote ref refs/heads/${GIT_REF} It doesn't matter if I choose "Lightweight checkout" or not. The result is the same. The all too familiar lack of variable substitution.   When using a regular (non-kubernetes) agent, it all works perfectly. Including lightweight checkout thanks to you. Using a regular branch expression instead of a param/env var also works fine with the kubernetes agent obviously. I'm not sure if you are familiar with the Kubernetes plugin or not. Should I inform the authors of your work on this issue so that they might implement it as well?

          Jesse Glick added a comment -

          paybas I am not aware of any way in which using a Kubernetes agent would affect the behavior of this feature. Most likely there is some other difference between your two cases.

          Jesse Glick added a comment - paybas I am not aware of any way in which using a Kubernetes agent would affect the behavior of this feature. Most likely there is some other difference between your two cases.

          Pay Bas added a comment - - edited

          jglick thank you for your reply.

          I've taken your advice to heart and did some more testing. Turns out that if I put the pod spec in-line it does work.

          #!/usr/bin/env groovy
          pipeline {
            agent {
              kubernetes {
                yaml '''
          apiVersion: v1
          kind: Pod
          spec:
            containers:
              - name: jnlp
                image: registry.corp/my-project/my-agent:latest
          '''
              }
            }
            parameters {
              string(name: 'GIT_REF', description: 'A git tag, branch or commit to check-out', trim: true)
            }
            stages {
              stage('Build') {
                steps {
                  echo "Hi there!"
                }
              }
            }
          }
          

          Results in (with GIT_REF == my-awesome-branch as build-parameter):

          > git fetch --tags --force --progress -- ssh://git@bitbucket.corp:7999/my-project/my-repo.git +refs/heads/*:refs/remotes/origin/* # timeout=10
          > git rev-parse refs/remotes/origin/my-awesome-branch^{commit} # timeout=10
          > git rev-parse my-awesome-branch^{commit} # timeout=10
          Checking out Revision 889bcecd6fa2e5d30214455360030eade3fe4b82 (refs/remotes/origin/my-awesome-branch) 

          But I'm not quite sure which conclusion to draw from this.

          edit: turns out I missed kirk_gm 's reply, in which he describes the exact same behavior. But no solution as of yet.

          edit 2: after digging through the kubernetes plugin code, I can't find anything remotely related to this behavior either.

          Pay Bas added a comment - - edited jglick thank you for your reply. I've taken your advice to heart and did some more testing. Turns out that if I put the pod spec in-line it does work. #!/usr/bin/env groovy pipeline { agent { kubernetes { yaml ''' apiVersion: v1 kind: Pod spec: containers: - name: jnlp image: registry.corp/my-project/my-agent:latest ''' } } parameters { string(name: 'GIT_REF' , description: 'A git tag, branch or commit to check-out' , trim: true ) }  stages { stage( 'Build' ) { steps { echo "Hi there!" } } } } Results in (with GIT_REF == my-awesome-branch as build-parameter): > git fetch --tags --force --progress -- ssh: //git@bitbucket.corp:7999/my-project/my-repo.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse refs/remotes/origin/my-awesome-branch^{commit} # timeout=10 > git rev-parse my-awesome-branch^{commit} # timeout=10 Checking out Revision 889bcecd6fa2e5d30214455360030eade3fe4b82 (refs/remotes/origin/my-awesome-branch) But I'm not quite sure which conclusion to draw from this. edit: turns out I missed kirk_gm 's reply, in which he describes the exact same behavior. But no solution as of yet. edit 2: after digging through the kubernetes plugin code, I can't find anything remotely related to this behavior either.

          Jesse Glick added a comment - Ah, yamlFile specifically. That runs the readTrusted step ( reference ) and I suppose https://github.com/jenkinsci/workflow-multibranch-plugin/blob/e0a02cbc6c0b72cd12fbcf04a5223481a9d098e3/src/main/java/org/jenkinsci/plugins/workflow/multibranch/SCMBinder.java#L104 and (more to the point) https://github.com/jenkinsci/workflow-multibranch-plugin/blob/e0a02cbc6c0b72cd12fbcf04a5223481a9d098e3/src/main/java/org/jenkinsci/plugins/workflow/multibranch/ReadTrustedStep.java#L106 need to use the new overload.

          Mark Waite added a comment - - edited

          Thanks for the pointer to the line that needs the fix jglick ! That will resolve the issue reported in JENKINS-70155 by codesuki.

          paybas would you like to submit the pull request to fix the problem?

          Mark Waite added a comment - - edited Thanks for the pointer to the line that needs the fix jglick ! That will resolve the issue reported in JENKINS-70155 by codesuki . paybas would you like to submit the pull request to fix the problem?

          Pay Bas added a comment -

          Thanks jglick that makes sense!

          markewaite I haven't really touched any Java code in years, and I'm not really familiar with Jenkins plugin development. Only some trivial plugins, never the fundamental plugins.

          So yes: I could give it a shot. But it will take time (which I won't have for a couple of weeks). Especially since it will need unit tests.

          Pay Bas added a comment - Thanks jglick that makes sense! markewaite I haven't really touched any Java code in years, and I'm not really familiar with Jenkins plugin development. Only some trivial plugins, never the fundamental plugins. So yes: I could give it a shot. But it will take time (which I won't have for a couple of weeks). Especially since it will need unit tests.

          Jesse Glick added a comment -

          The code change (src/main/java/) is likely trivial: adding something like , build in a couple of spots, to thread this information along to the git plugin. But the POM would need to be updated to pick up the new dependencies, and writing a test case can be tricky, so this is not something I would recommend for someone just getting started with plugin development unless you want to get practice!

          Or martink89 are you interested in taking this on?

          I am not sure whether JENKINS-70155 and/or JENKINS-60697 are related. Those seem to discuss exotic refspecs for some use case I am not familiar with. This is specifically about expansion of a build parameter.

          Jesse Glick added a comment - The code change ( src/main/java/ ) is likely trivial: adding something like , build in a couple of spots, to thread this information along to the git plugin. But the POM would need to be updated to pick up the new dependencies, and writing a test case can be tricky, so this is not something I would recommend for someone just getting started with plugin development unless you want to get practice! Or martink89 are you interested in taking this on? I am not sure whether JENKINS-70155 and/or JENKINS-60697 are related. Those seem to discuss exotic refspecs for some use case I am not familiar with. This is specifically about expansion of a build parameter.

          Does this resolution only work for the branch field? I am trying to use a parameter for the 'Repository URL' field as well but it does not expand. It is working for branch though.

          Adam Brousseau added a comment - Does this resolution only work for the branch field? I am trying to use a parameter for the 'Repository URL' field as well but it does not expand. It is working for branch though.

          Steven added a comment -

          This is also still a problem for the Repository URL (but not the branch) for me as well:
          Jenkins version 2.401.3
          git plugin 5.2.0

          • Jenkins/Manage System/Global Properties:
            • "SHARED_REPO" = "ssh://git@server:23/path/project.git"
            • "SHARED_BRANCH" = "my_test_branch"
          • In Pipeline specification:
            • Pipeline script from SCM
            • SCM = Git
            • Repository URL: "${SHARED_REPO}"
            • Branch Specifier (blank for 'any'): "*/${SHARED_BRANCH}"
            • Lightweight checkout: CHECKED

          output:

          hudson.plugins.git.GitException: Command "C:\Program Files\Git\bin\git.exe fetch --tags --force --progress --prune -- origin +refs/heads/my_test_branch:refs/remotes/origin/my_test_branch" returned status code 128:
          stdout: 
          stderr: fatal: '${SHARED_REPO}' does not appear to be a git repository
          

          So, from the output, the plugin is parsing the environment property in the branch field, but not in the Repository URL.

          Steven added a comment - This is also still a problem for the Repository URL (but not the branch) for me as well: Jenkins version 2.401.3 git plugin 5.2.0 Jenkins/Manage System/Global Properties: "SHARED_REPO" = "ssh://git@server:23/path/project.git" "SHARED_BRANCH" = "my_test_branch" In Pipeline specification: Pipeline script from SCM SCM = Git Repository URL: "${SHARED_REPO}" Branch Specifier (blank for 'any'): "*/${SHARED_BRANCH}" Lightweight checkout: CHECKED output: hudson.plugins.git.GitException: Command "C:\Program Files\Git\bin\git.exe fetch --tags --force --progress --prune -- origin +refs/heads/my_test_branch:refs/remotes/origin/my_test_branch" returned status code 128: stdout: stderr: fatal: '${SHARED_REPO}' does not appear to be a git repository So, from the output, the plugin is parsing the environment property in the branch field, but not in the Repository URL.

          Pay Bas added a comment - - edited

          jglick do you want me to create a new ticket specifically for https://issues.jenkins.io/browse/JENKINS-42971?focusedId=433866&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-433866 ?

          I have the feeling that this will never be resolved as long as it's just a comment thread on a "Resolved" ticket.

          edit: Nevermind. Sorry for the "mention". https://issues.jenkins.io/browse/JENKINS-70155 was buried below the fold so I missed it.

          Pay Bas added a comment - - edited jglick do you want me to create a new ticket specifically for https://issues.jenkins.io/browse/JENKINS-42971?focusedId=433866&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-433866 ? I have the feeling that this will never be resolved as long as it's just a comment thread on a "Resolved" ticket. edit: Nevermind. Sorry for the "mention". https://issues.jenkins.io/browse/JENKINS-70155 was buried below the fold so I missed it.

            martink89 Martin Kosicky
            blatinville Bertrand Latinville
            Votes:
            22 Vote for this issue
            Watchers:
            40 Start watching this issue

              Created:
              Updated:
              Resolved: