Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43894

Environment variables not resolved in Pipeline SCM -> Advanced clone behaviours -> Path of the reference repo

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Major Major
    • git-plugin
    • Jenkins 2.89.3, workflow-job 2.18-SNAPSHOT

      Environment variables are not resolved in 'Advanced clone behaviours' -> 'Path of the reference repo to use during clone' when configuring git reference repository in the Pipeline SCM section.

      Steps to reproduce:

      • Create pipeline project
      • In configure: fill in repository URL, Branch Specifier and Script Path (Jenkinsfile).
      • In 'Additional Behaviours' select 'Advanced Clone Behaviours'. Set 'Path of the reference repo to use during clone' to some value depending on an environment variable, e.g. ${HOME}/jenkins/reference/big-git-repo.git
      • At URL, create a git repository with Jenkinsfile. Put 'node('master') { checkout scm }' in the Jenkinsfile.
      • Run the job. It will report: 
        Cloning repository ...URL...
         > git init ... # timeout=10
        ERROR: Reference path does not exist: ${HOME}/jenkins/reference/big-git-repo.git

      Would be very handy to have this feature. In my case, I want to use the 'checkout scm' shortcut in a pipeline script that runs several builds in parallel on different platforms. So I want to specify where a reference repository exists on a specific node. For instance, on a Windows node I want to put it in D:\jenkins\reference, on a Mac – to /Users/jenkins/reference and, finally, on linux – /home/jenkins/reference. I could set an environment variable ${REFERENCE} to the above values in node configuration and than use it in 'Path of the reference repo to use during clone' as ${REFERENCE}.

       

          [JENKINS-43894] Environment variables not resolved in Pipeline SCM -> Advanced clone behaviours -> Path of the reference repo

          stevenfoster For cloud slaves you will need to support setting environment variables via Node Properties in Jenkins. That should be implemented in a plugin that provides those slaves.

          The patch above should work for environment variables set on an agent.

          Anna Tikhonova added a comment - stevenfoster  For cloud slaves you will need to support setting environment variables via Node Properties in Jenkins. That should be implemented in a plugin that provides those slaves. The patch above should work for environment variables set on an agent.

          As far as I can tell the fix for this https://github.com/jenkinsci/git-plugin/pull/575 was merged to master, so should be in the next 4.0 based release.

          Patrick Ruckstuhl added a comment - As far as I can tell the fix for this https://github.com/jenkinsci/git-plugin/pull/575 was merged to master, so should be in the next 4.0 based release.

          Jesse Glick added a comment -

          By the way the use case described in the issue description seems like a bad idea to me. More straightforward to have the script determine the Git commit being built, and then rather than running checkout scm on each node, explicitly construct a GitSCM configured exactly the way you want, according to whatever Groovy logic you need. The kind of plugin-based environment variable expansion seen in this PR should only be necessary in freestyle projects.

          Jesse Glick added a comment - By the way the use case described in the issue description seems like a bad idea to me. More straightforward to have the script determine the Git commit being built, and then rather than running checkout scm on each node, explicitly construct a GitSCM configured exactly the way you want, according to whatever Groovy logic you need. The kind of plugin-based environment variable expansion seen in this PR should only be necessary in freestyle projects.

          jglick maybe I'm missing something but to me this looks totally normal to use in pipeline jobs.
          We have some linux slaves and some windows slaves and they each have a different location of the reference repo. I can even think of cases where you have e.g. linux nodes which for whatever reason have different paths to the reference repo. Having this configurable on a slave level and then just use it in the multibranch config is perfect, that way the pipeline does not care about how the checkout happens.

          Patrick Ruckstuhl added a comment - jglick maybe I'm missing something but to me this looks totally normal to use in pipeline jobs. We have some linux slaves and some windows slaves and they each have a different location of the reference repo. I can even think of cases where you have e.g. linux nodes which for whatever reason have different paths to the reference repo. Having this configurable on a slave level and then just use it in the multibranch config is perfect, that way the pipeline does not care about how the checkout happens.

          Exactly that. Reference repo paths, library paths, tool paths: This all are node specific settings which have no place in the actual pipeline description. As an extreme example imagine that your project is published on Github or the like. Do you really want to share with the world the gritty details of your local Jenkins configuration?

          Mathias Hasselmann added a comment - Exactly that. Reference repo paths, library paths, tool paths: This all are node specific settings which have no place in the actual pipeline description. As an extreme example imagine that your project is published on Github or the like. Do you really want to share with the world the gritty details of your local Jenkins configuration?

          Jesse Glick added a comment -

          library paths, tool paths

          These are of course better handled via Docker containers or the like if you can manage it.

          node specific settings which have no place in the actual pipeline description

          I did mean to imply that the Jenkinsfile must list actual filesystem paths on each possible node, merely that the program has some way of determining this information on the fly, preferably not involving Jenkins global configuration. That could be an environment variable set on the agent’s account, etc. The point is that the selection and configuration of a reference repo is something under the control of the job maintainer or Jenkins admin, rather than being baked into a plugin.

          imagine that your project is published on Github

          I do not have to imagine that, it is a daily reality.

          Do you really want to share with the world the gritty details of your local Jenkins configuration?

          In the case of the Jenkins project, in fact we do, but in case you do not, there are various alternatives.

          Jesse Glick added a comment - library paths, tool paths These are of course better handled via Docker containers or the like if you can manage it. node specific settings which have no place in the actual pipeline description I did mean to imply that the Jenkinsfile must list actual filesystem paths on each possible node, merely that the program has some way of determining this information on the fly, preferably not involving Jenkins global configuration. That could be an environment variable set on the agent’s account, etc. The point is that the selection and configuration of a reference repo is something under the control of the job maintainer or Jenkins admin, rather than being baked into a plugin . imagine that your project is published on Github I do not have to imagine that, it is a daily reality. Do you really want to share with the world the gritty details of your local Jenkins configuration? In the case of the Jenkins project, in fact we do , but in case you do not, there are various alternatives .

          D Pasto added a comment -

          Any updates on releasing this?  I verified it does not work on my v4.0 plugin (can resolve $JENKINS_HOME but not $HOME so is useless) and the changelog does not list it for that or v4.1 beta

          We have pipelines across many nodes, including different platforms, so we can't take full advantage of reference repos (the workaround of manually re-defining the checkout for every stage would be maintenance disaster) — we need a way of dynamically building the reference repo path per node per pipeline.

          D Pasto added a comment - Any updates on releasing this?  I verified it does not work on my v4.0 plugin (can resolve $JENKINS_HOME but not $HOME so is useless) and the changelog does not list it for that or v4.1 beta We have pipelines across many nodes, including different platforms, so we can't take full advantage of reference repos (the workaround of manually re-defining the checkout for every stage would be maintenance disaster) — we need a way of dynamically building the reference repo path per node per pipeline.

          Mark Waite added a comment -

          I've proposed automatic repository caching as a Google Summer of Code idea and have offered the pull request as a starting point for consideration.

          Have you tried the alternative that is described by Jesse Glick? It seems like it would work in your case if you perform the environment variable expansion in the Jenkins checkout statement.

          Mark Waite added a comment - I've proposed automatic repository caching as a Google Summer of Code idea and have offered the pull request as a starting point for consideration. Have you tried the alternative that is described by Jesse Glick? It seems like it would work in your case if you perform the environment variable expansion in the Jenkins checkout statement.

          wgc123 did you actually specify HOME as an explicit variable on the node or are you just depending on an existing environment variable. If I remember correctly we're using this with some 4.* version for a while, but always with explicitly specified node variables.

          Patrick Ruckstuhl added a comment - wgc123 did you actually specify HOME as an explicit variable on the node or are you just depending on an existing environment variable. If I remember correctly we're using this with some 4.* version for a while, but always with explicitly specified node variables.

          D Pasto added a comment - - edited

          Good call tario  It works!  I assumed I could just use environment variables that I believe existed, but whether it doesn't really exist, or the plugin only sees a subset of environment variables, it correctly found the reference repo using an env defined in the node!  It's not as clean as an automatically defined variable would be, but it looks like it works!  We have a good sized repo so even though we're on-prem, I was seeing 2+ minute fetches for those stages where the hard-coded path was wrong, but this brings it down to 15 seconds in my first few runs

          Edit: forgot the most important part - this lets me define a single path (using forward slashes) that works on both Linux and Windows nodes.

          D Pasto added a comment - - edited Good call tario   It works!  I assumed I could just use environment variables that I believe existed, but whether it doesn't really exist, or the plugin only sees a subset of environment variables, it correctly found the reference repo using an env defined in the node!  It's not as clean as an automatically defined variable would be, but it looks like it works!  We have a good sized repo so even though we're on-prem, I was seeing 2+ minute fetches for those stages where the hard-coded path was wrong, but this brings it down to 15 seconds in my first few runs Edit: forgot the most important part - this lets me define a single path (using forward slashes) that works on both Linux and Windows nodes.

            atikhono Anna Tikhonova
            atikhono Anna Tikhonova
            Votes:
            3 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: