Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43894

Environment variables not resolved in Pipeline SCM -> Advanced clone behaviours -> Path of the reference repo

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Component/s: git-plugin
    • Labels:
    • Environment:
      Jenkins 2.89.3, workflow-job 2.18-SNAPSHOT
    • Similar Issues:

      Description

      Environment variables are not resolved in 'Advanced clone behaviours' -> 'Path of the reference repo to use during clone' when configuring git reference repository in the Pipeline SCM section.

      Steps to reproduce:

      • Create pipeline project
      • In configure: fill in repository URL, Branch Specifier and Script Path (Jenkinsfile).
      • In 'Additional Behaviours' select 'Advanced Clone Behaviours'. Set 'Path of the reference repo to use during clone' to some value depending on an environment variable, e.g. ${HOME}/jenkins/reference/big-git-repo.git
      • At URL, create a git repository with Jenkinsfile. Put 'node('master') { checkout scm }' in the Jenkinsfile.
      • Run the job. It will report: 
        Cloning repository ...URL...
         > git init ... # timeout=10
        ERROR: Reference path does not exist: ${HOME}/jenkins/reference/big-git-repo.git

      Would be very handy to have this feature. In my case, I want to use the 'checkout scm' shortcut in a pipeline script that runs several builds in parallel on different platforms. So I want to specify where a reference repository exists on a specific node. For instance, on a Windows node I want to put it in D:\jenkins\reference, on a Mac – to /Users/jenkins/reference and, finally, on linux – /home/jenkins/reference. I could set an environment variable ${REFERENCE} to the above values in node configuration and than use it in 'Path of the reference repo to use during clone' as ${REFERENCE}.

       

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            library paths, tool paths

            These are of course better handled via Docker containers or the like if you can manage it.

            node specific settings which have no place in the actual pipeline description

            I did mean to imply that the Jenkinsfile must list actual filesystem paths on each possible node, merely that the program has some way of determining this information on the fly, preferably not involving Jenkins global configuration. That could be an environment variable set on the agent’s account, etc. The point is that the selection and configuration of a reference repo is something under the control of the job maintainer or Jenkins admin, rather than being baked into a plugin.

            imagine that your project is published on Github

            I do not have to imagine that, it is a daily reality.

            Do you really want to share with the world the gritty details of your local Jenkins configuration?

            In the case of the Jenkins project, in fact we do, but in case you do not, there are various alternatives.

            Show
            jglick Jesse Glick added a comment - library paths, tool paths These are of course better handled via Docker containers or the like if you can manage it. node specific settings which have no place in the actual pipeline description I did mean to imply that the Jenkinsfile must list actual filesystem paths on each possible node, merely that the program has some way of determining this information on the fly, preferably not involving Jenkins global configuration. That could be an environment variable set on the agent’s account, etc. The point is that the selection and configuration of a reference repo is something under the control of the job maintainer or Jenkins admin, rather than being baked into a plugin . imagine that your project is published on Github I do not have to imagine that, it is a daily reality. Do you really want to share with the world the gritty details of your local Jenkins configuration? In the case of the Jenkins project, in fact we do , but in case you do not, there are various alternatives .
            Hide
            wgc123 D Pasto added a comment -

            Any updates on releasing this?  I verified it does not work on my v4.0 plugin (can resolve $JENKINS_HOME but not $HOME so is useless) and the changelog does not list it for that or v4.1 beta

            We have pipelines across many nodes, including different platforms, so we can't take full advantage of reference repos (the workaround of manually re-defining the checkout for every stage would be maintenance disaster) — we need a way of dynamically building the reference repo path per node per pipeline.

            Show
            wgc123 D Pasto added a comment - Any updates on releasing this?  I verified it does not work on my v4.0 plugin (can resolve $JENKINS_HOME but not $HOME so is useless) and the changelog does not list it for that or v4.1 beta We have pipelines across many nodes, including different platforms, so we can't take full advantage of reference repos (the workaround of manually re-defining the checkout for every stage would be maintenance disaster) — we need a way of dynamically building the reference repo path per node per pipeline.
            Hide
            markewaite Mark Waite added a comment -

            I've proposed automatic repository caching as a Google Summer of Code idea and have offered the pull request as a starting point for consideration.

            Have you tried the alternative that is described by Jesse Glick? It seems like it would work in your case if you perform the environment variable expansion in the Jenkins checkout statement.

            Show
            markewaite Mark Waite added a comment - I've proposed automatic repository caching as a Google Summer of Code idea and have offered the pull request as a starting point for consideration. Have you tried the alternative that is described by Jesse Glick? It seems like it would work in your case if you perform the environment variable expansion in the Jenkins checkout statement.
            Hide
            tario Patrick Ruckstuhl added a comment -

            D Pasto did you actually specify HOME as an explicit variable on the node or are you just depending on an existing environment variable. If I remember correctly we're using this with some 4.* version for a while, but always with explicitly specified node variables.

            Show
            tario Patrick Ruckstuhl added a comment - D Pasto did you actually specify HOME as an explicit variable on the node or are you just depending on an existing environment variable. If I remember correctly we're using this with some 4.* version for a while, but always with explicitly specified node variables.
            Hide
            wgc123 D Pasto added a comment - - edited

            Good call Patrick Ruckstuhl  It works!  I assumed I could just use environment variables that I believe existed, but whether it doesn't really exist, or the plugin only sees a subset of environment variables, it correctly found the reference repo using an env defined in the node!  It's not as clean as an automatically defined variable would be, but it looks like it works!  We have a good sized repo so even though we're on-prem, I was seeing 2+ minute fetches for those stages where the hard-coded path was wrong, but this brings it down to 15 seconds in my first few runs

            Edit: forgot the most important part - this lets me define a single path (using forward slashes) that works on both Linux and Windows nodes.

            Show
            wgc123 D Pasto added a comment - - edited Good call Patrick Ruckstuhl   It works!  I assumed I could just use environment variables that I believe existed, but whether it doesn't really exist, or the plugin only sees a subset of environment variables, it correctly found the reference repo using an env defined in the node!  It's not as clean as an automatically defined variable would be, but it looks like it works!  We have a good sized repo so even though we're on-prem, I was seeing 2+ minute fetches for those stages where the hard-coded path was wrong, but this brings it down to 15 seconds in my first few runs Edit: forgot the most important part - this lets me define a single path (using forward slashes) that works on both Linux and Windows nodes.

              People

              Assignee:
              atikhono Anna Tikhonova
              Reporter:
              atikhono Anna Tikhonova
              Votes:
              3 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: