-
Improvement
-
Resolution: Fixed
-
Major
-
Jenkins 2.89.3, workflow-job 2.18-SNAPSHOT
Environment variables are not resolved in 'Advanced clone behaviours' -> 'Path of the reference repo to use during clone' when configuring git reference repository in the Pipeline SCM section.
Steps to reproduce:
- Create pipeline project
- In configure: fill in repository URL, Branch Specifier and Script Path (Jenkinsfile).
- In 'Additional Behaviours' select 'Advanced Clone Behaviours'. Set 'Path of the reference repo to use during clone' to some value depending on an environment variable, e.g. ${HOME}/jenkins/reference/big-git-repo.git
- At URL, create a git repository with Jenkinsfile. Put 'node('master') { checkout scm }' in the Jenkinsfile.
- Run the job. It will report:
Cloning repository ...URL... > git init ... # timeout=10 ERROR: Reference path does not exist: ${HOME}/jenkins/reference/big-git-repo.git
Would be very handy to have this feature. In my case, I want to use the 'checkout scm' shortcut in a pipeline script that runs several builds in parallel on different platforms. So I want to specify where a reference repository exists on a specific node. For instance, on a Windows node I want to put it in D:\jenkins\reference, on a Mac – to /Users/jenkins/reference and, finally, on linux – /home/jenkins/reference. I could set an environment variable ${REFERENCE} to the above values in node configuration and than use it in 'Path of the reference repo to use during clone' as ${REFERENCE}.
- relates to
-
-
- Open
-
-
JENKINS-65315 Environment variables not resolved in Pipeline SCM -> Advanced sub-modules behaviours -> Path of the reference repo
-
- Open
-
- links to
[JENKINS-43894] Environment variables not resolved in Pipeline SCM -> Advanced clone behaviours -> Path of the reference repo
Anna Tikhonova
created issue -
| Assignee | Original: Mark Waite [ markewaite ] |
As another alternate, since you're using pipeline, the git plugin could be extended to automatically use a reference repository. The mercurial plugin uses that technique to reduce duplication on disc.
Anna Tikhonova
added a comment - markewaite Could you please clarify a bit how it works in the Mercurial plugin? I've no experience with Mercurial and can't find anything resembling reference repositories implementation in the source code.
Anna Tikhonova
added a comment - markewaite Could you please clarify a bit how it works in the Mercurial plugin? I've no experience with Mercurial and can't find anything resembling reference repositories implementation in the source code. I believe Jesse Glick described that the SCMFileSystem implementation in the Mercurial plugin uses the initial checkout of the repository (for polling purposes) as a cache for other copies on the disc. The concept was what was interesting to me, more than the specifics of the implementation in the Mercurial plugin.
It will be a different implementation with the git plugin, since git's concept of a reference repository is something that Mercurial does not have (as far as I know).
Anna Tikhonova
added a comment - markewaite Ah, I see. Thanks! Having an automatic cache for git repositories would be totally cool for Multi-branch pipelines. Last time I checked this project type (a long time ago), it created a new clone for each branch, which is not good for large repositories. Maybe it's been fixed somehow, dunno. Would be nice to have it as well.
This issue is unassigned right now. Do I have any chance to have it fixed soon?
Anna Tikhonova
added a comment - markewaite Ah, I see. Thanks! Having an automatic cache for git repositories would be totally cool for Multi-branch pipelines. Last time I checked this project type (a long time ago), it created a new clone for each branch, which is not good for large repositories. Maybe it's been fixed somehow, dunno. Would be nice to have it as well.
This issue is unassigned right now. Do I have any chance to have it fixed soon? I won't be working on this for a long time. My focus is on evaluating and including the pull requests submitted by others, while tracking bug reports and resolving bugs.
Anna Tikhonova
added a comment - markewaite I have an initial implementation of what you suggested. It's very short and simple. It works with certain limitations/assumptions (single SCM is used, all nodes are online, slave workspaces do not exist before the first run, etc).
I've extended GitSCM to use the same caches implemented in GitSCMSource for GitSCMFileSystem on master and manage similar caches on all slaves. Could you please have a look at it and advise me on improvements, so that we can have it in the upstream someday?
Surely I will remove the limitations. What bothers me is architectural decisions. E.g sharing caches with GitSCM might require refactoring the cache implementation out of AbstractGitSCMSource, which means some invasion into GitSCMFileSystem. It's not absolutely necessary because of the implementation details (cache entry is calculated from remote URL). But it might be a better implementation architecture wise. I'm willing to do it the best way.
The implementation: https://github.com/atikhono/git-plugin
Anna Tikhonova
added a comment - markewaite I have an initial implementation of what you suggested. It's very short and simple. It works with certain limitations/assumptions (single SCM is used, all nodes are online, slave workspaces do not exist before the first run, etc).
I've extended GitSCM to use the same caches implemented in GitSCMSource for GitSCMFileSystem on master and manage similar caches on all slaves. Could you please have a look at it and advise me on improvements, so that we can have it in the upstream someday?
Surely I will remove the limitations. What bothers me is architectural decisions. E.g sharing caches with GitSCM might require refactoring the cache implementation out of AbstractGitSCMSource, which means some invasion into GitSCMFileSystem. It's not absolutely necessary because of the implementation details (cache entry is calculated from remote URL). But it might be a better implementation architecture wise. I'm willing to do it the best way.
The implementation: https://github.com/atikhono/git-plugin
Originally, I asked about this behaviour here: https://issues.jenkins-ci.org/browse/JENKINS-28447
Was told to submit an RFE for git plugin.