annetheagile this is not yet implemented in any official release of the Jenkins git plugin. It is a large enough change that I wanted an extended beta period to test it. Breaking the authentication techniques of a plugin used in 60 000 installations was too scary for me to not want an extended beta period.

I'm not sure why you say that the "2.5 experimental patch ... is no longer on the beta". I just configured a fresh installed Jenkins to use the experimental plugins update center and confirmed that git plugin 2.5.0-beta4 is available there, as is git client plugin 1.20.0-beta3.

The git plugin changelog includes references to git plugin beta versions 2.5.0-beta1, 2.5.0-beta2, and 2.5.0-beta3. Sorry, I forgot to publish an update to describe 2.5.0-beta4 on the wiki.

The git client plugin changelog includes references to git client plugin beta versions 1.20.0-beta1 and 1.20.0-beta3.

Could you try the experimental update center to see if it shows you the same thing it shows me?

Why not add a Checkbox on Repository Clone config asking if we want a recursive clone?

driverpt I don't know why we would add a repository clone checkbox for recursive submodule clone when there is already a checkbox for recursive clone in the "Advanced submodule behaviours" section. Can you explain further where you mean by "Repository Clone config"?

Refer to

markewaite

The problem is that the "Advanced submodule behaviours" is not integrating properly with the Credentials Plugin, meaning it does not use the Credentials for the --recursive update modules. And we don't upload Private Keys to Slave machines.

driverpt If you want to use credentials with submodules, you need to use the experimental update center with the git plugin 2.5.0 beta and the git client plugin 1.20.0 beta. That has nothing to do with the location of the "recursive update submodules" checkbox as far as I can tell.

Refer to older postings in this bug report for the details of how to install those two beta plugin versions.

I was just making a suggestion to add the "--recursive" checkbox below the Repository URL so that the SubModules are cloned recursively on the inital clone and not a Post-Action.
You have any ETA on the Git Plugin 2.5.0 Final and Git Client Plugin 1.20.0 Final ?

I'm not sure what you mean by "cloned recursively on the initial clone and not a Post-Action". The initial clone (fetch) brings in the remote repository and performs a checkout of the selected branch. It can't know the submodules being referenced until after the selected branch checkout has completed. Once the initial branch checkout is complete, then the submodules are cloned.

All those steps happen in the "checkout" of the initial repository, without any "post-action" as far as I know.

markewaite
What i'm trying to explain is that the git clone is executed with the Credentials, but the git submodule update --recursive doesn't use Credentials, therefore causing our builds to fail.

Today I am using SSH Agent Plugin, which works with Git Plugin both for clone and submodule as is.

Does this new beta feature removes the need for SSH Agent Plugin in that case?

Yes, the beta plugin does not require ssh-agent.

Hi lkraider,

I am struggling with the same issue. I tried the SSH Agent Plugin, setting private key. The parent repository is still checked out as usual but it seem to have done nothing for the submodules. Do you have any advice?
Thanks

Our environment is OSX master and slave.

I compiled SSH Agent plugin to enable the native ssh-agent with this patch: https://github.com/jenkinsci/ssh-agent-plugin/pull/2
The credentials provided work for both the main and submodule clone operations. In my setup the credential is a single SSH key that has read access to the main and all submodule repos. I have not tested if you need to unlock multiple keys.

Thanks a lot lkraider. I will give that a try.

The 2.5.0-beta3 plugin is not working for me. Switching to experimental plugins, and downloading and restart shows no errors in the startup log.
This is on a clean 1.642.3 installation.

Two problems that I see:

1) Corrupted jobs after restart. Any prior job will get an exception in the configure page.

Caused by: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
jar:file:/var/lib/jenkins/plugins/git/WEB-INF/lib/git.jar!/hudson/plugins/git/extensions/impl/SubmoduleOption/config.groovy: 52: expecting '}', found '' @ line 52, column 1.

2) Creating a new job from scratch does not allow me to set sub-module behavior. The "Additional Behaviors" Add-> pop down, then choosing "Advanced sub-module behaviors" doesn't do anything. No sub-pane is added to allow me to check "recursive".

The log show the same exception.

Manually downloaded and manually installed the 2.5.0-beta4 plugin, and get the same exception.

Top few lines:

Apr 15, 2016 8:21:36 PM WARNING hudson.widgets.RenderOnDemandClosure$1 generateResponse
Failed to evaluate the template closure
org.apache.commons.jelly.JellyTagException: jar:file:/var/cache/jenkins/war/WEB-INF/lib/jenkins-core-1.642.3.jar!/lib/form/hetero-list.jelly:131:100: <st:include> org.kohsuke.stapler.ScriptLoadException: org.apache.commons.jelly.JellyException: Failed to load /hudson/plugins/git/extensions/impl/SubmoduleOption/config from org.kohsuke.stapler.jelly.groovy.GroovyFacet@5c993a07
	at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:726)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:281)

bo3bber My apologies! You're absolutely correct. The beta4 build of the plugin won't allow the user to set the submodule options at all. I'll need to research why it is failing and deliver a new beta release to fix the problem.

I've released git plugin 2.5.0-beta5 to fix the problem. I ran some basic interactive regression tests with it and confirmed that it correctly passes credentials from the parent project to the submodule for ssh based credentials.

markewaiteThanks! Really appreciate the fast turnaround for this problem.

I confirm that the 2.5.0-beta5 fixes the exception in my use case.

I also confirm that once I set the Recursively update submodules and Use credentials from default remote of parent repository settings in the Advanced sub-modules behaviors pane, that I can also successfully clone my repot with supporting sub-module, with SSH key authentication.

Using git:2.5.0-beta5 and git-client:1.20.0-beta3

I've looked in http://mirror.xmission.com/jenkins/updates/experimental/update-center.json for git:2.5.0-beta5, but all I see is git:2.5.0-beta4. Is there some other place I can look for it? Do I just need to wait for it to propagate to other mirrors?

That's a different URL than I'm accustomed to using for the experimental update center.

If you don't mind downloading it directly from artifactory (and then uploading it using the Advanced page of "Manage plugins"), you can get it at https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/git/2.5.0-beta5/git-2.5.0-beta5.hpi

We have Jenkins ver. 1.596.2 with plugins:

• git 2.5.0-beta5
• git-client 1.20.0-beta3
  Unfortunately, we still see the error when retrieving submodules.:

  C:\git\bin\git.exe config --get remote.origin.url # timeout=10
  > C:\git\bin\git.exe config --get-regexp ^submodule # timeout=10
  > C:\git\bin\git.exe config --get submodule.helloivy.url # timeout=10
  > C:\git\bin\git.exe submodule update --init --recursive helloivy
  FATAL: Command "C:\git\bin\git.exe submodule update --init --recursive helloivy" returned status code 1:
  stdout:
  stderr: Cloning into 'helloivy'...
  Permission denied (publickey).
  fatal: Could not read from remote repository.

adf2112, the same credentials are used for the parent repository and the submodule repositories. That generally means you need to use the same protocol for the parent repository and the submodules. For example, if the parent is git@github.com:MarkEWaite/git-plugin.git (ssh protocol), then the submodules cannot be https://github.com/MarkEWaite/git-client-plugin.git, they must also be ssh protocol.

Are you using the same protocol for parent repository and submodule repositories?

Do the the parent repository credentials work for the submodule repositories outside the Jenkins job?

Thank you for the quick response!
Yes, we are using SSH in the git plugin settings referencing a global credentials store in Jenkins/Manage/Credentials (username/key).
SSH works fine with the git plugin except when we attempt to retrieve submodules. We have the "Recursively update submodules" checkbox checked.
I can do this no problem using command line git....the submodules are resolved ok.
Any ideas?

I didn't get from your response that you had actually checked that the same protocol is used in the submodules as is used in the parent. Can you check that?

Hi
I've got the same issue

Cloning repository ssh://root@192.168.0.20:22/srv/repos/git/root-module.git
Clone of 'ssh://root@XXX.XXX.XXX.XXX:YYYYY/srv/repos/git/sub-module.git' into submodule path 'marksman-c' failed
Permission denied (publickey,password).

I think plugin doesn't use the original credentials

BR
AlexZ

P.S.

I use ssh user name with private key authentication
Scope is Global

git-client-plugin 1.20.0-beta3 works fine for me.

However I can't find the way to set the "Use credentials from default remote of parent repository" option from a DSL script, I have to alter the jobs manually after they've been generated.

It would be nice to be able to set it with a parentCredentials(true) method call inside the job.scm.git.extensions.submoduleOptions section.

I'm still having trouble with this bug even with the 1.20.0-beta3 update.
I'm using an http repo with a username and password login. It looks like it's trying to use the credentials but still fails:

using GIT_ASKPASS to set credentials Gitlab Jenkins User
> git submodule update --init --recursive libraries
FATAL: Command "git submodule update --init --recursive libraries" returned status code 1:
stdout:
stderr: Cloning into 'libraries'...
Permission denied (publickey).
fatal: Could not read from remote repository.

captrespect have you confirmed that all the authenticated submodules in the repository use the same protocol (https) as the parent repository? One of the limitations of the current technique is that all authenticated submodules must use the same authentication technique as the parent.

Yes, thanks that looks like my issue. Any easy way to override the url of the submodule for this jenkins instance?

Nevermind! I was able to update the the submodule path in .gitmodule a relative path to get them to clone.

Hi,

I noticed that if the .gimodules submodule name is not the same as the path spec the submodule update fails by trying to use the name.

submodule defined like this fails
[submodule "rpm"]
path = linux/rpm
url = ../rpm.git

with error
> git config --get submodule.rpm.url # timeout=60
> git submodule update --init --recursive rpm # timeout=100
FATAL: Command "git submodule update --init --recursive rpm" returned status code 1:
stdout:
stderr: error: pathspec 'rpm' did not match any file(s) known to git.
Did you forget to 'git add'?

the workaround is to name the submodule the same as the path
[submodule "linux/rpm"]
path = linux/rpm
url = ../rpm.git

using
git 2.5.0-beta5
git-client 1.20.0-beta3

We are having this issue using SSH for the primary project and the submodules. I have not tried the beta plugin as the description above indicates that the issue is related to http git links. Does this fix include a fix for refreshing the submodules for SSH connections?

habbie the beta version of the plugin supports https (username/password) parent repositories referencing https (username/password) submodules and it supports ssh (private key) parent repositories referencing ssh (private key) submodules. I've recently checked that it works with a github ssh (private key) parent repository and github ssh (private key) submodules.

Guys, do someone have any information about possible date of solution ?

chypalex can you further describe what you mean by "possible date of solution"?

If you want to use credentials with submodules today, you can use the Jenkins experimental update center to install the beta versions of the git plugin and the git client plugin.

Update:
submodules are working for us now with the beta release.

Sorry - I did not have the: "Use credentials from default remote of parent repository" checked before.

Thank you!

Thanks markewaite. It looks like the Wiki is having issues:
https://wiki.jenkins-ci.org/pages/viewpage.action?pageId=99057971

Just looking for an update and if I can get notification when 1.20.0 will be released to production.

Thanks!

That has been a relatively common problem with that wiki page. The solution I've been told to use is to add a parameter to the end of the URL. I used https://wiki.jenkins-ci.org/display/JENKINS/Git+Client+Plugin?foo=bar to read the correct page.

The git client plugin 2.0.0-beta1 has been released to the experimental update center. It includes git submodule authentication, JGit 4.3, and requires JDK 7. It requires at least Jenkins 1.625 (the first version to mandate JDK 7).

The process of preparing that release (with JGit 4.3) exposed an incompatibility which I haven't yet decided how to resolve. The incompatibility is described in a posting to the Jenkins developers mailing list. I need some more time to choose the best compromise. I really want JDK 7 because it makes the git client plugin cleaner and less susceptible to file leaks. I also want JGit 4, because JGit 4 is receiving active development, while JGit 3 is not.

FYI, very sadly, the ability to select advanced git behaviors in multibranch pipeline projects is gone again after upgrade from 2.5.2 to 3.0.0-beta1 of the git plugin. I was assuming 3.0.0-beta1 is necessary to go with 2.0.0-beta1 of the git client plugin. After I downgraded to git plugin 2.5.2 again, but left git client 2.0.0-beta1 in, submodule (public key) authentication on the agents still doesn't seem to work even though the main repository is checked out just fine

> git submodule update --init --recursive XXX
hudson.plugins.git.GitException: Command "git submodule update --init --recursive XXX" returned status code 128:
stdout: 
stderr: Cloning into 'XXX'...
Permission denied, please try again.
Permission denied, please try again.
Permission denied (publickey,password).
fatal: Could not read from remote repository.

P.S. I was not able to find "Use credentials from default remote of parent repository" under "Additional Behaviours", maybe for that git plugin 3.0.0-beta1 is needed, but see above: if I install it, then "Additional Behaviours" vanish from the multibranch pipeline job scm settings.

zaytsev_work thanks for detecting that, and thanks for reporting it.

Git plugin 3.0.0-beta1 was released a month ago (changelog). Git plugin 2.5.1 (followed by 2.5.2) was released a few days ago (changelog) with the advanced git behaviors in multibranch pipeline projects (and in literate projects). A new release of git plugin 3.0.0-beta will be needed before the advanced git behaviors will be visible in multibranch pipeline projects with a plugin version that supports submodule authentication.

I hope to release that new version of git plugin 3.0.0-beta within the next 12 hours.

markewaite, I've just upgraded to 3.0.0-beta2 and "Additional Behaviors" are back for the multibranch pipeline jobs, also the "Use credentials from default remote of parent repository" checkbox is there and I can confirm that it seems to be working fine for me now! Many thanks and keep up the excellent work

Hi all,

As of today with official plugins the issue is still outstanding. So no way to work it around. You will have to recur to the beta plugins of both:

• Git plugin
• Gut client plugin

Steps to achieve this are:

• enable experimental plugins, process https://jenkins.io/blog/2013/09/23/experimental-plugins-update-center/
• go to "Manage Jenkins" >> "Plugin Manager" >> "Updates" and then press "Check Now"
• you should something like below:

Download the new experimental plugins, and restart.

In the git "Advanced Submodules Behaviour" block tick the options like below:

Happy days

Any chance this will ever be released? The way I see it:

• It's been in QA for nearly a year with some willing individuals reporting back it works for them using the beta builds.
• Meanwhile, the active development on the affected plugin(s) continues.
• The longer this stays in QA, regular maintenance is required to occur over these changes' lifetime due to upstream active development.

Is anyone willing to pull the trigger and say it's ready? If not, should it be abandoned and re-attempted if it's no longer relevant to upstream changes? I'd hate to see this major feature continue to be broken in Jenkins. I feel like this behavior is git 101. It should work but doesn't in Jenkins (omitting any technicalities in actual implementation).

Also, I feel the priority should be increased to the highest priority rather than just major. I'll leave that to someone else to agree/disagree and change the priority. Not being able to submodule using built-in features makes Jenkins feel like a rough product than the real quality rating it deserves.

My goal is to release it in time for Jenkins World (Sep 13-15, 2016), in combination with the switch of the git client plugin and git plugin to Java 7. That will require a minimum version of Jenkins 1.625, and requires some detailed review of other plugins since it also includes an update from JGit 3 to JGit 4.

The update from JGit 3 to JGit 4 simplifies the git client plugin by allowing it to use Java 7 "try with resources". It (unfortunately) also requires that dependent plugins either upgrade to handle the change, or adapt to allow them to be used with either JGit 3 or JGit 4. One coexistence technique is now included in git plugin 2.5.3. Refer to commit 71946a2 for the details.

The maintenance is already happening to regularly merge the git plugin master branch to the 3.0.0-beta branch. Likewise, the maintenance is already happening to regularly merge the git client plugin master branch to the 2.0.0-beta branch.

@Mark Waite, your suggestion to me on 19 April worked. Thank you.

Thanks for the update, Mark.

The version 3.0.0-beta2 did the trick, thank you.
Can somebody help me please to use the new features in a "pipeline script"?

I do the checkout in following way:

git branch: "master", credentialsId: '<ID>', url: '<URL>'

How can i use the checkboxes "Update submodules..." and "Use credentials from parent.." in this pipeline script?

Instead of using the `git` step, you need to use the `checkout` step.

I create my pipeline definitions by creating a "disposable" pipeline job ("Pipeline" in the Jenkins 2 "New Item" choices). Inside that job, there is a "Pipeline Syntax" hyperlink. I click that and choose "checkout" from the drop down list. Then when I choose "Git" as the SCM, there is an "Add" button which shows all the additional behaviors. Submodule authentication settings are available there, and the resulting pipeline script prototype will be written to the window so that you can cut and paste it as needed in your pipeline script.

markewaite thank you so much! i'm happy!

I'm testing with the 3.0.0 beta2 plugin and encountering an error. In particular this is the error reported:
hudson.plugins.git.GitException: Command "git submodule update --init --recursive php-libraries" returned status code 1:
stdout:
stderr: error: pathspec 'php-libraries' did not match any file(s) known to git.

Now, if I run the same command as stated here from my local git I get the same error, but I can run git submodule init and git submodule update without any issues.

My git submodule status shows the submodule in question (though with its path, not its submodule name:
git submodule status
4f27645ae23a3955d72043afffb75a34449bbbaf var/www/members/libraries (heads/release/v1.0)

My .gitmodules includes the following, which is correct:
[submodule "php-libraries"]
path = var/www/members/libraries
url = http://blades:7990/scm/phpapi/php-libraries.git
branch = release/v1.0

And my .git/config file contains this reference, which seems correct:
[submodule "php-libraries"]
url = http://blades:7990/scm/phpapi/php-libraries.git

Running this from my local git client; however works:
git submodule update --init --recursive var/www/members/libraries

So, while not an expert, it seems that the git submodule command is expecting the path, but instead what is being passed to it is the name of the submodule. I'm not sure if the name should work as well, but at least in my case it is not.

If I arrived at an incorrect conclusion please fill me in on where I went wrong.

Thanks markewaite. At least I know I'm not losing my mind. In the meantime I suppose I can just rename my submodule(s) to work around the issue.

Git plugin 3.0.0 and git client plugin 2.0.0 were released 10 Sep 2016 to resolve this issue. Special thanks to Jacob Keller for his patience with a one year beta test, and thanks to those users who performed the beta testing.

Git plugin 3.0.0 and git client plugin 2.0.0 require a minimum of Jenkins 1.625 and JDK 7.

Anybody able/getting this to work? I haven't been (user/password) using what I believe are the latest versions :| https://gist.github.com/rdp/291bda289a94eae96fc32a30e6db9497

rogerdpack yes, it is working fine for me. I installed the 3.0.0 version of the plugin using the Plugin Manager in Jenkins, nothing special. (Note: You have to add the Advanced sub-modules behavior to your SCM config and check the box to "Use credentials from default remote of parent repository".)

Code changed in jenkins
User: Jacob Keller
Path:
src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
http://jenkins-ci.org/commit/git-client-plugin/deaf40377a70f9b6995344fbc54a32e29c61098a
Log:
git-client: teach submoduleUpdate how to pass credentials

Teach the submoduleUpdate call to perform submodule URL lookup, and call
"submodule update" separately for each submodule. Lookup the
credentials, and call it using launchCommandWithCredentials. This
enables submodules to correctly get the SSH or HTTP setup necessary such
that the Jenkins credentials will be passed into each submodule.

We can't just call "git submodule update" since it may be possible
(however unlikely!) that each submodule wants to use a separate
credential. Thus, perform lookup using each URL and run through a
forloop to actually update each submodule instead of depending on the
git implementation to do this for us.

JENKINS-20941 - Credentials and Submodules

Signed-off-by: Jacob Keller <jacob.e.keller@intel.com>

creste3, can you explain why you assigned yourself as owner of this resolved bug?

Is there some problem that you intend to investigate or resolve?

Sorry, I definitely did not intend to assign myself as the owner of this bug. I suspect that I accidentally pressed one of the JIRA shortcut keys to assign this to myself. I have since disabled keyboard shortcuts because it appears that I am too clumsy to have them enabled.

There is a problem related to git credentials and submodules that I intend to resolve, but that work should be done in a separate issue because it is related to git LFS.