-
Bug
-
Resolution: Fixed
-
Minor
-
Jenkins 2.16
Windows Server 2012 R2
-
-
git plugin 4.4.3
Description of the problem
I've a multibranch pipeline project configured with the local branch extension and many branch following the git flow model (features/xxxx). When I push a commit on the feature branch, this commit is detected by Jenkins, and pulled by the master to read the Jenkinsfile. The first time it works, but after that, the local branch is selected instead of the remote branch, amd the new commits are not fetch. The bug only occurs on branches with a slash in their names.
Investigation
I've connected my debugger to my Jenkins installation.
It appears that Jenkins has detected 2 branches matching the name. One local and one remote. The local one is created by the local branch git extension.
I can't explicitly specify to Jenkins to use the remote one because I use the pipeline plugin, and this checkout is made automatically to retrieve the Jenkinsfile. So, the git implementation takes the first one, which is generally the local branch which is not yet updated...
Revision marked = candidates.iterator().next();
Logs
16:10:31 Multiple candidate revisions 16:10:31 Checking out Revision 604398f062b459c797f31d6c13b568dd7612362d (features/my-feature-branch) 16:10:31 > git.exe config core.sparsecheckout # timeout=30 16:10:31 > git.exe checkout -f 604398f062b459c797f31d6c13b568dd7612362d # timeout=120 16:10:32 > git.exe branch -a -v --no-abbrev # timeout=30 16:10:32 > git.exe branch -D features/my-feature-branch # timeout=30 16:10:32 > git.exe checkout -b features/my-feature-branch 604398f062b459c797f31d6c13b568dd7612362d # timeout=120 16:10:32 > git.exe rev-list 0fa6b0371aa1e77a35acdc9f82bb68bb2c32fb2d # timeout=30
How to fix it
Possibilities I see :
- Update the workflow-multibranch-plugin / branch-api-plugin to checkout on origin (prefix with refs/remotes/origin/)
- Update the git-plugin to order the list, and prioritize remote branches
Personally, I think the first one is safer, but I've not yet played with this plugin.
Here is my proposition on Github: https://github.com/jenkinsci/branch-api-plugin/pull/47
How to reproduce
- Create a new repository and initialize it (on github, with a README.md for example)
- Clone it on your computer:
git clone https://lausivoiduts.visualstudio.com/QuarahMC/_git/JENKINS-37263 cd JENKINS-37263
- Create a new branch:
git checkout -b features/this-bug
- Add something in the Jenkinsfile:
echo "echo 'first commit' " > Jenkinsfile
- Commit and push it:
git add Jenkinfile git commit -m "first commit" git push --set-upstream origin features/this-bug
- Add a new Multibranch Pipeline Project in Jenkins, add as Source your git repository.
Click on "Add Behaviour", select "Checkout to a local branch".
Set its value to "**" (without the quotes).
Validate it. - Check that the branch was built, and the message "first commit" appears in the log
- Modify your Jenkinsfile:
echo "echo 'second commit' " > Jenkinsfile
- Commit and push it:
git add Jenkinfile git commit -m "second commit" git push
- Run the branch indexing
- Check that the branch is built. Check the log, you'll see "first commit" instead of "second commit".
- is duplicated by
-
-
- Open
-
[JENKINS-37263] Jenkins checkout the wrong commit when used with the local branch behaviour on a branch with a / (slash)
I'm prone to prefer the second approach rather than the first (change the order of evaluation in the git plugin to look for more specific matches before it looks for less specific matches, then take the first match). To retain best compatibility, we could do that only if the LocalBranch extension was enabled on the running job.
Quentin Dufour
added a comment - A huge thanks for your reply markewaite. I agree that modifying the git plugin would be more universal.
If you don't have time, I can try to work on a fix this week-end, as usual no promise, I'll do my best.
Quentin Dufour
added a comment - A huge thanks for your reply markewaite . I agree that modifying the git plugin would be more universal.
If you don't have time, I can try to work on a fix this week-end, as usual no promise, I'll do my best. superboum I definitely will not be working on it this weekend.
This weekend will be focused on credentials 2.1 support in the git plugin (as added in a pull request 412, but showing a problem with scp syntax URL's) and on supporting the command line git implementation in pipeline branch indexing (as added in a pull request 424, but showing a problem in one of the use cases I've been testing).
I probably won't look at this bug until after mid-September, since I need to complete those two items, release git plugin and git client plugin versions with those capabilities, then shift to preparing the release of git plugin 3.0 and git client plugin 2.0. Those major releases add support for submodule authentication (JENKINS-20941, the most requested enhancement to the git plugin), switch from JGit 3 to JGit 4, and move the minimum Jenkins version to 1.625 for JDK 7 support (try with resources, etc.).
This is not fixed in git plugin 2.6.0, released 2 Sep 2016. It worked for me occasionally in my test environment, but there is something wrong with the job or how it is behaving. It fails my tests at least every second test.
superboum could you check it with that release in your environment?
Quentin Dufour
added a comment - I'm not sure to understand what you expect of me. You want that I reproduce the bug with the new version of the plugin without any patch ? While awaiting your answer, I'll do that.
I've also opened a pull request here with a non complete proposed fix, but waiting for your or others feedbacks: https://github.com/jenkinsci/git-plugin/pull/435
Quentin Dufour
added a comment - I'm not sure to understand what you expect of me. You want that I reproduce the bug with the new version of the plugin without any patch ? While awaiting your answer, I'll do that.
I've also opened a pull request here with a non complete proposed fix, but waiting for your or others feedbacks: https://github.com/jenkinsci/git-plugin/pull/435 I initially thought this might have been fixed by the call to prune being integrated into the fetch call (part of AbstractGitSCMSource in git plugin 2.6.0). I was wrong. I suggested you check it confirm that the problem is not fixed in the latest plugin release.
I'm shifting my focus for the next 1-2 weeks to prepare the release of git client plugin 2.0 and git plugin 3.0 (submodule authentication, switch to JDK 7, etc.). I probably won't evaluate PR435 until some time after the release of those two versions.
Quentin Dufour
added a comment - I've setup a fresh Jenkins instance (v2.7.3) on Linux with the recommended plugins.
I reproduced the steps above, but I couldn't reproduce the bug.
I tried with the following behaviours activated :
- Nothing
- Clone to a local branch '**'
- Clone to a local branch '**' + checkout timeout + clone timeout + clean before checkout
But the fact that I didn't succeed to reproduce the bug doesn't necessarily mean that this bug is totally fixed.
Quentin Dufour
added a comment - I've setup a fresh Jenkins instance (v2.7.3) on Linux with the recommended plugins.
I reproduced the steps above, but I couldn't reproduce the bug.
I tried with the following behaviours activated :
Nothing
Clone to a local branch '**'
Clone to a local branch '**' + checkout timeout + clone timeout + clean before checkout
But the fact that I didn't succeed to reproduce the bug doesn't necessarily mean that this bug is totally fixed. superboum thanks very much for investigating. I won't be able to do any further checks until after 16 Sep 2016. I think I saw the problem at least once with my docker instance and its verification jobs, but will need further investigation to confirm the conditions.
Josh Ancill
added a comment - This is still a problem with Jenkins 2.25 with up to date plugins as of October 24th 2016. Using the git branch naming scheme: feature/new-feature fairly reliably causes the issue to occur more often than not.
Josh Ancill
added a comment - This is still a problem with Jenkins 2.25 with up to date plugins as of October 24th 2016. Using the git branch naming scheme: feature/new-feature fairly reliably causes the issue to occur more often than not. 
Anne van der Bom
added a comment - Just a confirmation that the error still persists in version 2.32.3.
Anne van der Bom
added a comment - Just a confirmation that the error still persists in version 2.32.3. We're having the same problem on our Jenkins system too. We have the same configuration of checking out to a local branch. The logs show the same "Multiple candidate revisions" message, and then it picks the revision that's currently checked out.
This doesn't happen with every branch. I assume this is because the order of the revisions is either random or determined by some sort of ordering that can put either the remote or local first, depending on something like the branch name. So sometimes it picks the remote branch and sometimes it picks the local.
I was having this issue when using "origin" as the "Name" of my git repository in the job's settings, and then using e.g. "abc/def" as my branch specifier. Removing the Name and instead using "origin/abc/def" as my branch specifier fixed the issue.
This problem occurs sometimes because the local and remote branches are collected in a HashSet of candidate revisions. Depending on sha1 of corresponding revisions, the first candidate that will be picked can be a local or a remote branch.
I submitted a pull request https://github.com/jenkinsci/git-plugin/pull/942 with a fix and tests that reproduce the problem.
I confirmed this is behaving as recena described (ugh, it's a bug). Refer to jenkins-bugs/JENKINS-37263 for the branch containing the Jenkinsfile I used to verify the problem. That repository is used in a multi-branch pipeline job defined in my lts-with-plugins docker image.
With initial experiments, it seems that one temporary work-around may be to enable the "Wipe workspace" additional behavior. My experiments seemed to show that it would consistently checkout the correct SHA1 on the first build in a workspace.