Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Won't Fix
-
None
-
Jenkins 1.643 Linux
Multiple SCMs Plugin 0.5
Git Plugin 2.5.0-beta3
Description
I have yet to have a reliable production, and I am not sure if the bug is in Multiple SCM or within the Git plugin.
I have 3 repositories, one checked out at the tip of workspace, and two checked out into subdirectories src/A and src/B.
Sometimes the src/A or src/B fail to be checked out, despite the log looking perfectly normal, and saying it cloned and checked them out.
Sometimes re-running the job simply fixes the issue.
A sample of the log that Isee when this issue occurs:
Started by upstream project "ixgbe/gerrit_compat" build number 17
originally caused by:
Manually triggered by user jekeller for Gerrit: https://git-amr-1.devtools.intel.com/gerrit/74271
Building remotely on rhel5sp11-60c709d8 (rhel5sp11 swarm linux rhel5) in workspace /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5
Cloning the remote Git repository
Cloning repository ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat
> git init /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src/COMPAT # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat
> git --version # timeout=10
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat +refs/heads/:refs/remotes/origin/
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat # timeout=10
> git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat refs/changes/71/74271/1
Checking out Revision 73f1f63fb1f4c5da8030d46b3cdf762cefc28e10 (master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 73f1f63fb1f4c5da8030d46b3cdf762cefc28e10
First time build. Skipping changelog.
Cloning the remote Git repository
Cloning repository ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe
> git init /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src/SHARED # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe
> git --version # timeout=10
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe +refs/heads/:refs/remotes/origin/
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe # timeout=10
> git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe +refs/heads/:refs/remotes/origin/
Checking out Revision c0443ba24e527146ce7e8a331047aff366aaa578 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f c0443ba24e527146ce7e8a331047aff366aaa578
First time build. Skipping changelog.
Cloning the remote Git repository
Cloning repository ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe
> git init /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5 # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe
> git --version # timeout=10
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe +refs/heads/:refs/remotes/origin/
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe # timeout=10
> git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe +refs/heads/:refs/remotes/origin/
Checking out Revision 08bb35e104c5b440e20933cec05f1ed91abcd6b0 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 08bb35e104c5b440e20933cec05f1ed91abcd6b0
First time build. Skipping changelog.
[rhel5] $ /bin/bash -el /tmp/hudson1549523813705894193.sh
+ set -o pipefail
++ echo ixgbe/gerrit_compat/build=PROJECT,label=rhel5
++ sed s/label=//
++ sed s/BUILD_TYPE=//
++ tr /, -
+ project=ixgbe-gerrit_compat-build=PROJECT-rhel5
++ echo ixgbe-gerrit_compat-build=PROJECT-rhel5-17
++ tr '[:upper:]' '[:lower:]'
+ name=ixgbe-gerrit_compat-build=project-rhel5-17
+ logfile=ixgbe-gerrit_compat-build=project-rhel5-17.log
+ [[ PROJECT = ESX* ]]
+ which sparse
which: no sparse in (/usr/kerberos/bin:/sbin:/usr/sbin:/bin:/usr/bin:/home/jenkins/bin)
+ C=0
+ CFLAGS='-Wno-missing-field-initializers -Wno-aggregate-return'
+ CHECKIDS_FLAGS=
+ W=1
+ make -C src -f build.mk TEST_BUILD=YES USE_ISYSTEM=YES CHECKIDS_FLAGS= BUILD=PROJECT
+ tee ixgbe-gerrit_compat-build=project-rhel5-17.log
make: Entering directory `/home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src'
build.mk:61: *** Cannot find COMPAT/compat.inc for kcompat flags. Stop.
make: Leaving directory `/home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src'
Build step 'Execute shell' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Finished: FAILURE
As you can see, 3 repositories are "checked out" according to the docs, but the workspace for this job appears to only include the nd_linux-i40e project, and not the nd_shared-i40e or nd_linux-compat projects.
This results in a compilation failure due to inability to locate the COMPAT files.
What are the next steps I should take to help debug this issue?
Attachments
Activity
Craig Rodrigues
added a comment - Suggest that you migrate to Pipeline plugin, which offers a supported way of checking out from multiple scms
https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin
Craig Rodrigues
added a comment - Suggest that you migrate to Pipeline plugin, which offers a supported way of checking out from multiple scms
https://wiki.jenkins-ci.org/display/JENKINS/Pipeline+Plugin 
Terry Ewing
added a comment - I think this issue is the same as https://issues.jenkins-ci.org/browse/JENKINS-27668.
Terry Ewing
added a comment - I think this issue is the same as https://issues.jenkins-ci.org/browse/JENKINS-27668 . Can we get some input from developers of Git plugin and Multiple-SCM plugin?
The issue stems from ordering of repositories since the Git Plugin's "clone" operation calls a full delete of the current workspace. I have a possible solution revolving around making the Git Plugin report its actual sub directory using the getModuleRoot() interface, and then having Multiple-SCM "sort"/"order" the repositories in such a way as to avoid checking out into a sub directory first.
Another option would be for Multiple SCM to implement for itself "checkout to a sub directory" instead of relying on the plugins to do so themselves. I believe this would be preferred since it would be much more universal, and allow easier correct access since my fix requires both a change to the git plugin and the multiple SCM plugin.
Finally it may be possible to not need to remove the working directory when implementing clone() in the Git plugin. However, this requires continued use of the init+fetch setup, which currently breaks other uses such as Git-LFS and should be returned back to clone. There does not appear to be a good way to make git-clone behave correctly and allow cloning into a non-empty directory.
An alternative would be to implement getModuleRoots for the Git Plugin to return the actual root when Git is using the extension, and then order checkouts based on this. I am thinking this is simpler and will try to implement this.
Unfortunately this would still result in the same problem, because the git plugin wipes out the workspace when cloning, so it is still possible to have the issue occur. It is easier to avoid in the pipeline as long as the pipeline steps are ordered correctly.
The user just has to know and understand that the git plugin will wipe out the workspace path so you must ensure that the repositories are ordered correctly. In general, for pipeline this is obvious, but it is not obvious for the multi-scm plugin.