I have yet to have a reliable production, and I am not sure if the bug is in Multiple SCM or within the Git plugin.

I have 3 repositories, one checked out at the tip of workspace, and two checked out into subdirectories src/A and src/B.

Sometimes the src/A or src/B fail to be checked out, despite the log looking perfectly normal, and saying it cloned and checked them out.

Sometimes re-running the job simply fixes the issue.

A sample of the log that Isee when this issue occurs:

Started by upstream project "ixgbe/gerrit_compat" build number 17
originally caused by:
Manually triggered by user jekeller for Gerrit: https://git-amr-1.devtools.intel.com/gerrit/74271
Building remotely on rhel5sp11-60c709d8 (rhel5sp11 swarm linux rhel5) in workspace /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5
Cloning the remote Git repository
Cloning repository ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat
> git init /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src/COMPAT # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat
> git --version # timeout=10
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat +refs/heads/:refs/remotes/origin/
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat # timeout=10
> git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-compat refs/changes/71/74271/1
Checking out Revision 73f1f63fb1f4c5da8030d46b3cdf762cefc28e10 (master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 73f1f63fb1f4c5da8030d46b3cdf762cefc28e10
First time build. Skipping changelog.
Cloning the remote Git repository
Cloning repository ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe
> git init /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src/SHARED # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe
> git --version # timeout=10
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe +refs/heads/:refs/remotes/origin/
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe # timeout=10
> git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_shared-ixgbe +refs/heads/:refs/remotes/origin/
Checking out Revision c0443ba24e527146ce7e8a331047aff366aaa578 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f c0443ba24e527146ce7e8a331047aff366aaa578
First time build. Skipping changelog.
Cloning the remote Git repository
Cloning repository ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe
> git init /home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5 # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe
> git --version # timeout=10
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe +refs/heads/:refs/remotes/origin/
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe # timeout=10
> git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
> git config remote.origin.url ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe # timeout=10
Fetching upstream changes from ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe
using GIT_SSH to set credentials ND Linux CI Server
> git fetch --tags --progress ssh://jbrandeb-host.jf.intel.com/home/jenkins/mirrors/nd_linux-ixgbe +refs/heads/:refs/remotes/origin/
Checking out Revision 08bb35e104c5b440e20933cec05f1ed91abcd6b0 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 08bb35e104c5b440e20933cec05f1ed91abcd6b0
First time build. Skipping changelog.
[rhel5] $ /bin/bash -el /tmp/hudson1549523813705894193.sh
+ set -o pipefail
++ echo ixgbe/gerrit_compat/build=PROJECT,label=rhel5
++ sed s/label=//
++ sed s/BUILD_TYPE=//
++ tr /, -
+ project=ixgbe-gerrit_compat-build=PROJECT-rhel5
++ echo ixgbe-gerrit_compat-build=PROJECT-rhel5-17
++ tr '[:upper:]' '[:lower:]'
+ name=ixgbe-gerrit_compat-build=project-rhel5-17
+ logfile=ixgbe-gerrit_compat-build=project-rhel5-17.log
+ [[ PROJECT = ESX* ]]
+ which sparse
which: no sparse in (/usr/kerberos/bin:/sbin:/usr/sbin:/bin:/usr/bin:/home/jenkins/bin)
+ C=0
+ CFLAGS='-Wno-missing-field-initializers -Wno-aggregate-return'
+ CHECKIDS_FLAGS=
+ W=1
+ make -C src -f build.mk TEST_BUILD=YES USE_ISYSTEM=YES CHECKIDS_FLAGS= BUILD=PROJECT
+ tee ixgbe-gerrit_compat-build=project-rhel5-17.log
make: Entering directory `/home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src'
build.mk:61: *** Cannot find COMPAT/compat.inc for kcompat flags. Stop.
make: Leaving directory `/home/jenkins/workspace/ixgbe/gerrit_compat/build/PROJECT/label/rhel5/src'
Build step 'Execute shell' marked build as failure
[WARNINGS] Skipping publisher since build result is FAILURE
Archiving artifacts
Finished: FAILURE

As you can see, 3 repositories are "checked out" according to the docs, but the workspace for this job appears to only include the nd_linux-i40e project, and not the nd_shared-i40e or nd_linux-compat projects.

This results in a compilation failure due to inability to locate the COMPAT files.

What are the next steps I should take to help debug this issue?