pipeline job hangs forever at checkout GitSCM - after git rev-list (sporadic)

Sometimes a pipeline job hangs around checkout. 

Lately this seems to be after git rev-list. 

 > git rev-list --no-walk db25209b84b12cca5214227a7999fe280c82017e # timeout=10

It will hang here until job is killed. A restart is fine - and it only happens occasionally. 

This has been more commonly reported since 2018 February/January, a recent update. 

A support bundle of this when it was happening: support_2018-02-07_22.54.25.zip

IT was happening on slave-05 at the time (can see in its thread dump a few waiting things). Timeout does not work, it will wait indefinitely. 

My pipeline job is getting sporadically hung on checkout step.

Our git repo is relatively small and clone should not take more than 1 min.

The problem occurring really often once a couple builds. Once it's occurring, the only way that I know to get rid of this state so far is, is disconnect the slave and re-connect it back.

Build log:

Started by user <USER>
Checking out git <REPO> to read Jenkinsfile.bootstrap
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url <REPO> # timeout=10
Fetching upstream changes from <REPO>
> git --version # timeout=10
using GIT_SSH to set credentials jenkinsslave SSH key
> git fetch --tags --progress <REPO> +refs/heads/:refs/remotes/origin/
> git rev-parse refs/remotes/origin/devops_jenkins^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/devops_jenkins^{commit} # timeout=10
Checking out Revision 2fd998fbe8b23f814b57d8c6eb7b46004268eedc (refs/remotes/origin/devops_jenkins)
> git config core.sparsecheckout # timeout=10
> git checkout -f 2fd998fbe8b23f814b57d8c6eb7b46004268eedc
> git rev-list 2fd998fbe8b23f814b57d8c6eb7b46004268eedc # timeout=10
[Pipeline] timestamps
[Pipeline] {
[Pipeline] node
00:00:01.249 Running on <SLAVE> in <WORKSPACE>
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Load Jenkinsfile)
[Pipeline] echo
00:00:01.267 Branch: origin/devops_jenkins Repo: <REPO>
[Pipeline] checkout
00:03:27.565 > git rev-parse -is-inside-work-tree # timeout=10 <- This and further lines are being printed only after slave disconnect.
00:03:27.566 Fetching changes from the remote Git repository
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] End of Pipeline
hudson.remoting.RemotingSystemException: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:272)
at com.sun.proxy.$Proxy98.setRemoteUrl(Unknown Source)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl.setRemoteUrl(RemoteGitImpl.java:295)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:794)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1070)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1101)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:83)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:73)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:221)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Backing channel 'jenkinsslave_192.168.45.26' is disconnected.
at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:191)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:256)
... 16 more
Caused by: hudson.remoting.Channel$OrderlyShutdown
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1121)
at hudson.remoting.Channel$1.handle(Channel.java:526)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:83)
Caused by: Command close created at
at hudson.remoting.Command.<init>(Command.java:59)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1115)
at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:1113)
at hudson.remoting.Channel.close(Channel.java:1273)
at hudson.remoting.Channel.close(Channel.java:1255)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1120)
... 2 more
Finished: FAILURE