-
Bug
-
Resolution: Duplicate
-
Minor
-
Debian 8.8. Jenkins 2.60. Tomcat 8.0.14.
Pipeline 2.5, Pipeline: Groovy 2.38, Git plugin 3.5.1, Lockable Resources 2.0.
We have a linux master (4 executors, never full) running a number of pipeline jobs that all pull Jenkinsfile from git before triggering the actual build on one of a pool of a half dozen workers (osx, linux, and windows). Most builds (dozens a day) work fine, but occasionally (1-3 times a week) things will hang before actually starting to run any Groovy.
This has been happening for months now, and various packages have been upgraded in response to no avail.
Initially, I thought that the issue was network connectivity to the git server - but I have since pulled the pipeline script onto a local file:// repo and experience the same issue.
When affected, builds look like this:
Started by upstream project "Project/Project-Sync" build number 1682
originally caused by:
Started by an SCM change
Checking out git file:///var/lib/jenkins/git-cache into /var/lib/jenkins/workspace/Project/Project-WIN@script to read Jenkinsfile
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url file:///var/lib/jenkins/git-cache # timeout=10
Fetching upstream changes from file:///var/lib/jenkins/git-cache
> git --version # timeout=10
> git fetch --tags --progress file:///var/lib/jenkins/git-cache +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision fd354d30a6b141d6fc81267d4708de10de5b5966 (refs/remotes/origin/master)
Commit message: "Typo."
> git config core.sparsecheckout # timeout=10
> git checkout -f fd354d30a6b141d6fc81267d4708de10de5b5966
> git rev-list fd354d30a6b141d6fc81267d4708de10de5b5966 # timeout=10
And then they just sit there like that forever.
A good goes more like this:
Started by upstream project "Project/Project-Sync" build number 1684
originally caused by:
Started by user Ammon Lauritzen
...
Checking out Revision fd354d30a6b141d6fc81267d4708de10de5b5966 (refs/remotes/origin/master)
Commit message: "Typo."
> git config core.sparsecheckout # timeout=10
> git checkout -f fd354d30a6b141d6fc81267d4708de10de5b5966
> git rev-list fd354d30a6b141d6fc81267d4708de10de5b5966 # timeout=10
[Pipeline] node
Running on master in /var/lib/jenkins/workspace/Project/Project-WIN
[Pipeline] {
[Pipeline] echo
prepping master environment
[Pipeline] sh
[Project-WIN] Running shell script
...
The only difference in the log is that it stops before the [Pipeline] lines start logging.
The build cannot be cancelled normally via the UI, by restarting Tomcat, or by rebooting the server - I have to instead abort it via the script console.
It happens more often with nightly builds than SCM-triggered CI builds, and more often on the weekends... but it happens plenty of times in the middle of the day as well. It is not always the same projects that hang, and not always projects destined for Windows slaves that hang (that was just the most recent instance of the problem).
- duplicates
-
-
- Closed
-
- links to
