-
Bug
-
Resolution: Duplicate
-
Minor
-
Debian 8.8. Jenkins 2.60. Tomcat 8.0.14.
Pipeline 2.5, Pipeline: Groovy 2.38, Git plugin 3.5.1, Lockable Resources 2.0.
We have a linux master (4 executors, never full) running a number of pipeline jobs that all pull Jenkinsfile from git before triggering the actual build on one of a pool of a half dozen workers (osx, linux, and windows). Most builds (dozens a day) work fine, but occasionally (1-3 times a week) things will hang before actually starting to run any Groovy.
This has been happening for months now, and various packages have been upgraded in response to no avail.
Initially, I thought that the issue was network connectivity to the git server - but I have since pulled the pipeline script onto a local file:// repo and experience the same issue.
When affected, builds look like this:
Started by upstream project "Project/Project-Sync" build number 1682
originally caused by:
Started by an SCM change
Checking out git file:///var/lib/jenkins/git-cache into /var/lib/jenkins/workspace/Project/Project-WIN@script to read Jenkinsfile
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url file:///var/lib/jenkins/git-cache # timeout=10
Fetching upstream changes from file:///var/lib/jenkins/git-cache
> git --version # timeout=10
> git fetch --tags --progress file:///var/lib/jenkins/git-cache +refs/heads/*:refs/remotes/origin/*
> git rev-parse refs/remotes/origin/master^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
Checking out Revision fd354d30a6b141d6fc81267d4708de10de5b5966 (refs/remotes/origin/master)
Commit message: "Typo."
> git config core.sparsecheckout # timeout=10
> git checkout -f fd354d30a6b141d6fc81267d4708de10de5b5966
> git rev-list fd354d30a6b141d6fc81267d4708de10de5b5966 # timeout=10
And then they just sit there like that forever.
A good goes more like this:
Started by upstream project "Project/Project-Sync" build number 1684
originally caused by:
Started by user Ammon Lauritzen
...
Checking out Revision fd354d30a6b141d6fc81267d4708de10de5b5966 (refs/remotes/origin/master)
Commit message: "Typo."
> git config core.sparsecheckout # timeout=10
> git checkout -f fd354d30a6b141d6fc81267d4708de10de5b5966
> git rev-list fd354d30a6b141d6fc81267d4708de10de5b5966 # timeout=10
[Pipeline] node
Running on master in /var/lib/jenkins/workspace/Project/Project-WIN
[Pipeline] {
[Pipeline] echo
prepping master environment
[Pipeline] sh
[Project-WIN] Running shell script
...
The only difference in the log is that it stops before the [Pipeline] lines start logging.
The build cannot be cancelled normally via the UI, by restarting Tomcat, or by rebooting the server - I have to instead abort it via the script console.
It happens more often with nightly builds than SCM-triggered CI builds, and more often on the weekends... but it happens plenty of times in the middle of the day as well. It is not always the same projects that hang, and not always projects destined for Windows slaves that hang (that was just the most recent instance of the problem).
- duplicates
-
-
- Closed
-
- links to
[JENKINS-46503] Pipeline hanging after updating Jenkinsfile but before starting to build
Sverre Moe
added a comment - Running with downgraded Git plugin 3.6.4 and Git Client plugin 2.6.0 we are hardly experiencing the problem. It is far less frequent.
A build that didn't hang, where the git commit hash on rev-list where different from checkout.
> git checkout -f bc4dd5d1939f16e0124251327b0490a0059b353d
Commit message: "Fix something"
> git rev-list 1ced60c576f7dcd1e030b92fc08a405e4c8a5800 # timeout=10
Sverre Moe
added a comment - Running with downgraded Git plugin 3.6.4 and Git Client plugin 2.6.0 we are hardly experiencing the problem. It is far less frequent.
A build that didn't hang, where the git commit hash on rev-list where different from checkout.
> git checkout -f bc4dd5d1939f16e0124251327b0490a0059b353d
Commit message: "Fix something"
> git rev-list 1ced60c576f7dcd1e030b92fc08a405e4c8a5800 # timeout=10
Hung Vo
added a comment - In my case i was able to fix by downgrade the jira-plugin to 2.5 as there was a bug with the JiraChangeLogAnnotator. If any one upgrade to 2.5.1 should downgrade to 2.5 to fix the issue SEVERE: ChangeLogAnnotator hudson.plugins.jira.JiraChangeLogAnnotator@xxx failed to annotate message.
Hung Vo
added a comment - In my case i was able to fix by downgrade the jira-plugin to 2.5 as there was a bug with the JiraChangeLogAnnotator. If any one upgrade to 2.5.1 should downgrade to 2.5 to fix the issue SEVERE: ChangeLogAnnotator hudson.plugins.jira.JiraChangeLogAnnotator@xxx failed to annotate message. Sverre Moe
added a comment - Why would the JIRA plugin affect this? We are not even using it in our pipeline.
Sverre Moe
added a comment - Why would the JIRA plugin affect this? We are not even using it in our pipeline. djviking refer to JENKINS-48357 for details of the impact of Jira plugin 2.5.1. This bug was reported before the release of Jira plugin 2.5.1. I suspect that Jira plugin 2.5.1 related bugs are a "side path" from this bug. The real bug is likely still in the system as reported originally.
markewaite I hit this too - I am trying downgrade to 2.5.0 and will report back if it is all ok after that. This may be a dupe of https://issues.jenkins-ci.org/browse/JENKINS-43106 but I put a support bundle there. in what log did people see the JIRA error?
I swear that JIRA plugin (and the Atlassian dependencies) are a disaster.
Sverre Moe
added a comment - We have also downgraded JIRA plugin to 2.5.0 and it seems to have resolved the problem.
Sverre Moe
added a comment - We have also downgraded JIRA plugin to 2.5.0 and it seems to have resolved the problem. 
Sam Van Oort
added a comment - markewaite I'm downgrading the priority to reflect the fix via JIRA... not clear if this should stay as a pipeline bug though or if we can close it out.
Sam Van Oort
added a comment - markewaite I'm downgrading the priority to reflect the fix via JIRA... not clear if this should stay as a pipeline bug though or if we can close it out. 
Carlton Brown
added a comment - Seeing the same issue here... no luck with the workarounds around downgrading the JRIA plugin
Carlton Brown
added a comment - Seeing the same issue here... no luck with the workarounds around downgrading the JRIA plugin
Just confirmed that behaviour is consistent regardless if the job is executed on openshift hosted dynamic slave, master node or the bare metal slave.
Also, it is observed when the git repository is hosted on the github or the git repo hosted on the linux server which is accessed over the SSH.
Will test it over another Jenkins installation that was running perfectly well just two weeks ago.