-
Bug
-
Resolution: Unresolved
-
Major
-
None
We've been noticing problems with our jenkins branch indexing, which seems to hang from several minutes to hours. We see a lot of Branch indexing jobs on the queue and in the Branch indexing log itself, it shows (what we believe) a complete log with a loading spinner gif at the end.
Not sure what the error is. Would be happy to dig in and provide more info if needed.
Jenkins version: 2.7.1
Installed Plugins:
Ant Plugin - 1.3
Authentication Tokens API Plugin - 1.3
Branch API Plugin - 1.10
build timeout plugin - 1.17
CloudBees Docker Pipeline - 1.6
Credentials Plugin - 2.1.4
CVS Plug-in - 2.12
Docker Commons Plugin - 1.4.0
Durable Task Plugin - 1.11
External Monitor Job Type Plugin - 1.5
Folders Plugin - 5.12
Git client plugin - 1.19.6
Git plugin - 2.5.2
GIT server Plugin - 1.7
GitHub API Plugin - 1.76
GitHub Branch Source Plugin - 1.8.1
GitHub Organization Folder Plugin - 1.3
GitHub plugin - 1.19.2
Google Login Plugin - 1.2.1
Icon Shim Plugin - 2.0.3
Javadoc Plugin - 1.4
JavaScript GUI Lib: ACE Editor bundle plugin - 1.1
JavaScript GUI Lib: Handlebars bundle plugin - 1.1.1
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin - 1.2.1
JavaScript GUI Lib: Moment.js bundle plugin - 1.1.1
JUnit Plugin - 1.15
LDAP Plugin - 1.12
Mailer Plugin - 1.17
MapDB API Plugin - 1.0.9.0
Matrix Authorization Strategy Plugin - 1.4
Matrix Project Plugin - 1.7.1
Maven Integration plugin - 2.13
OWASP Markup Formatter Plugin - 1.5
PAM Authentication plugin - 1.3
Pipeline - 2.2
Pipeline: API - 2.1
Pipeline: Basic Steps - 2.0
Pipeline: Build Step - 2.2
Pipeline: Groovy - 2.9
Pipeline: Input Step - 2.0
Pipeline: Job - 2.3
Pipeline: Multibranch - 2.8
Pipeline: Nodes and Processes - 2.3
Pipeline: REST API Plugin - 1.5
Pipeline: SCM Step - 2.2
Pipeline: Shared Groovy Libraries - 2.1
Pipeline: Stage Step - 2.1
Pipeline: Stage View Plugin - 1.5
Pipeline: Step API - 2.2
Pipeline: Supporting APIs - 2.2
Plain Credentials Plugin - 1.2
Role-based Authorization Strategy - 2.3.2
SCM API Plugin - 1.2
Script Security Plugin - 1.21
Simple Theme Plugin - 0.3
Slack Notification Plugin - 2.0.1
SSH Credentials Plugin - 1.12
SSH Slaves plugin - 1.11
Structs Plugin - 1.2
Subversion Plug-in - 2.6
ThinBackup - 1.7.4
Timestamper - 1.8.4
Token Macro Plugin - 1.12.1
Translation Assistance plugin - 1.15
Windows Slaves Plugin - 1.1
[JENKINS-36724] Branch indexing hangs
Unless the branch indexing log makes it clear why indexing is blocked, you would need to look in a Jenkins thread dump to see the cause of the issue.
Please verify if this issue is an issue with GitHub Branch Source 2.0.0-beta-1 (available from the experimental update center now or 2.0.0 (available in early January 2017)
Looking at the screenshots, I suspect that this is actually that the jobs triggered by branch indexing are just sitting in the queue.
Your branch indexing is running, and re-submitting the job for execution. As the job is already queued, the cause gets added to the queue item's cause and indexing completes.
You probably want to look at why the jobs are not getting executed this could be a missing label or a concurrent job being executed.
Alexandre Aubert
added a comment - Hi,
i'm sorry to re-open this but it seems there is a real problem with multiple branch indexing causing Jenkins to hang and be unusable.
if we have a lot of multibranch pipeline jobs, it triggers 'branch indexing' operations and it regularly hangs when there are too many of those. I commented on ticket 37570 already but i think the problem is also related to this one.
Could you please have a look on that?
I of course available for any additional test or information in order to investigate to find out a solution.
Thanks.
Alexandre Aubert
added a comment - Hi,
i'm sorry to re-open this but it seems there is a real problem with multiple branch indexing causing Jenkins to hang and be unusable.
if we have a lot of multibranch pipeline jobs, it triggers 'branch indexing' operations and it regularly hangs when there are too many of those. I commented on ticket 37570 already but i think the problem is also related to this one.
Could you please have a look on that?
I of course available for any additional test or information in order to investigate to find out a solution.
Thanks. Marking as incomplete again because we need more details to be able to make any progress, in the absence of any details this appears to be an unrelated issue that I have described previously
David Taylor
added a comment - I don't believe this is fully fixed yet. A similar issue is happening to our multibranch pipeline builds when they run their scheduled `Scan Multibranch Pipeline Now` command. The jobs are stuck on Master and they can not be stopped unless I shutdown Master, restart Jenkins, kill any stuck jobs that are left, restart Master.
David Taylor
added a comment - I don't believe this is fully fixed yet. A similar issue is happening to our multibranch pipeline builds when they run their scheduled `Scan Multibranch Pipeline Now` command. The jobs are stuck on Master and they can not be stopped unless I shutdown Master, restart Jenkins, kill any stuck jobs that are left, restart Master. As far as I can tell, when the plugin was updated to HttpClient 4.x, the BitbucketServerAPIClient class lost its socket timeout configuration, but the BitbucketCloudApiClient didn't: https://github.com/jenkinsci/bitbucket-branch-source-plugin/commit/8f81d4026153e7ee250252180c34ed2eaa7a784f
This defect is hitting us hard (it goes beyond branch/folder indexing - many runs are getting stuck, too, with a stack trace that includes this plugin) and I'm working on a pull request.
Oops, hadn't noticed the component for this issue was the github-branch-source-plugin. I can open a separate issue for the bitbucket-branch-source-plugin, if you want.
FYI, posted PR #135: Restore timeouts in BitbucketServerAPIClient, in case it helps the diagnosis of what appears to be a similar issue in the github-branch-source-plugin.
After a few days there are so many "Branch Indexing" messages that it fills up the screen and doesn't process new PRs anymore.
Jenkins needs to be restarted for this to be bypassed.