-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Jenkins version: 2.319.1
Priority Sorter Plugin: 4.1.0
Hi,
I recently tried upgrading the priority sorter plugin from 3.6.0 to 4.1.0. This eventually caused Jenkins to become unresponsive and fail to load after about 4 hours of runtime. The Jenkins process was still running when we investigated, but needed a `systemctl restart` to become responsive again. After the restart, Jenkins again slowed and eventually started throwing 500 errors again after another 4-8 hours. Downgrading the plugin back to version 3.6.0 brought it back to a place of stability.
[JENKINS-67620] Priority Sorter Plugin Crashes Jenkins when Queue is large
Andrew Savino
added a comment - I agree, looking at the diff, nothing immediately looks suspicious. I'm not the most familiar with Jenkins -are there logs that might help point to the cause of the slowdown/crash?
Andrew Savino
added a comment - I agree, looking at the diff, nothing immediately looks suspicious. I'm not the most familiar with Jenkins -are there logs that might help point to the cause of the slowdown/crash? Might be in the Jenkins log file that is written to disc. Usually at /var/log/jenkins/*
I think we observed this yesterday. After a few hours, jenkins became unable to schedule new work (either through cron triggers or POST requests). Threads were started but would hang and Jetty would never return to the user in the case of manually started jobs. We didn't find anything relevant in the logs and after restarting jenkins we saw the same issue within a few hours. We're looking into a reproducer but I'm not confident we'll be able to simulate the load.
We have a similar version of jenkins and were not actively using Priority Sorter (it had been installed the day before).
tapdancingrodent thanks for reporting the issue. I reviewed the pull request https://github.com/jenkinsci/priority-sorter-plugin/pull/51/files that was mentioned in a previous comment and found nothing suspicious. If more details are available, please share them in case it helps others with this type of failure.
Unfortunately it would seem like queue length isn’t the only factor, here. When we first observed this issue it was with 45-50 busy built-in executors and a modest queue (<50 jobs). We absolutely thrashed our dev instance (exact same container as saw the failures in prod, test cases below) with simulated workloads but were unable to replicate the problem with version 4.1.
100 busy built-in executors, 300+ jobs in the queue.
100 busy EC2 executors, ~100 jobs in the queue.
Based on this, you can probably rule out queue size, overall load, and rate of jobs entering the queue as root causes - maybe another plugin is interacting badly with Priorty Sorter and causing it to enter a hanging state? I've attached a list of plugin versions we're using with jenkins at `2.319.2`.
Sorry that it failed like that. Pull requests that have been merged between 3.6.0 and 4.1.0 include:
JENKINS-64694-fix-plugin (UI form changes)I don't see any of those that seem obviously like they would cause a performance regression or a dramatic change of behavior. Dependencies are updated, UI forms are updated, and there are some changes in https://github.com/jenkinsci/priority-sorter-plugin/pull/51/files that are intended to remain binary compatible.