-
Bug
-
Resolution: Done
-
Blocker
-
Jenkins - 1.625.2
Active Directory plugin - 1.41
AnsiColor - 0.4.1
Ant Plugin - 1.2
Build Blocker Plugin - 1.7.1
Build Name Setter Plugin - 1.3
build timeout plugin - 1.15
Build Trigger Badge Plugin - 2.1
buildgraph-view - 1.1.1
CloudBees Build Flow plugin - 0.17
CloudBees Folder Plugin - 4.10
conditional-buildstep - 1.3.3
Copy Artifact Plugin - 1.35
Credentials Plugin - 1.22
CVS Plug-in - 2.12
Dashboard View - 2.9.6
Delivery Pipeline Plugin - 0.9.7
Email Extension Plugin - 2.40.5
Environment Injector Plugin - 1.91.1
Extended Choice Parameter Plug-In - 0.36
External Monitor Job Type Plugin - 1.4
Green Balls - 1.14
Groovy - 1.27
Groovy Postbuild - 2.2
HTML Publisher plugin - 1.6
Javadoc Plugin - 1.3
Job DSL - 1.39
Job Generator - 1.22
Job Node Stalker - 1.0.4
jQuery plugin - 1.11.2-0
jQuery UI plugin - 1.0.2
JUnit Plugin - 1.9
LDAP Plugin - 1.11
Log Parser Plugin - 1.0.8
Mailer Plugin - 1.15
MapDB API Plugin - 1.0.6.0
Matrix Authorization Strategy Plugin - 1.2
Matrix Project Plugin - 1.4.1
Maven Integration plugin - 2.9
Monitoring - 1.57.0
Node and Label parameter plugin - 1.5.1
OWASP Markup Formatter Plugin - 1.3
PAM Authentication plugin - 1.2
Plug plugin - 1.9
Powershell plugin - 1.3
Project statistics Plugin - 0.4
Promoted Builds (Simple) - 1.9
Rebuilder - 1.25
Run Condition Plugin - 1.0
SCM API Plugin - 0.2
Script Security Plugin - 1.13
Self Organizing Swarm Plug-in Modules - 2.0
Sidebar Link - 1.7
SSH Credentials Plugin - 1.11
SSH Slaves plugin - 1.9
Static Analysis Utilities - 1.71
Subversion Plug-in - 2.5.3
Throttle Concurrent Builds Plug-in - 1.8.4
Timestamper - 1.7.2
Toke Macro Plugin - 1.10
TraceTronic ECU-TEST Plugin - 1.0
Translation Assistance plugin - 1.12
user build vars plugin - 1.4
Warnings Plug-in - 4.47
Windows Slaves Plugin 1.0
Workflow Step API - 1.7
Workspace Cleanup Plugin - 0.25
xUnit plugin 1.97
Zentimestamp plugin - 3.3
Jenkins - 1.625.2 Active Directory plugin - 1.41 AnsiColor - 0.4.1 Ant Plugin - 1.2 Build Blocker Plugin - 1.7.1 Build Name Setter Plugin - 1.3 build timeout plugin - 1.15 Build Trigger Badge Plugin - 2.1 buildgraph-view - 1.1.1 CloudBees Build Flow plugin - 0.17 CloudBees Folder Plugin - 4.10 conditional-buildstep - 1.3.3 Copy Artifact Plugin - 1.35 Credentials Plugin - 1.22 CVS Plug-in - 2.12 Dashboard View - 2.9.6 Delivery Pipeline Plugin - 0.9.7 Email Extension Plugin - 2.40.5 Environment Injector Plugin - 1.91.1 Extended Choice Parameter Plug-In - 0.36 External Monitor Job Type Plugin - 1.4 Green Balls - 1.14 Groovy - 1.27 Groovy Postbuild - 2.2 HTML Publisher plugin - 1.6 Javadoc Plugin - 1.3 Job DSL - 1.39 Job Generator - 1.22 Job Node Stalker - 1.0.4 jQuery plugin - 1.11.2-0 jQuery UI plugin - 1.0.2 JUnit Plugin - 1.9 LDAP Plugin - 1.11 Log Parser Plugin - 1.0.8 Mailer Plugin - 1.15 MapDB API Plugin - 1.0.6.0 Matrix Authorization Strategy Plugin - 1.2 Matrix Project Plugin - 1.4.1 Maven Integration plugin - 2.9 Monitoring - 1.57.0 Node and Label parameter plugin - 1.5.1 OWASP Markup Formatter Plugin - 1.3 PAM Authentication plugin - 1.2 Plug plugin - 1.9 Powershell plugin - 1.3 Project statistics Plugin - 0.4 Promoted Builds (Simple) - 1.9 Rebuilder - 1.25 Run Condition Plugin - 1.0 SCM API Plugin - 0.2 Script Security Plugin - 1.13 Self Organizing Swarm Plug-in Modules - 2.0 Sidebar Link - 1.7 SSH Credentials Plugin - 1.11 SSH Slaves plugin - 1.9 Static Analysis Utilities - 1.71 Subversion Plug-in - 2.5.3 Throttle Concurrent Builds Plug-in - 1.8.4 Timestamper - 1.7.2 Toke Macro Plugin - 1.10 TraceTronic ECU-TEST Plugin - 1.0 Translation Assistance plugin - 1.12 user build vars plugin - 1.4 Warnings Plug-in - 4.47 Windows Slaves Plugin 1.0 Workflow Step API - 1.7 Workspace Cleanup Plugin - 0.25 xUnit plugin 1.97 Zentimestamp plugin - 3.3
Since a couple of days (since updating to 1.609.3 LTS) we have the problem that is the builds stick in the bild executor.If you open the build details the build is finished allready. But the executor is still blocked leading to a Build-queue that is constantly growing. Sometime the executor will display "DEAD 
" (German: "Tot "). There are no specific entries in the log file.
Master is running on a virtual Windows Server 2012. Slaves are Windoes Server 2012 and Windows 7.
- relates to
-
-
- Open
-
[JENKINS-31536] Jobs hanging in Build Executor even if it is finished
The problem still exists. I have tried to evaluete it a bit further:
- the problem only arises if there are a couple of jobs in the build queue (e.g. > 100)
- The Executors in the build processor status shows jobs as "in progress". But if you go on the job details it is already finished. Some time later the executor is marked as dead.
- Build queue is constantly filling, because jobs are executed very slowely
- CPU usage on master grows until it is running with 90 to 100% CPU usage
- ThreadDump shows an "AtmostOneTaskExecutor" that is waiting on "java.util.concurrent.locks.ReentrantLock$NonfairSync" owned by "jenkins.util.Timer x".
- The Timer threads are waiting
- If i kill the timer threads and the "AtmostOneTaskExecutor" work goes on. But this can not be a final solution.
| Labels | New: deadlock executor |
| Attachment | New: Thread Dump [Jenkins].pdf [ 31323 ] |
stephenconnolly Any idea what's going on here? Possibly related to your recent queue work?
Are there any ideas what causes this behaviour. We have a fairly large Jenkins architecture with over 4000 CI Jobs and i had to disable a whole bunch of them in order to prevent the server from failing all the time.
Al Shepherd
added a comment - I don't know if this helps any, but we have experienced this issue when running scheduled master jobs.
Details: we have a master job that is scheduled to run at 1am every day. This master job invokes other jobs via Conditional step (single) entries in the Build section. When a conditional step triggers a job that has the Warning plug-in configured for it (let's call it job 2), the triggered job (job 2)completes successfully but then control is never returned to the calling (master) job. When the plug-in configuration is removed from job 2 everything runs as it should. If we replace the Warnings plug-in job 2 then trigger a build of this job manually (not via the master job), the plug-in works as advertised (beautifully done by the way). Also, if we set up job 2 as a scheduled job and let it fire off via this trigger the Warnings plug-in behaves as it should.
Al Shepherd
added a comment - I don't know if this helps any, but we have experienced this issue when running scheduled master jobs.
Details: we have a master job that is scheduled to run at 1am every day. This master job invokes other jobs via Conditional step (single) entries in the Build section. When a conditional step triggers a job that has the Warning plug-in configured for it (let's call it job 2), the triggered job (job 2)completes successfully but then control is never returned to the calling (master) job. When the plug-in configuration is removed from job 2 everything runs as it should. If we replace the Warnings plug-in job 2 then trigger a build of this job manually (not via the master job), the plug-in works as advertised (beautifully done by the way). Also, if we set up job 2 as a scheduled job and let it fire off via this trigger the Warnings plug-in behaves as it should. I find the Node Stalker Plugin in the stack trace of the thread everyone else is waiting for very suspicious. What happens when you disable the plugin, or disable its features (if it offers such options)?
I found a workaround. After looking at the thread dump it seemd that threads where waiting for a jenkins.util.timer thread. After killing those threads work continued as normal.