[JENKINS-53384] Jobs complain no free executor or offline node

Type: Bug
Resolution: Duplicate
Priority: Minor
Component/s: build-blocker-plugin
Labels:
- jenkins
Environment:
Jenkins 2.89.4
Windows OS
Build-blocker-plugin V1.7.3

Similar Issues:
Powered by SuggestiMate

Show

Since two weeks we are experiencing that occasionally the Jenkins pipeline jobs are stating that there is no free executor or the node required to run the job is offline. Therefore the jobs are not executed but queued. When checking the respective node, it is running and has free executors. After restarting the master the situation recovers and queued builds start to run.

I can see the following log entry on the master around the time when the jobs start stucking

Sep 02, 2018 5:46:51 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished DockerContainerWatchdog Asynchronous Periodic Work. 1 ms
Sep 02, 2018 5:49:59 AM jenkins.util.AtmostOneTaskExecutor$1 call
WARNING: null
java.lang.NullPointerException
    at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
    at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
    at hudson.model.Queue.maintain(Queue.java:1597)
    at hudson.model.Queue$1.call(Queue.java:320)
    at hudson.model.Queue$1.call(Queue.java:317)
    at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108)
    at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
    at java.lang.Thread.run(Thread.java:748)
 
Sep 02, 2018 5:50:00 AM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.model.Queue$MaintainTask@495c6cda failed
java.lang.NullPointerException
    at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
    at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
    at hudson.model.Queue.maintain(Queue.java:1597)
    at hudson.model.Queue$MaintainTask.doRun(Queue.java:2851)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

~~In contrary to issue JENKINS-38729 which reports a similar problem (same error stack) we do not use the Build Blocker plugin.~~

Yes we also have the build blocker plugin installed: V1.7.3 but we are not using it.

duplicates

JENKINS-28513 Build-Blocker-Plugin blocks on builds queued leading to deadlock

Resolved

Oleg Nenashev added a comment - 2018-09-03 11:08

The plugin is still installed, and the Extension is enabled. So the issue happens even if you do not use it. I would recommend uninstalling the plugin as a solution for your instance.

I do not exactly know what causes NPE, but after quick glance at the code, it looks like a bug in the plugin

Oleg Nenashev added a comment - 2018-09-03 11:08 The plugin is still installed, and the Extension is enabled. So the issue happens even if you do not use it. I would recommend uninstalling the plugin as a solution for your instance. I do not exactly know what causes NPE, but after quick glance at the code, it looks like a bug in the plugin

Oleg Nenashev added a comment - 2018-09-03 11:20

From what I see it is a full duplicate of ~~JENKINS-28513~~

Oleg Nenashev added a comment - 2018-09-03 11:20 From what I see it is a full duplicate of JENKINS-28513

papanito added a comment - 2018-09-10 09:21

I have uninstalled the build blocker plugin and restarted Jenkins. This weekend we ran again into the same problem as described in the issue. However, there is no NPE anymore (obviously) but also I don't see any suspicious in the log file

papanito added a comment - 2018-09-10 09:21 I have uninstalled the build blocker plugin and restarted Jenkins. This weekend we ran again into the same problem as described in the issue. However, there is no NPE anymore (obviously) but also I don't see any suspicious in the log file

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Oleg Nenashev added a comment - 2018-09-03 11:08

Expand comment: Oleg Nenashev added a comment - 2018-09-03 11:08

Collapse comment: Oleg Nenashev added a comment - 2018-09-03 11:20

Expand comment: Oleg Nenashev added a comment - 2018-09-03 11:20

Collapse comment: papanito added a comment - 2018-09-10 09:21

Expand comment: papanito added a comment - 2018-09-10 09:21

People

Dates