Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53384

Jobs complain no free executor or offline node

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Minor Minor
    • build-blocker-plugin
    • Jenkins 2.89.4
      Windows OS
      Build-blocker-plugin V1.7.3

      Since two weeks we are experiencing that occasionally the Jenkins pipeline jobs are stating that there is no free executor or the node required to run the job is offline. Therefore the jobs are not executed but queued. When checking the respective node, it is running and has free executors. After restarting the master the situation recovers and queued builds start to run.

      I can see the following log entry on the master around the time when the jobs start stucking

      Sep 02, 2018 5:46:51 AM hudson.model.AsyncPeriodicWork$1 run
INFO: Finished DockerContainerWatchdog Asynchronous Periodic Work. 1 ms
Sep 02, 2018 5:49:59 AM jenkins.util.AtmostOneTaskExecutor$1 call
WARNING: null
java.lang.NullPointerException
    at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
    at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
    at hudson.model.Queue.maintain(Queue.java:1597)
    at hudson.model.Queue$1.call(Queue.java:320)
    at hudson.model.Queue$1.call(Queue.java:317)
    at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108)
    at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98)
    at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
    at java.lang.Thread.run(Thread.java:748)
 
Sep 02, 2018 5:50:00 AM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.model.Queue$MaintainTask@495c6cda failed
java.lang.NullPointerException
    at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
    at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
    at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
    at hudson.model.Queue.maintain(Queue.java:1597)
    at hudson.model.Queue$MaintainTask.doRun(Queue.java:2851)
    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

      In contrary to issue JENKINS-38729 which reports a similar problem (same error stack) we do not use the Build Blocker plugin.

      Yes we also have the build blocker plugin installed: V1.7.3 but we are not using it.

          [JENKINS-53384] Jobs complain no free executor or offline node

          papanito created issue -
          papanito made changes -
          Description Original: Since two weeks we are experiencing that occasionally the Jenkins pipeline jobs are stating that there is no free executor or the node required to run the job is offline. Therefore the jobs are not executed but queued. When checking the respective node, it is running and has free executors. After restarting the master the situation recovers and queued builds start to run.

          I can see the following log entry on the master around the time when the jobs start stucking
          {code:java}
          Sep 02, 2018 5:46:51 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished DockerContainerWatchdog Asynchronous Periodic Work. 1 ms
          Sep 02, 2018 5:49:59 AM jenkins.util.AtmostOneTaskExecutor$1 call
          WARNING: null
          java.lang.NullPointerException
              at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
              at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
              at hudson.model.Queue.maintain(Queue.java:1597)
              at hudson.model.Queue$1.call(Queue.java:320)
              at hudson.model.Queue$1.call(Queue.java:317)
              at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108)
              at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98)
              at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
              at java.lang.Thread.run(Thread.java:748)
           
          Sep 02, 2018 5:50:00 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task hudson.model.Queue$MaintainTask@495c6cda failed
          java.lang.NullPointerException
              at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
              at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
              at hudson.model.Queue.maintain(Queue.java:1597)
              at hudson.model.Queue$MaintainTask.doRun(Queue.java:2851)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:748)
          {code}
          In contrary to issue JENKINS-38729 which reports a similar problem (same error stack) we do not use the Build Blocker plugin.           		New: Since two weeks we are experiencing that occasionally the Jenkins pipeline jobs are stating that there is no free executor or the node required to run the job is offline. Therefore the jobs are not executed but queued. When checking the respective node, it is running and has free executors. After restarting the master the situation recovers and queued builds start to run.

          I can see the following log entry on the master around the time when the jobs start stucking
          {code:java}
          Sep 02, 2018 5:46:51 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished DockerContainerWatchdog Asynchronous Periodic Work. 1 ms
          Sep 02, 2018 5:49:59 AM jenkins.util.AtmostOneTaskExecutor$1 call
          WARNING: null
          java.lang.NullPointerException
              at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
              at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
              at hudson.model.Queue.maintain(Queue.java:1597)
              at hudson.model.Queue$1.call(Queue.java:320)
              at hudson.model.Queue$1.call(Queue.java:317)
              at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108)
              at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98)
              at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
              at java.lang.Thread.run(Thread.java:748)
           
          Sep 02, 2018 5:50:00 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task hudson.model.Queue$MaintainTask@495c6cda failed
          java.lang.NullPointerException
              at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
              at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
              at hudson.model.Queue.maintain(Queue.java:1597)
              at hudson.model.Queue$MaintainTask.doRun(Queue.java:2851)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:748)
          {code}
          -In contrary to issue JENKINS-38729 which reports a similar problem (same error stack) we do not use the Build Blocker plugin.-

          Yes we also have the build blocker plugin installed: V1.7.3
          papanito made changes -
          Environment Original: Jenkins 2.89.4
          Windows OS           		New: Jenkins 2.89.4
          Windows OS
          Build-blocker-plugin V1.7.3
          papanito made changes -
          Description Original: Since two weeks we are experiencing that occasionally the Jenkins pipeline jobs are stating that there is no free executor or the node required to run the job is offline. Therefore the jobs are not executed but queued. When checking the respective node, it is running and has free executors. After restarting the master the situation recovers and queued builds start to run.

          I can see the following log entry on the master around the time when the jobs start stucking
          {code:java}
          Sep 02, 2018 5:46:51 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished DockerContainerWatchdog Asynchronous Periodic Work. 1 ms
          Sep 02, 2018 5:49:59 AM jenkins.util.AtmostOneTaskExecutor$1 call
          WARNING: null
          java.lang.NullPointerException
              at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
              at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
              at hudson.model.Queue.maintain(Queue.java:1597)
              at hudson.model.Queue$1.call(Queue.java:320)
              at hudson.model.Queue$1.call(Queue.java:317)
              at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108)
              at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98)
              at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
              at java.lang.Thread.run(Thread.java:748)
           
          Sep 02, 2018 5:50:00 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task hudson.model.Queue$MaintainTask@495c6cda failed
          java.lang.NullPointerException
              at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
              at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
              at hudson.model.Queue.maintain(Queue.java:1597)
              at hudson.model.Queue$MaintainTask.doRun(Queue.java:2851)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:748)
          {code}
          -In contrary to issue JENKINS-38729 which reports a similar problem (same error stack) we do not use the Build Blocker plugin.-

          Yes we also have the build blocker plugin installed: V1.7.3           		New: Since two weeks we are experiencing that occasionally the Jenkins pipeline jobs are stating that there is no free executor or the node required to run the job is offline. Therefore the jobs are not executed but queued. When checking the respective node, it is running and has free executors. After restarting the master the situation recovers and queued builds start to run.

          I can see the following log entry on the master around the time when the jobs start stucking
          {code:java}
          Sep 02, 2018 5:46:51 AM hudson.model.AsyncPeriodicWork$1 run
          INFO: Finished DockerContainerWatchdog Asynchronous Periodic Work. 1 ms
          Sep 02, 2018 5:49:59 AM jenkins.util.AtmostOneTaskExecutor$1 call
          WARNING: null
          java.lang.NullPointerException
              at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
              at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
              at hudson.model.Queue.maintain(Queue.java:1597)
              at hudson.model.Queue$1.call(Queue.java:320)
              at hudson.model.Queue$1.call(Queue.java:317)
              at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108)
              at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98)
              at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
              at java.lang.Thread.run(Thread.java:748)
           
          Sep 02, 2018 5:50:00 AM hudson.triggers.SafeTimerTask run
          SEVERE: Timer task hudson.model.Queue$MaintainTask@495c6cda failed
          java.lang.NullPointerException
              at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
              at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
              at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
              at hudson.model.Queue.maintain(Queue.java:1597)
              at hudson.model.Queue$MaintainTask.doRun(Queue.java:2851)
              at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
              at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:748)
          {code}
          -In contrary to issue JENKINS-38729 which reports a similar problem (same error stack) we do not use the Build Blocker plugin.-

          Yes we also have the build blocker plugin installed: V1.7.3 but we are not using it.

          Oleg Nenashev added a comment -

          The plugin is still installed, and the Extension is enabled. So the issue happens even if you do not use it. I would recommend uninstalling the plugin as a solution for your instance.

          I do not exactly know what causes NPE, but after quick glance at the code, it looks like a bug in the plugin

           

          Oleg Nenashev added a comment - The plugin is still installed, and the Extension is enabled. So the issue happens even if you do not use it. I would recommend uninstalling the plugin as a solution for your instance. I do not exactly know what causes NPE, but after quick glance at the code, it looks like a bug in the plugin  
          Oleg Nenashev made changes -
          Component/s New: build-blocker-plugin [ 17470 ]
          Component/s Original: _unsorted [ 19622 ]
          Oleg Nenashev made changes -
          Assignee Original: Oleg Nenashev [ oleg_nenashev ]

          Oleg Nenashev added a comment -

          From what I see it is a full duplicate of JENKINS-28513

          Oleg Nenashev added a comment - From what I see it is a full duplicate of JENKINS-28513
          Oleg Nenashev made changes -
          Link New: This issue duplicates JENKINS-28513 [ JENKINS-28513 ]
          Oleg Nenashev made changes -
          Resolution New: Duplicate [ 3 ]
          Status Original: Open [ 1 ] New: Fixed but Unreleased [ 10203 ]

            oleg_nenashev Oleg Nenashev
            papanito papanito
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: