Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53384

Jobs complain no free executor or offline node

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Fixed but Unreleased (View Workflow)
    • Priority: Minor
    • Resolution: Duplicate
    • Component/s: build-blocker-plugin
    • Labels:
    • Environment:
      Jenkins 2.89.4
      Windows OS
      Build-blocker-plugin V1.7.3
    • Similar Issues:

      Description

      Since two weeks we are experiencing that occasionally the Jenkins pipeline jobs are stating that there is no free executor or the node required to run the job is offline. Therefore the jobs are not executed but queued. When checking the respective node, it is running and has free executors. After restarting the master the situation recovers and queued builds start to run.

      I can see the following log entry on the master around the time when the jobs start stucking

      Sep 02, 2018 5:46:51 AM hudson.model.AsyncPeriodicWork$1 run
      INFO: Finished DockerContainerWatchdog Asynchronous Periodic Work. 1 ms
      Sep 02, 2018 5:49:59 AM jenkins.util.AtmostOneTaskExecutor$1 call
      WARNING: null
      java.lang.NullPointerException
          at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
          at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
          at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
          at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
          at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
          at hudson.model.Queue.maintain(Queue.java:1597)
          at hudson.model.Queue$1.call(Queue.java:320)
          at hudson.model.Queue$1.call(Queue.java:317)
          at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:108)
          at jenkins.util.AtmostOneTaskExecutor$1.call(AtmostOneTaskExecutor.java:98)
          at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110)
          at java.lang.Thread.run(Thread.java:748)
       
      Sep 02, 2018 5:50:00 AM hudson.triggers.SafeTimerTask run
      SEVERE: Timer task hudson.model.Queue$MaintainTask@495c6cda failed
      java.lang.NullPointerException
          at hudson.plugins.buildblocker.BlockingJobsMonitor.checkNodeForQueueEntries(BlockingJobsMonitor.java:108)
          at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkAccordingToProperties(BuildBlockerQueueTaskDispatcher.java:171)
          at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.checkForBlock(BuildBlockerQueueTaskDispatcher.java:127)
          at hudson.plugins.buildblocker.BuildBlockerQueueTaskDispatcher.canTake(BuildBlockerQueueTaskDispatcher.java:110)
          at hudson.model.Queue$JobOffer.getCauseOfBlockage(Queue.java:273)
          at hudson.model.Queue.maintain(Queue.java:1597)
          at hudson.model.Queue$MaintainTask.doRun(Queue.java:2851)
          at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
          at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
      

      In contrary to issue JENKINS-38729 which reports a similar problem (same error stack) we do not use the Build Blocker plugin.

      Yes we also have the build blocker plugin installed: V1.7.3 but we are not using it.

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The plugin is still installed, and the Extension is enabled. So the issue happens even if you do not use it. I would recommend uninstalling the plugin as a solution for your instance.

            I do not exactly know what causes NPE, but after quick glance at the code, it looks like a bug in the plugin

             

            Show
            oleg_nenashev Oleg Nenashev added a comment - The plugin is still installed, and the Extension is enabled. So the issue happens even if you do not use it. I would recommend uninstalling the plugin as a solution for your instance. I do not exactly know what causes NPE, but after quick glance at the code, it looks like a bug in the plugin  
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            From what I see it is a full duplicate of JENKINS-28513

            Show
            oleg_nenashev Oleg Nenashev added a comment - From what I see it is a full duplicate of JENKINS-28513
            Hide
            papanito Adrian Wyssmann added a comment -

            I have uninstalled the build blocker plugin and restarted Jenkins. This weekend we ran again into the same problem as described in the issue. However, there is no NPE anymore (obviously) but also I don't see any suspicious in the log file

            Show
            papanito Adrian Wyssmann added a comment - I have uninstalled the build blocker plugin and restarted Jenkins. This weekend we ran again into the same problem as described in the issue. However, there is no NPE anymore (obviously) but also I don't see any suspicious in the log file

              People

              Assignee:
              oleg_nenashev Oleg Nenashev
              Reporter:
              papanito Adrian Wyssmann
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: