Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46782

Scheduling failing

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Duplicate
    • Component/s: blueocean-plugin
    • Labels:
    • Environment:
      64-bit Linux CentOS 7, Jenkins 2.77, plenty of plugins (including pipeline)
    • Similar Issues:

      Description

      We've been having scheduling problems on our Jenkins server in the past few weeks. I haven't been able to figure out what is causing these problems, but I've noticed the following things:

      1. Pipeline jobs are displayed as 'pending: waiting for executors on master', even though only 1 job is running, and there are 19 available executors.
      2. Triggering a (non-pipeline) job manually appears to have no effect (no job appears in the queue), until several minutes later.
      3. On both slaves and master, jobs are displayed as running, even though they have already finished minutes ago.(which can be seen by clicking on the job, where you can see that the job has been successful/failed).

      I have not been able to pinpoint a reason for this. It doesn't appear to be related to load, as I've seen this issue with a single running (light-weight) job and 1 job in the queue, with low load and low disk access on the master.

      This is making Jenkins (at the times when it occurs) unusable for us.

        Attachments

          Issue Links

            Activity

            Hide
            mathiasdm Mathias De Maré added a comment -

            Not sure if it's related, but the following stack trace occurs sometimes in our logs:

             

            WARNING: Error publishing Run queued event.
            org.jenkinsci.plugins.pubsub.MessageException: Event enrichment failure due to unexpected exception in io.jenkins.blueocean.events.BlueMessageEnricher.
            {{ at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:135)}}
            {{ at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.publish(SyncQueueListener.java:178)}}
            {{ at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.publish(SyncQueueListener.java:172)}}
            {{ at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.onEnterWaiting(SyncQueueListener.java:139)}}
            {{ at hudson.model.Queue$WaitingItem.enter(Queue.java:2393)}}
            {{ at hudson.model.Queue.scheduleInternal(Queue.java:632)}}
            {{ at hudson.model.Queue.schedule2(Queue.java:588)}}
            {{ at jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:155)}}
            {{ at jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:115)}}
            {{ at jenkins.model.ParameterizedJobMixIn$ParameterizedJob.scheduleBuild(ParameterizedJobMixIn.java:405)}}
            {{ at hudson.triggers.TimerTrigger.run(TimerTrigger.java:67)}}
            {{ at hudson.triggers.Trigger.checkTriggers(Trigger.java:278)}}
            {{ at hudson.triggers.Trigger$Cron.doRun(Trigger.java:226)}}
            {{ at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)}}
            {{ at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)}}
            {{ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}
            {{ at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)}}
            {{ at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)}}
            {{ at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)}}
            {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}}
            {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}}
            {{ at java.lang.Thread.run(Thread.java:745)}}
            Caused by: java.lang.NullPointerException

             

            Searching around turns up this report: https://issues.jenkins-ci.org/browse/JENKINS-44014

            Priority sorter is not present on our system though.

            Show
            mathiasdm Mathias De Maré added a comment - Not sure if it's related, but the following stack trace occurs sometimes in our logs:   WARNING: Error publishing Run queued event. org.jenkinsci.plugins.pubsub.MessageException: Event enrichment failure due to unexpected exception in io.jenkins.blueocean.events.BlueMessageEnricher. {{ at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:135)}} {{ at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.publish(SyncQueueListener.java:178)}} {{ at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.publish(SyncQueueListener.java:172)}} {{ at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.onEnterWaiting(SyncQueueListener.java:139)}} {{ at hudson.model.Queue$WaitingItem.enter(Queue.java:2393)}} {{ at hudson.model.Queue.scheduleInternal(Queue.java:632)}} {{ at hudson.model.Queue.schedule2(Queue.java:588)}} {{ at jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:155)}} {{ at jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:115)}} {{ at jenkins.model.ParameterizedJobMixIn$ParameterizedJob.scheduleBuild(ParameterizedJobMixIn.java:405)}} {{ at hudson.triggers.TimerTrigger.run(TimerTrigger.java:67)}} {{ at hudson.triggers.Trigger.checkTriggers(Trigger.java:278)}} {{ at hudson.triggers.Trigger$Cron.doRun(Trigger.java:226)}} {{ at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)}} {{ at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)}} {{ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}} {{ at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)}} {{ at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)}} {{ at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)}} {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)}} {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)}} {{ at java.lang.Thread.run(Thread.java:745)}} Caused by: java.lang.NullPointerException   Searching around turns up this report: https://issues.jenkins-ci.org/browse/JENKINS-44014 Priority sorter is not present on our system though.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            It seems the part of the log is missing (one below "Caused by: java.lang.NullPointerException"). Could you please provide it?

            Show
            oleg_nenashev Oleg Nenashev added a comment - It seems the part of the log is missing (one below " Caused by: java.lang.NullPointerException "). Could you please provide it?
            Hide
            mmitche Matthew Mitchell added a comment -

            I think it's this (i'm getting the same)

            Caused by: java.lang.NullPointerException
             at io.jenkins.blueocean.service.embedded.rest.QueueUtil.getQueuedItem(QueueUtil.java:24)
             at io.jenkins.blueocean.events.BlueMessageEnricher.enrich(BlueMessageEnricher.java:100)
             at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:133)
             ... 16 more
            Show
            mmitche Matthew Mitchell added a comment - I think it's this (i'm getting the same) Caused by: java.lang.NullPointerException at io.jenkins.blueocean.service.embedded. rest .QueueUtil.getQueuedItem(QueueUtil.java:24) at io.jenkins.blueocean.events.BlueMessageEnricher.enrich(BlueMessageEnricher.java:100) at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:133) ... 16 more
            Hide
            jamesdumay James Dumay added a comment -

            The stacktrace indicates that this problem is a duplicate of JENKINS-45589 and has been fixed.

            Show
            jamesdumay James Dumay added a comment - The stacktrace indicates that this problem is a duplicate of JENKINS-45589 and has been fixed.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              mathiasdm Mathias De Maré
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: