Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44014

job triggering and queue broken since update to 2.57

    XMLWordPrintable

Details

    Description

      Since update to jenkins 2.57 (and still after update to 2.58), there is a lot of delay when clicking on 'build' button for a job. It could reach 5 minutes before really starting the build.

      Queue handling seems also to be altered (jobs don't come in queue rapidely)

      And the job triggering by upstream job is also really slow, reaching sometimes internal 'timeout'.

      What i can get from logs is (not sure which of those could be really related but it seems that those errors are linked together) :

      • NPE SyncQueueListener

      mai 03, 2017 12:42:02 PM org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener publish
      WARNING: Error publishing Run queued event.
      org.jenkinsci.plugins.pubsub.MessageException: Event enrichment failure due to unexpected exception in io.jenkins.blueocean.events.BlueMessageEnricher.
      at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:113)
      at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.publish(SyncQueueListener.java:178)
      at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.publish(SyncQueueListener.java:172)
      at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.onEnterWaiting(SyncQueueListener.java:139)
      at hudson.model.Queue$WaitingItem.enter(Queue.java:2392)
      at hudson.model.Queue.scheduleInternal(Queue.java:633)
      at hudson.model.Queue.schedule2(Queue.java:589)
      at jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
      at jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
      at hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:838)
      at hudson.triggers.TimerTrigger.run(TimerTrigger.java:67)
      at hudson.triggers.Trigger.checkTriggers(Trigger.java:278)
      at hudson.triggers.Trigger$Cron.doRun(Trigger.java:226)
      at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
      at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
      Caused by: java.lang.NullPointerException

      • Warnings on AdvancedQueueSorterQueueListener

      mai 03, 2017 8:18:37 AM jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener onEnterBuildable
      WARNING: onEnterBuilding() called without prior call to onEnterWaiting() for 'API-availability-dev'

      • Warnings on hudson.model.Queue$WaitingItem enter

      mai 03, 2017 8:25:11 AM hudson.model.Queue$WaitingItem enter
      WARNING: QueueListener failed while processing hudson.model.Queue$WaitingItem:ExecutorStepExecution.PlaceholderTask

      {runId=Certifications/Sparktacus/Development_Sparktacus#12,label=Certif-BDXBLD006,context=CpsStepContext[3:null]:OwnerCertifications/Sparktacus/Development_Sparktacus/12:Certifications/Sparktacus/Development_Sparktacus #12,cookie=8d0b6bd9-e673-409f-9234-2279205b290b}

      :69158
      java.lang.ClassCastException: org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask cannot be cast to hudson.model.Job
      at jenkins.advancedqueue.PriorityConfigurationPlaceholderTaskHelper.getPriority(PriorityConfigurationPlaceholderTaskHelper.java:19)
      at jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:213)
      at jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:205)
      at jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
      at jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
      at hudson.model.Queue$WaitingItem.enter(Queue.java:2392)
      at hudson.model.Queue.scheduleInternal(Queue.java:633)
      at hudson.model.Queue.schedule2(Queue.java:589)
      at hudson.model.Queue.schedule2(Queue.java:709)
      at org.jenkinsci.plugins.workflow.support.pickles.ExecutorPickle$1.tryResolve(ExecutorPickle.java:79)
      at org.jenkinsci.plugins.workflow.support.pickles.ExecutorPickle$1.tryResolve(ExecutorPickle.java:73)
      at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:92)
      at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

      • NPE on hudson.model.Queue$LeftItem enter

      mai 03, 2017 8:36:03 AM hudson.model.Queue$LeftItem enter
      WARNING: QueueListener failed while processing hudson.model.Queue$LeftItem:ExecutorStepExecution.PlaceholderTask

      {runId=Certifications/Sparktacus/Development_Sparktacus#12,label=Certif-BDXBLD006,context=CpsStepContext[3:null]:OwnerCertifications/Sparktacus/Development_Sparktacus/12:Certifications/Sparktacus/Development_Sparktacus #12,cookie=8d0b6bd9-e673-409f-9234-2279205b290b}

      :69158
      java.lang.NullPointerException
      at jenkins.advancedqueue.sorter.AdvancedQueueSorter.onLeft(AdvancedQueueSorter.java:145)
      at jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onLeft(AdvancedQueueSorterQueueListener.java:51)
      at hudson.model.Queue$LeftItem.enter(Queue.java:2660)
      at hudson.model.Queue.onStartExecuting(Queue.java:1156)
      at hudson.model.Executor$1.call(Executor.java:352)
      at hudson.model.Executor$1.call(Executor.java:338)
      at hudson.model.Queue._withLock(Queue.java:1401)
      at hudson.model.Queue.withLock(Queue.java:1262)
      at hudson.model.Executor.run(Executor.java:338)

      Attachments

        Issue Links

          Activity

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/jenkins/advancedqueue/ItemTransitionLogger.java
            src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java
            src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java
            src/main/java/jenkins/advancedqueue/sorter/SorterStrategy.java
            http://jenkins-ci.org/commit/priority-sorter-plugin/35c7a2286b4e680486ec241b504e1cdf745a89bb
            Log:
            JENKINS-44014 - Prevent NullPointerException in AdvancedQueueSorter#onLeft()

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/jenkins/advancedqueue/ItemTransitionLogger.java src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java src/main/java/jenkins/advancedqueue/sorter/SorterStrategy.java http://jenkins-ci.org/commit/priority-sorter-plugin/35c7a2286b4e680486ec241b504e1cdf745a89bb Log: JENKINS-44014 - Prevent NullPointerException in AdvancedQueueSorter#onLeft()

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            .gitignore
            src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java
            src/main/java/jenkins/advancedqueue/priority/strategy/UpstreamCauseStrategy.java
            src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java
            src/main/java/jenkins/advancedqueue/sorter/ItemInfo.java
            src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java
            src/main/java/jenkins/advancedqueue/sorter/StartedJobItemCache.java
            src/test/java/jenkins/advancedqueue/test/UpstreamTest.java
            src/test/java/jenkins/advancedqueue/testutil/JobHelper.java
            src/test/resources/jenkins/advancedqueue/test/UpstreamTest/config.xml
            src/test/resources/jenkins/advancedqueue/test/UpstreamTest/jenkins.advancedqueue.PriorityConfiguration.xml
            src/test/resources/jenkins/advancedqueue/test/UpstreamTest/jenkins.advancedqueue.PrioritySorterConfiguration.xml
            http://jenkins-ci.org/commit/priority-sorter-plugin/a564afc71cb7cc3abb157a4a75bd73dd206dfc9e
            Log:
            Merge commit 'ca2ba818d4c9b17533d9b550883a1064354f6f1b' into JENKINS-44014

            Conflicts:
            src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java
            src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: .gitignore src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java src/main/java/jenkins/advancedqueue/priority/strategy/UpstreamCauseStrategy.java src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java src/main/java/jenkins/advancedqueue/sorter/ItemInfo.java src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java src/main/java/jenkins/advancedqueue/sorter/StartedJobItemCache.java src/test/java/jenkins/advancedqueue/test/UpstreamTest.java src/test/java/jenkins/advancedqueue/testutil/JobHelper.java src/test/resources/jenkins/advancedqueue/test/UpstreamTest/config.xml src/test/resources/jenkins/advancedqueue/test/UpstreamTest/jenkins.advancedqueue.PriorityConfiguration.xml src/test/resources/jenkins/advancedqueue/test/UpstreamTest/jenkins.advancedqueue.PrioritySorterConfiguration.xml http://jenkins-ci.org/commit/priority-sorter-plugin/a564afc71cb7cc3abb157a4a75bd73dd206dfc9e Log: Merge commit 'ca2ba818d4c9b17533d9b550883a1064354f6f1b' into JENKINS-44014 Conflicts: src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java
            http://jenkins-ci.org/commit/priority-sorter-plugin/ed9db551ddba80ccb3ec26e39b29f808f9fc16c8
            Log:
            JENKINS-44014 - Also set the default priority for non-Job ownerTasks

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java http://jenkins-ci.org/commit/priority-sorter-plugin/ed9db551ddba80ccb3ec26e39b29f808f9fc16c8 Log: JENKINS-44014 - Also set the default priority for non-Job ownerTasks

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/jenkins/advancedqueue/ItemTransitionLogger.java
            src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java
            src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java
            src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java
            src/main/java/jenkins/advancedqueue/sorter/SorterStrategy.java
            http://jenkins-ci.org/commit/priority-sorter-plugin/d39a17b4026b178085d4434cc02ce8ef4ce5792b
            Log:
            Merge pull request #39 from oleg-nenashev/JENKINS-44014

            JENKINS-44014 - Fix the reported runtime exceptions (Pipeline integration)

            Compare: https://github.com/jenkinsci/priority-sorter-plugin/compare/ca2ba818d4c9...d39a17b4026b

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/jenkins/advancedqueue/ItemTransitionLogger.java src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java src/main/java/jenkins/advancedqueue/sorter/SorterStrategy.java http://jenkins-ci.org/commit/priority-sorter-plugin/d39a17b4026b178085d4434cc02ce8ef4ce5792b Log: Merge pull request #39 from oleg-nenashev/ JENKINS-44014 JENKINS-44014 - Fix the reported runtime exceptions (Pipeline integration) Compare: https://github.com/jenkinsci/priority-sorter-plugin/compare/ca2ba818d4c9...d39a17b4026b
            oleg_nenashev Oleg Nenashev added a comment -

            The fix has been released in 3.5.1

            oleg_nenashev Oleg Nenashev added a comment - The fix has been released in 3.5.1

            People

              oleg_nenashev Oleg Nenashev
              splashnenen Alexandre Aubert
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: