Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44014

job triggering and queue broken since update to 2.57

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • prioritysorter-plugin
    • None
    • jenkins 2.58

      Since update to jenkins 2.57 (and still after update to 2.58), there is a lot of delay when clicking on 'build' button for a job. It could reach 5 minutes before really starting the build.

      Queue handling seems also to be altered (jobs don't come in queue rapidely)

      And the job triggering by upstream job is also really slow, reaching sometimes internal 'timeout'.

      What i can get from logs is (not sure which of those could be really related but it seems that those errors are linked together) :

      • NPE SyncQueueListener

      mai 03, 2017 12:42:02 PM org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener publish
      WARNING: Error publishing Run queued event.
      org.jenkinsci.plugins.pubsub.MessageException: Event enrichment failure due to unexpected exception in io.jenkins.blueocean.events.BlueMessageEnricher.
      at org.jenkinsci.plugins.pubsub.PubsubBus.publish(PubsubBus.java:113)
      at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.publish(SyncQueueListener.java:178)
      at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.publish(SyncQueueListener.java:172)
      at org.jenkinsci.plugins.pubsub.listeners.SyncQueueListener.onEnterWaiting(SyncQueueListener.java:139)
      at hudson.model.Queue$WaitingItem.enter(Queue.java:2392)
      at hudson.model.Queue.scheduleInternal(Queue.java:633)
      at hudson.model.Queue.schedule2(Queue.java:589)
      at jenkins.model.ParameterizedJobMixIn.scheduleBuild2(ParameterizedJobMixIn.java:138)
      at jenkins.model.ParameterizedJobMixIn.scheduleBuild(ParameterizedJobMixIn.java:94)
      at hudson.model.AbstractProject.scheduleBuild(AbstractProject.java:838)
      at hudson.triggers.TimerTrigger.run(TimerTrigger.java:67)
      at hudson.triggers.Trigger.checkTriggers(Trigger.java:278)
      at hudson.triggers.Trigger$Cron.doRun(Trigger.java:226)
      at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:51)
      at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
      Caused by: java.lang.NullPointerException

      • Warnings on AdvancedQueueSorterQueueListener

      mai 03, 2017 8:18:37 AM jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener onEnterBuildable
      WARNING: onEnterBuilding() called without prior call to onEnterWaiting() for 'API-availability-dev'

      • Warnings on hudson.model.Queue$WaitingItem enter

      mai 03, 2017 8:25:11 AM hudson.model.Queue$WaitingItem enter
      WARNING: QueueListener failed while processing hudson.model.Queue$WaitingItem:ExecutorStepExecution.PlaceholderTask

      {runId=Certifications/Sparktacus/Development_Sparktacus#12,label=Certif-BDXBLD006,context=CpsStepContext[3:null]:OwnerCertifications/Sparktacus/Development_Sparktacus/12:Certifications/Sparktacus/Development_Sparktacus #12,cookie=8d0b6bd9-e673-409f-9234-2279205b290b}

      :69158
      java.lang.ClassCastException: org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask cannot be cast to hudson.model.Job
      at jenkins.advancedqueue.PriorityConfigurationPlaceholderTaskHelper.getPriority(PriorityConfigurationPlaceholderTaskHelper.java:19)
      at jenkins.advancedqueue.PriorityConfiguration.getPriorityInternal(PriorityConfiguration.java:213)
      at jenkins.advancedqueue.PriorityConfiguration.getPriority(PriorityConfiguration.java:205)
      at jenkins.advancedqueue.sorter.AdvancedQueueSorter.onNewItem(AdvancedQueueSorter.java:136)
      at jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onEnterWaiting(AdvancedQueueSorterQueueListener.java:46)
      at hudson.model.Queue$WaitingItem.enter(Queue.java:2392)
      at hudson.model.Queue.scheduleInternal(Queue.java:633)
      at hudson.model.Queue.schedule2(Queue.java:589)
      at hudson.model.Queue.schedule2(Queue.java:709)
      at org.jenkinsci.plugins.workflow.support.pickles.ExecutorPickle$1.tryResolve(ExecutorPickle.java:79)
      at org.jenkinsci.plugins.workflow.support.pickles.ExecutorPickle$1.tryResolve(ExecutorPickle.java:73)
      at org.jenkinsci.plugins.workflow.support.pickles.TryRepeatedly$1.run(TryRepeatedly.java:92)
      at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

      • NPE on hudson.model.Queue$LeftItem enter

      mai 03, 2017 8:36:03 AM hudson.model.Queue$LeftItem enter
      WARNING: QueueListener failed while processing hudson.model.Queue$LeftItem:ExecutorStepExecution.PlaceholderTask

      {runId=Certifications/Sparktacus/Development_Sparktacus#12,label=Certif-BDXBLD006,context=CpsStepContext[3:null]:OwnerCertifications/Sparktacus/Development_Sparktacus/12:Certifications/Sparktacus/Development_Sparktacus #12,cookie=8d0b6bd9-e673-409f-9234-2279205b290b}

      :69158
      java.lang.NullPointerException
      at jenkins.advancedqueue.sorter.AdvancedQueueSorter.onLeft(AdvancedQueueSorter.java:145)
      at jenkins.advancedqueue.sorter.AdvancedQueueSorterQueueListener.onLeft(AdvancedQueueSorterQueueListener.java:51)
      at hudson.model.Queue$LeftItem.enter(Queue.java:2660)
      at hudson.model.Queue.onStartExecuting(Queue.java:1156)
      at hudson.model.Executor$1.call(Executor.java:352)
      at hudson.model.Executor$1.call(Executor.java:338)
      at hudson.model.Queue._withLock(Queue.java:1401)
      at hudson.model.Queue.withLock(Queue.java:1262)
      at hudson.model.Executor.run(Executor.java:338)

          [JENKINS-44014] job triggering and queue broken since update to 2.57

          Daniel Wilmer added a comment -

          After updating to 2.62 I also have this error:

          java.lang.ClassCastException: org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask cannot be cast to hudson.model.Job

          I currently cannot run any of our pipelines.

          Daniel Wilmer added a comment - After updating to 2.62 I also have this error: java.lang.ClassCastException: org.jenkinsci.plugins.workflow.support.steps.ExecutorStepExecution$PlaceholderTask cannot be cast to hudson.model.Job I currently cannot run any of our pipelines.

          After a whole block of our continuous integration because of this issue, i think i've found the root cause of the problem. Here is what i've done :

          • fresh install of last LTS version (2.46.3)
          • reinstall proper plugins and reimport jobs (configs only)

          and then i faced the same problem after a restart of jenkins master has been done while a multibranch pipeline was running.

          I noticed that a 'state.xml' file had been added in the job folder, next to the config.xml. I think it's done by workfolw-durable-task plugin which can't be uninstalled if we want to use multibranch pipelines.

          Then at restart, multibranch pipeline tried to replay ongoing step before the jenkins stop. And this hangs the whole mechanism of build triggering and queue handling. We have a message in multibranch pipeline console log saying 'trying to resume part of job #Build_number after jenkins restart'....

          this is the problem freezing all. Removing the state.xml file and restarting jenkins has solved the problem and all jobs are now well triggered.

          Could you please investigate on that ?

          Alexandre Aubert added a comment - After a whole block of our continuous integration because of this issue, i think i've found the root cause of the problem. Here is what i've done : fresh install of last LTS version (2.46.3) reinstall proper plugins and reimport jobs (configs only) and then i faced the same problem after a restart of jenkins master has been done while a multibranch pipeline was running. I noticed that a 'state.xml' file had been added in the job folder, next to the config.xml. I think it's done by workfolw-durable-task plugin which can't be uninstalled if we want to use multibranch pipelines. Then at restart, multibranch pipeline tried to replay ongoing step before the jenkins stop. And this hangs the whole mechanism of build triggering and queue handling. We have a message in multibranch pipeline console log saying 'trying to resume part of job #Build_number after jenkins restart'.... this is the problem freezing all. Removing the state.xml file and restarting jenkins has solved the problem and all jobs are now well triggered. Could you please investigate on that ?

          Jesse Glick added a comment -

          Nothing in Pipeline that I am aware of writes a state.xml file. And the stack trace is clearly a bug in prioritysorter. Beyond that, I know nothing.

          Jesse Glick added a comment - Nothing in Pipeline that I am aware of writes a state.xml file. And the stack trace is clearly a bug in prioritysorter . Beyond that, I know nothing.

          Oleg Nenashev added a comment -

          Yes, needs a fix in Priority sorter

          Oleg Nenashev added a comment - Yes, needs a fix in Priority sorter

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java
          http://jenkins-ci.org/commit/priority-sorter-plugin/932179ea56270c5723556b8b13de2c0a2d9d72e6
          Log:
          JENKINS-44014 - Prevent class cast exception and add diagnostics if the ownerTask of Pipeline placeholder is not a Job. Multibranch?

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java http://jenkins-ci.org/commit/priority-sorter-plugin/932179ea56270c5723556b8b13de2c0a2d9d72e6 Log: JENKINS-44014 - Prevent class cast exception and add diagnostics if the ownerTask of Pipeline placeholder is not a Job. Multibranch?

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/jenkins/advancedqueue/ItemTransitionLogger.java
          src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java
          src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java
          src/main/java/jenkins/advancedqueue/sorter/SorterStrategy.java
          http://jenkins-ci.org/commit/priority-sorter-plugin/35c7a2286b4e680486ec241b504e1cdf745a89bb
          Log:
          JENKINS-44014 - Prevent NullPointerException in AdvancedQueueSorter#onLeft()

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/jenkins/advancedqueue/ItemTransitionLogger.java src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java src/main/java/jenkins/advancedqueue/sorter/SorterStrategy.java http://jenkins-ci.org/commit/priority-sorter-plugin/35c7a2286b4e680486ec241b504e1cdf745a89bb Log: JENKINS-44014 - Prevent NullPointerException in AdvancedQueueSorter#onLeft()

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          .gitignore
          src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java
          src/main/java/jenkins/advancedqueue/priority/strategy/UpstreamCauseStrategy.java
          src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java
          src/main/java/jenkins/advancedqueue/sorter/ItemInfo.java
          src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java
          src/main/java/jenkins/advancedqueue/sorter/StartedJobItemCache.java
          src/test/java/jenkins/advancedqueue/test/UpstreamTest.java
          src/test/java/jenkins/advancedqueue/testutil/JobHelper.java
          src/test/resources/jenkins/advancedqueue/test/UpstreamTest/config.xml
          src/test/resources/jenkins/advancedqueue/test/UpstreamTest/jenkins.advancedqueue.PriorityConfiguration.xml
          src/test/resources/jenkins/advancedqueue/test/UpstreamTest/jenkins.advancedqueue.PrioritySorterConfiguration.xml
          http://jenkins-ci.org/commit/priority-sorter-plugin/a564afc71cb7cc3abb157a4a75bd73dd206dfc9e
          Log:
          Merge commit 'ca2ba818d4c9b17533d9b550883a1064354f6f1b' into JENKINS-44014

          Conflicts:
          src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java
          src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: .gitignore src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java src/main/java/jenkins/advancedqueue/priority/strategy/UpstreamCauseStrategy.java src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java src/main/java/jenkins/advancedqueue/sorter/ItemInfo.java src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java src/main/java/jenkins/advancedqueue/sorter/StartedJobItemCache.java src/test/java/jenkins/advancedqueue/test/UpstreamTest.java src/test/java/jenkins/advancedqueue/testutil/JobHelper.java src/test/resources/jenkins/advancedqueue/test/UpstreamTest/config.xml src/test/resources/jenkins/advancedqueue/test/UpstreamTest/jenkins.advancedqueue.PriorityConfiguration.xml src/test/resources/jenkins/advancedqueue/test/UpstreamTest/jenkins.advancedqueue.PrioritySorterConfiguration.xml http://jenkins-ci.org/commit/priority-sorter-plugin/a564afc71cb7cc3abb157a4a75bd73dd206dfc9e Log: Merge commit 'ca2ba818d4c9b17533d9b550883a1064354f6f1b' into JENKINS-44014 Conflicts: src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java
          http://jenkins-ci.org/commit/priority-sorter-plugin/ed9db551ddba80ccb3ec26e39b29f808f9fc16c8
          Log:
          JENKINS-44014 - Also set the default priority for non-Job ownerTasks

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java http://jenkins-ci.org/commit/priority-sorter-plugin/ed9db551ddba80ccb3ec26e39b29f808f9fc16c8 Log: JENKINS-44014 - Also set the default priority for non-Job ownerTasks

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/jenkins/advancedqueue/ItemTransitionLogger.java
          src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java
          src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java
          src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java
          src/main/java/jenkins/advancedqueue/sorter/SorterStrategy.java
          http://jenkins-ci.org/commit/priority-sorter-plugin/d39a17b4026b178085d4434cc02ce8ef4ce5792b
          Log:
          Merge pull request #39 from oleg-nenashev/JENKINS-44014

          JENKINS-44014 - Fix the reported runtime exceptions (Pipeline integration)

          Compare: https://github.com/jenkinsci/priority-sorter-plugin/compare/ca2ba818d4c9...d39a17b4026b

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/jenkins/advancedqueue/ItemTransitionLogger.java src/main/java/jenkins/advancedqueue/PriorityConfigurationPlaceholderTaskHelper.java src/main/java/jenkins/advancedqueue/sorter/AdvancedQueueSorter.java src/main/java/jenkins/advancedqueue/sorter/QueueItemCache.java src/main/java/jenkins/advancedqueue/sorter/SorterStrategy.java http://jenkins-ci.org/commit/priority-sorter-plugin/d39a17b4026b178085d4434cc02ce8ef4ce5792b Log: Merge pull request #39 from oleg-nenashev/ JENKINS-44014 JENKINS-44014 - Fix the reported runtime exceptions (Pipeline integration) Compare: https://github.com/jenkinsci/priority-sorter-plugin/compare/ca2ba818d4c9...d39a17b4026b

          Oleg Nenashev added a comment -

          The fix has been released in 3.5.1

          Oleg Nenashev added a comment - The fix has been released in 3.5.1

            oleg_nenashev Oleg Nenashev
            splashnenen Alexandre Aubert
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: