Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-32266

Priority is not respected for downstream jobs triggered automatically

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Not A Defect
    • Component/s: prioritysorter-plugin
    • Labels:
      None
    • Environment:
      Jenkins ver. 1.641
      Priority Sorter Plugin 3.4
    • Similar Issues:

      Description

      Description

      If jobs are added to Build Queue by starting them manually, they are added to the list in the required order (according to priorities). However, this is not the case when jobs are triggered automatically based on upstream/downstream relationship - they are re-sorted only after some observable timeout:

      • If new job is added to Build Queue by triggering it from upstream one, the new job stays on top of Build Queue job list (the last to run) even if its priority is higher (priority number is smaller) than any other job in the Build Queue. Refreshing the main page continuously shows the same job list order.
      • After some timeout, refreshing the main page finally re-sorts the job list in the Build Queue according to priorities.

      In order to easily observe this behaviour, use quiet down mode ("Manage Jenkins" => "Prepare for Shutdown") to keep jobs in the Build Queue without running them.

      Steps to reproduce

      • Create 2 priority groups:
        high_priority_group (e.g. priority number = 3)
        low_prirority_group (e.g. priority number = 5)
      • Create 3 jobs:
        low_a (priority = low_priority_group)
        high_b (priority = high_priority_group)
        high_c (priority = high_priority_group)
      • Configure job high_c as downstream of high_b. In other words, when high_b competes, high_c is supposed to be started.
      • Set quiet down mode ("Manage Jenkins" => "Prepare for Shutdown").
      • Schedule both low_a and high_b jobs in any order manually. Build Queue will list low_a first and high_b second as expected (high_b to be run first). Refresh Jenkins main page to confirm this.
      • Cancel quiet down mode for a while to get high_b job started. Set quiet down mode again to be able to see Build Queue sorting mode when high_c is triggered automatically.
      • As soon as high_c is triggered automatically, it is first listed at the top of the Build Queue (in a wrong order) which says low_a is to be run first. Refresh Jenkins main page to confirm this.
      • After a while refreshing Jenkins main page re-sorts jobs in the Build Queue to correct order.

      The problem is that without quiet down mode, the low_a job with lower priority will start ahead of high_c job (which is triggered automatically by upstream high_b job).

        Attachments

          Issue Links

            Activity

            Hide
            emsa23 Magnus Sandberg added a comment -

            A few things to note:

            • the queue ui is updated on a timer this means that what you see there is not always the truth (use the logging to see reality)
            • there's a delay from when items are added to the queue until they are sorted (things can [will] be removes/scheduled from the queue before new items are sorted)
            • items are added to the queue and sorted the same way regardless how they where triggered (priority sorter does not know util looking at the item to be sorted, during the sorting)

            In any case you can never rely on that jobs are always started in the priority order that depends on other (inner) workings of Jenkins such as when scheduling, starts and sorting happens. If you need jobs to run in a specific order use other means to make sure that the order is respected.

            If you like to see exactly how the queue looks, when and how the queue is sorted please turn on logging as described on the wiki - if you find something wrong with the sorting please provide relevant log pieces in the issue.

            Show
            emsa23 Magnus Sandberg added a comment - A few things to note: the queue ui is updated on a timer this means that what you see there is not always the truth (use the logging to see reality) there's a delay from when items are added to the queue until they are sorted (things can [will] be removes/scheduled from the queue before new items are sorted) items are added to the queue and sorted the same way regardless how they where triggered (priority sorter does not know util looking at the item to be sorted, during the sorting) In any case you can never rely on that jobs are always started in the priority order that depends on other (inner) workings of Jenkins such as when scheduling, starts and sorting happens. If you need jobs to run in a specific order use other means to make sure that the order is respected. If you like to see exactly how the queue looks, when and how the queue is sorted please turn on logging as described on the wiki - if you find something wrong with the sorting please provide relevant log pieces in the issue.
            Hide
            uvsmtid Alexey Pakseykin added a comment - - edited

            Magnus, thank you for helpful detailed comment! It clarified lots of observations.

            The most important takeaway which actually affects execution:

            • There's a delay from when items are added to the queue until they are sorted...
              In other words, even if two tasks are queued, there is a time window when they can be started against the order set by their different priorities.

            It would add usefulness and wider applicability to the plugin if this issue could be avoided:

            • In any case you can never rely on that jobs are always started in the priority order that depends on other (inner) workings of Jenkins... If you need jobs to run in a specific order use other means to make sure that the order is respected.
              I agree that ordering and priorities are two different concepts. However, in my case I actually try to enforce priority - jobs with higher priority must never be preempted by jobs of lower priority (provided that they both were queued at the time when one of them was started).
              My point is that if this limitation stays true, it largely limits purposes for which plugin can be used as guarantees are loose.

            As you suggested other means, I want to bring up example of Build Blocker Plugin:
            https://wiki.jenkins-ci.org/display/JENKINS/Build+Blocker+Plugin
            I use it to block builds inside build queue until there is specific job running OR queued.
            Based on your expertise, do you think it is possible to somehow make sure that jobs are blocked inside build queue until they get sorted?
            In other words, jobs should never be started while they are in unsorted order (and should stay blocked in the build queue).

            There is some incentive of functionality in it because Build Blocker Plugin is very difficult to configure to imitate priorities - it blocks jobs based on single regex and it is complex to express in that regex what jobs are higher and what lower in priority based on their name only.

            Show
            uvsmtid Alexey Pakseykin added a comment - - edited Magnus, thank you for helpful detailed comment! It clarified lots of observations. The most important takeaway which actually affects execution: There's a delay from when items are added to the queue until they are sorted... In other words, even if two tasks are queued, there is a time window when they can be started against the order set by their different priorities. It would add usefulness and wider applicability to the plugin if this issue could be avoided: In any case you can never rely on that jobs are always started in the priority order that depends on other (inner) workings of Jenkins... If you need jobs to run in a specific order use other means to make sure that the order is respected. I agree that ordering and priorities are two different concepts. However, in my case I actually try to enforce priority - jobs with higher priority must never be preempted by jobs of lower priority (provided that they both were queued at the time when one of them was started). My point is that if this limitation stays true, it largely limits purposes for which plugin can be used as guarantees are loose. As you suggested other means, I want to bring up example of Build Blocker Plugin: https://wiki.jenkins-ci.org/display/JENKINS/Build+Blocker+Plugin I use it to block builds inside build queue until there is specific job running OR queued . Based on your expertise, do you think it is possible to somehow make sure that jobs are blocked inside build queue until they get sorted? In other words, jobs should never be started while they are in unsorted order (and should stay blocked in the build queue). There is some incentive of functionality in it because Build Blocker Plugin is very difficult to configure to imitate priorities - it blocks jobs based on single regex and it is complex to express in that regex what jobs are higher and what lower in priority based on their name only.
            Hide
            emsa23 Magnus Sandberg added a comment -

            I again urge you to turn on logging as that will very clearly show when items enter and leaves the queue and when sorting happen.

            No items can get started without being sorted so no nothing is ever started against order in regards to items already in the queue.

            The delay I refer to is the delay from the job being triggered until it enters the queue, during that time jobs already in the queue will most likely be started.

            And no the purpose of the plugin is not to enforce order (pipelining) is to make sure more important jobs are run before less important ones, this is valuable when you have hundreds or thousand of jobs in the queue.

            Show
            emsa23 Magnus Sandberg added a comment - I again urge you to turn on logging as that will very clearly show when items enter and leaves the queue and when sorting happen. No items can get started without being sorted so no nothing is ever started against order in regards to items already in the queue. The delay I refer to is the delay from the job being triggered until it enters the queue, during that time jobs already in the queue will most likely be started. And no the purpose of the plugin is not to enforce order (pipelining) is to make sure more important jobs are run before less important ones, this is valuable when you have hundreds or thousand of jobs in the queue.
            Hide
            uvsmtid Alexey Pakseykin added a comment -

            Magnus, I enabled log directly in our CI (so job names are different from those in task description). The conditions are essentially the same.

            The log is attached. And I also commented sequence of events (below).

            Correct me if I'm wrong. What I noticed is that PrioritySorter affects jobs in BUILDABLE status only. Those jobs which are in WAITING mode do not appear in the Queue table until their status changes to BUILDABLE. The confusion comes from the fact that jobs with (both) BUILDABLE and WAITING status appear in the Build Queue in the UI (left side bar on Jenkins default main page).

            If so, this is the limitation I hit. Ultimately, the problem is that BUILDABLE jobs may have lower priority (higher value) compared to WAITING jobs. If WAITING jobs are not taken into account, BUILDABLE ones jump ahead into execution.
            Is it possible to ask PrioritySorter sort BUILDABLE and WAITING combined together?


            These are my comments to attached logs. The priorities were adjusted so that jobs with lower number cannot preempt sequence of jobs with higher number.

            Start of the log. Job 02.03 is added to queue.

            Log line 002: Jan 19, 2016 3:07:31 PM Buildable: Id: 31029, JobName: 02.03.update_pipeline.run_salt_highstate, jobGroupId: 7, reason: <none>, priority: 7, weight: 7.0, status: BUILDABLE
            Log line 009: Table shows it is the only one in the queue.

            The job is started immediately.

            3 minutes later. Job 00.01 was added (by timer).
            Log line 014: Jan 19, 2016 3:10:00 PM New Item: Id: 31030, JobName: 00.01.poll_pipeline.verify_approval, jobGroupId: 9, reason: <none>, priority: 9, weight: 9.0, status: WAITING

            The job waits as there is already 02.03 building.

            3 seconds later. Job 02.04 is added (triggered by completion of 02.03).
            Log line 125: Jan 19, 2016 3:10:30 PM New Item: Id: 31031, JobName: 02.04.update_pipeline.reconnect_jenkins_slaves, jobGroupId: 7, reason: <none>, priority: 7, weight: 7.0, status: WAITING
            Log line 169: Table still lists only job 00.01 even though 02.04 has just been added.

            NOTE: Why table does not show both 00.01 and 02.04?

            Immediately (at the same timestamp 3:10:30) following the addition of job 02.04, job 00.01 gets started.

            Log line 181: Jan 19, 2016 3:10:30 PM Starting: Id: 31030, JobName: 00.01.poll_pipeline.verify_approval, jobGroupId: 9, reason: <none>, priority: 9, weight: 9.0, status: BUILDABLE

            NOTE: The expectation is that 00.01 will never start if there is pending 02.04. However, 02.04 didn't even appear in the table yet.

            3 seconds later, job 00.02 is added (triggered by completion of 00.01 which was started unexpectedly).

            Log line 237: Job 02.04 is finally displayed in the table. However, it is alone! Job 00.02 (which has just been added) is not in the table.

            Show
            uvsmtid Alexey Pakseykin added a comment - Magnus, I enabled log directly in our CI (so job names are different from those in task description). The conditions are essentially the same. The log is attached. And I also commented sequence of events (below). Correct me if I'm wrong. What I noticed is that PrioritySorter affects jobs in BUILDABLE status only. Those jobs which are in WAITING mode do not appear in the Queue table until their status changes to BUILDABLE. The confusion comes from the fact that jobs with (both) BUILDABLE and WAITING status appear in the Build Queue in the UI (left side bar on Jenkins default main page). If so, this is the limitation I hit. Ultimately, the problem is that BUILDABLE jobs may have lower priority (higher value) compared to WAITING jobs. If WAITING jobs are not taken into account, BUILDABLE ones jump ahead into execution. Is it possible to ask PrioritySorter sort BUILDABLE and WAITING combined together? These are my comments to attached logs. The priorities were adjusted so that jobs with lower number cannot preempt sequence of jobs with higher number. — Start of the log. Job 02.03 is added to queue. Log line 002: Jan 19, 2016 3:07:31 PM Buildable: Id: 31029, JobName: 02.03.update_pipeline.run_salt_highstate, jobGroupId: 7, reason: <none>, priority: 7, weight: 7.0, status: BUILDABLE Log line 009: Table shows it is the only one in the queue. The job is started immediately. — 3 minutes later. Job 00.01 was added (by timer). Log line 014: Jan 19, 2016 3:10:00 PM New Item: Id: 31030, JobName: 00.01.poll_pipeline.verify_approval, jobGroupId: 9, reason: <none>, priority: 9, weight: 9.0, status: WAITING The job waits as there is already 02.03 building. — 3 seconds later. Job 02.04 is added (triggered by completion of 02.03). Log line 125: Jan 19, 2016 3:10:30 PM New Item: Id: 31031, JobName: 02.04.update_pipeline.reconnect_jenkins_slaves, jobGroupId: 7, reason: <none>, priority: 7, weight: 7.0, status: WAITING Log line 169: Table still lists only job 00.01 even though 02.04 has just been added. NOTE: Why table does not show both 00.01 and 02.04? — Immediately (at the same timestamp 3:10:30) following the addition of job 02.04, job 00.01 gets started. Log line 181: Jan 19, 2016 3:10:30 PM Starting: Id: 31030, JobName: 00.01.poll_pipeline.verify_approval, jobGroupId: 9, reason: <none>, priority: 9, weight: 9.0, status: BUILDABLE NOTE: The expectation is that 00.01 will never start if there is pending 02.04. However, 02.04 didn't even appear in the table yet. — 3 seconds later, job 00.02 is added (triggered by completion of 00.01 which was started unexpectedly). Log line 237: Job 02.04 is finally displayed in the table. However, it is alone! Job 00.02 (which has just been added) is not in the table.
            Hide
            emsa23 Magnus Sandberg added a comment -

            WAITING jobs are not in the queue, they are shown at the top (last) of the queue i the ui but they are not eligible to get started.
            BUILDABLE are jobs that can be started so they are in the queue.

            The queue being printed contains the jobs that Jenkins tells the plugin are in the queue at the time of printing.

            Even if I show WAITING jobs in the sorting (same goes for BLOCKED jobs) they are not still not eligible to get started and not considered by the scheduler.

            The only thing that (potentially) can be done is to block all jobs from being started while any job is WAITING but I would not, in the general case, think that that is a particular good idea.

            Show
            emsa23 Magnus Sandberg added a comment - WAITING jobs are not in the queue, they are shown at the top (last) of the queue i the ui but they are not eligible to get started. BUILDABLE are jobs that can be started so they are in the queue. The queue being printed contains the jobs that Jenkins tells the plugin are in the queue at the time of printing. Even if I show WAITING jobs in the sorting (same goes for BLOCKED jobs) they are not still not eligible to get started and not considered by the scheduler. The only thing that (potentially) can be done is to block all jobs from being started while any job is WAITING but I would not, in the general case, think that that is a particular good idea.
            Hide
            uvsmtid Alexey Pakseykin added a comment -

            OK, at least I reached clear understanding in what was idea behind the current implementation.
            BUILDABLE/WAITING status also makes exact practical sense separation between ordering and priorities now.

            I simply expected that PrioritySorter does not let any job with lower priority run if there is anything in the Jenkins UI queue with higher priority.

            Actually, the Build Blocker plugin has related two options (text was copied&pasted):

            • "check buildable queued builds" = "consider ready to run builds of the defined jobs for the blocking decision"
            • "check all queued builds" = "consider all builds of the defined jobs for the blocking decision, including blocked, waiting, pending and buildable"

            The solution within PrioritySorter would be option similar to the second one - considering priorities of all builds ("including blocked, waiting, pending and buildable"). However, this would mean that PrioritySorter need to block builds if higher priority (lower value) jobs are not BUILDABLE for any reasons.

            I guess my immediate action is to use something else for now.

            Show
            uvsmtid Alexey Pakseykin added a comment - OK, at least I reached clear understanding in what was idea behind the current implementation. BUILDABLE/WAITING status also makes exact practical sense separation between ordering and priorities now. I simply expected that PrioritySorter does not let any job with lower priority run if there is anything in the Jenkins UI queue with higher priority. Actually, the Build Blocker plugin has related two options (text was copied&pasted): "check buildable queued builds" = "consider ready to run builds of the defined jobs for the blocking decision" "check all queued builds" = "consider all builds of the defined jobs for the blocking decision, including blocked, waiting, pending and buildable" The solution within PrioritySorter would be option similar to the second one - considering priorities of all builds ("including blocked, waiting, pending and buildable"). However, this would mean that PrioritySorter need to block builds if higher priority (lower value) jobs are not BUILDABLE for any reasons. I guess my immediate action is to use something else for now.
            Hide
            emsa23 Magnus Sandberg added a comment -

            As I said above it is potentially possible to do something like that but that will result in blocked jobs and unused workers and the point of the plugin is to maximize the usage of the workers for prioritized and important jobs not to enforce strict order.

            Adding checks on waiting jobs will not solve all problems for you. As you can see in your logs the job in the queue can for example be started before the the new job even becomes waiting. Also consider the case when you have many workers that will build in parallell even if the jobs are in the 'correct' order they may be started almost at the same time on different workers.

            The best way to solve the order issue where one jobs needs to be done before the next starts is not to trigger the next jobs until it is allowed to run and/or to use pipeline/workflow plugins available and block downstream jobs until they are allowed to run.

            Show
            emsa23 Magnus Sandberg added a comment - As I said above it is potentially possible to do something like that but that will result in blocked jobs and unused workers and the point of the plugin is to maximize the usage of the workers for prioritized and important jobs not to enforce strict order. Adding checks on waiting jobs will not solve all problems for you. As you can see in your logs the job in the queue can for example be started before the the new job even becomes waiting. Also consider the case when you have many workers that will build in parallell even if the jobs are in the 'correct' order they may be started almost at the same time on different workers. The best way to solve the order issue where one jobs needs to be done before the next starts is not to trigger the next jobs until it is allowed to run and/or to use pipeline/workflow plugins available and block downstream jobs until they are allowed to run.
            Hide
            emsa23 Magnus Sandberg added a comment -

            Closing this issue as this is not a defect the plugin works as expected.

            If you like a change to the plugin please create a new feature request describing the feature you would like to see.

            Show
            emsa23 Magnus Sandberg added a comment - Closing this issue as this is not a defect the plugin works as expected. If you like a change to the plugin please create a new feature request describing the feature you would like to see.

              People

              Assignee:
              emsa23 Magnus Sandberg
              Reporter:
              uvsmtid Alexey Pakseykin
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: