Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26228

Allow for minimum time to live on item in queue, to ensure api consumers can connect a queue id to a job id

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • core
    • None

      Currently when you start a job using the api, you do get back the queue location url. This is great. But the problem is that requires me to keep checking until a build number get's assigned. I don't know what the time to live is on the queue id, but it seem short lived. So if I happen to miss it, the connection between the queue id and the job is gone forever.
      What I think would be a good idea is to associated the queue id with the build id. So basically, even if the item isn't in the queue anymore, I can still lookup the job via the queue id. Or allow for a setting for min time to live on the queue. Thoughts?

      Relates to JENKINS-12827

          [JENKINS-26228] Allow for minimum time to live on item in queue, to ensure api consumers can connect a queue id to a job id

          Daniel Beck added a comment -

          Thanks for filing an independent issue!

          Daniel Beck added a comment - Thanks for filing an independent issue!

          Daniel Beck added a comment -

          FWIW it's a hard-coded 300 seconds as per
          https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Queue.java#L199

          I have some difficulty imagining this needing to be longer, but it's certainly doable, at least via the usual System property approach for Options Nobody Should Ever Need Unless In Really Weird Situations:
          https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties

          Would that be acceptable?

          Daniel Beck added a comment - FWIW it's a hard-coded 300 seconds as per https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/model/Queue.java#L199 I have some difficulty imagining this needing to be longer, but it's certainly doable, at least via the usual System property approach for Options Nobody Should Ever Need Unless In Really Weird Situations: https://wiki.jenkins-ci.org/display/JENKINS/Features+controlled+by+system+properties Would that be acceptable?

          Daniel Beck added a comment -

          Note that a balance needs to be struck between availability for outside clients and the memory requirements for keeping data around nobody needs on very busy instances. Five minutes seems like a sane default, and a viable choice for most clients (making it far too obscure for a UI option IMO).

          Daniel Beck added a comment - Note that a balance needs to be struck between availability for outside clients and the memory requirements for keeping data around nobody needs on very busy instances. Five minutes seems like a sane default, and a viable choice for most clients (making it far too obscure for a UI option IMO).

          Thanks for the quick response. 300 seconds does sound decent. What do you think about storing the queue id with the job id? And allow for lookup based on that? Then there isn't any issue of ever loosing the job from an outside cosumer.

          Patrick Waters added a comment - Thanks for the quick response. 300 seconds does sound decent. What do you think about storing the queue id with the job id? And allow for lookup based on that? Then there isn't any issue of ever loosing the job from an outside cosumer.

          Daniel Beck added a comment -

          My opinion? It appears to solve a problem that does not really exist with a bunch of complexity that needs to be documented and maintained.

          So I think there's no point in adding it, certainly not in core. (Allowing to tweak a constant as long as it continues to be used via System property is low impact though, so that could be done.)

          A plugin OTOH, and it's certainly easily possible by implementing a QueueListener that records any onLeft(Queue.LeftItem), would be a possible solution for people in a situation such as yours.

          Daniel Beck added a comment - My opinion? It appears to solve a problem that does not really exist with a bunch of complexity that needs to be documented and maintained. So I think there's no point in adding it, certainly not in core. (Allowing to tweak a constant as long as it continues to be used via System property is low impact though, so that could be done.) A plugin OTOH, and it's certainly easily possible by implementing a QueueListener that records any onLeft(Queue.LeftItem) , would be a possible solution for people in a situation such as yours.

          Thanks for the insight. I'm just getting into the jenkins world. I am not in dire need of this, it was just a thought that came to mind. I am getting around it by adding a parameter with a uuid that I can more reliably track.

          I do think that this would be a core feature, not a plugin. It deals with core components and core apis. Any user who wants to build a job tracking system using the API's will run into the same challenges. I think having something like this would solve that issue.

          Patrick Waters added a comment - Thanks for the insight. I'm just getting into the jenkins world. I am not in dire need of this, it was just a thought that came to mind. I am getting around it by adding a parameter with a uuid that I can more reliably track. I do think that this would be a core feature, not a plugin. It deals with core components and core apis. Any user who wants to build a job tracking system using the API's will run into the same challenges. I think having something like this would solve that issue.

          Daniel Beck added a comment -

          I am getting around it by adding a parameter with a uuid that I can more reliably track.

          Could you please explain why you'd prefer this over a longer retention period for left queue items (if it's more than just immediate availability)? Maybe I'm missing something here, like an unstated requirement?


          I do think that this would be a core feature, not a plugin. It deals with core components and core apis.

          A lot of plugins deal with core components and API, often providing rather simple functional additions. Actually, there's a strong trend to take everything non-essential out of core (e.g. Windows Slaves Plugin, the 'Safe HTML' markup formatter, the Matrix Project type, and the JUnit post-build step in the last year alone). So from that POV, adding a whole new (somewhat redundant) API endpoint for your use case falls on the 'plugin' side. Only my personal view of course, but based on past experience.

          Daniel Beck added a comment - I am getting around it by adding a parameter with a uuid that I can more reliably track. Could you please explain why you'd prefer this over a longer retention period for left queue items (if it's more than just immediate availability)? Maybe I'm missing something here, like an unstated requirement? I do think that this would be a core feature, not a plugin. It deals with core components and core apis. A lot of plugins deal with core components and API, often providing rather simple functional additions. Actually, there's a strong trend to take everything non-essential out of core (e.g. Windows Slaves Plugin, the 'Safe HTML' markup formatter, the Matrix Project type, and the JUnit post-build step in the last year alone ). So from that POV, adding a whole new (somewhat redundant) API endpoint for your use case falls on the 'plugin' side. Only my personal view of course, but based on past experience.

          This is not just for my use.

          I am building a devops website for handling deployments and maintenance actions for around 2000 websites. I am building a UI for launching tasks that makes more sense to our staff, rather than the jenkins UI. So simply put, I have button that trigger a jenkins job, then I track that job in my system, and provide light information of jobs to the user.

          Anyone who wants to launch jobs via the api and track them will run into this issue. Here is a bunch of other with a similar issues on stack overflow http://stackoverflow.com/questions/18699555/retrieve-id-of-jenkins-build-started-with-the-api-using-the-location-informati

          Patrick Waters added a comment - This is not just for my use. I am building a devops website for handling deployments and maintenance actions for around 2000 websites. I am building a UI for launching tasks that makes more sense to our staff, rather than the jenkins UI. So simply put, I have button that trigger a jenkins job, then I track that job in my system, and provide light information of jobs to the user. Anyone who wants to launch jobs via the api and track them will run into this issue. Here is a bunch of other with a similar issues on stack overflow http://stackoverflow.com/questions/18699555/retrieve-id-of-jenkins-build-started-with-the-api-using-the-location-informati

          Daniel Beck added a comment -

          As long as you're querying each pending job's queue item once every five minutes, that should work without a problem. Alternatively, write a plugin that provides the specific aggregate data you're interested in. Please correct me if I'm wrong, but it doesn't appear to be something that cannot be solved reliably with the current APIs (possibly with making the queue item lifetime configurable to handle longer outages of the custom UI and/or network connection between the two).

          Regarding the linked question, it looks like the user would be happy if he got the queue item URL to query, but a few things prevent that in that specific case AFAICT:

          • The user uses the wrong URL for a parameterized build
          • The client is pretending to be a web browser using Structured Form Submission rather than submitting plain form parameters
          • The behavior of buildWithParameters changed in 1.561 and the question is about 1.529.

          Daniel Beck added a comment - As long as you're querying each pending job's queue item once every five minutes, that should work without a problem. Alternatively, write a plugin that provides the specific aggregate data you're interested in. Please correct me if I'm wrong, but it doesn't appear to be something that cannot be solved reliably with the current APIs (possibly with making the queue item lifetime configurable to handle longer outages of the custom UI and/or network connection between the two). — Regarding the linked question, it looks like the user would be happy if he got the queue item URL to query, but a few things prevent that in that specific case AFAICT: The user uses the wrong URL for a parameterized build The client is pretending to be a web browser using Structured Form Submission rather than submitting plain form parameters The behavior of buildWithParameters changed in 1.561 and the question is about 1.529.

          Dell Green added a comment - - edited

          Polling repeatedly, within a time sensitive window is not a great solution when integrating with other large systems and large build queues. Its expected that these other systems may persist a queue reference/receipt and then be paused/taken offline for a period of time. When they are brought back on-line, it's not unreasonable that the reference they hold should be able to be still resolved in some sensible way to update their state.

          Maybe using a combination  of tree and xpath url parameters with jobname to query only builds containing the queueID of interest could be used as a workaround for now

           

           http://localhost/jenkins/job/MyJob/api/xml?tree=builds[queueId,result]&xpath=//build[queueId=1234][1]
           
          

           

          note: this will require more server-side processing/memory

           

          Dell Green added a comment - - edited Polling repeatedly, within a time sensitive window is not a great solution when integrating with other large systems and large build queues. Its expected that these other systems may persist a queue reference/receipt and then be paused/taken offline for a period of time. When they are brought back on-line, it's not unreasonable that the reference they hold should be able to be still resolved in some sensible way to update their state. Maybe using a combination  of tree and xpath url parameters with jobname to query only builds containing the queueID of interest could be used as a workaround for now    http://localhost/jenkins/job/MyJob/api/xml?tree=builds[queueId,result]&xpath=//build[queueId=1234][1]     note: this will require more server-side processing/memory  

            Unassigned Unassigned
            pwaterz Patrick Waters
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: