Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38877

Build History by Node doesn't work with Pipeline

      Browse to <jenkinsServer>/computer/<nodeName>/builds to see build history. For pipeline this page seems disabled. This is understandable because a single pipeline might span multiple nodes so showing a simple pass/fail list of pipeline jobs for a node might not make sense. However this has made it hard to tell at a glance when a certain node is failing more often than other nodes.

      I would like to see the build history page for nodes display the pipeline job name and build number and then maybe the stage name and step number or something that is definitely unique to that node. That way we could quickly browse history at the node level to determine if a particular node has problems.

          [JENKINS-38877] Build History by Node doesn't work with Pipeline

          Jesse Glick added a comment -

          The way the build history page is implemented is not compatible with Pipeline. This page should not be used anyway—the design is fundamentally unsound in terms of performance (you can easily bring a large installation to its knees by browsing this page). There are various longer-term plans for analytic capabilities.

          Jesse Glick added a comment - The way the build history page is implemented is not compatible with Pipeline. This page should not be used anyway—the design is fundamentally unsound in terms of performance (you can easily bring a large installation to its knees by browsing this page). There are various longer-term plans for analytic capabilities.

          Thanks for the information.
          Could you provide a link here to some ticket which I could use to track the progress on the planned analytic capabilities?
          Thanks.

          Joerg Schwaerzler added a comment - Thanks for the information. Could you provide a link here to some ticket which I could use to track the progress on the planned analytic capabilities? Thanks.

          Jesse Glick added a comment -

          No such ticket.

          Jesse Glick added a comment - No such ticket.

          Dennis Cheung added a comment -

          Is there a commanded way to check if concurrent running jobs on same node cause some problems/regression?

          In the old freestyle job, there are records on the history page. It is easy to identify overlapping and blame it for performance regression.

          However, same information lost and quite hidden from the Pipeline. You cannot even tell which nodes were used without look into the full log.

          Dennis Cheung added a comment - Is there a commanded way to check if concurrent running jobs on same node cause some problems/regression? In the old freestyle job, there are records on the history page. It is easy to identify overlapping and blame it for performance regression. However, same information lost and quite hidden from the Pipeline. You cannot even tell which nodes were used without look into the full log.

          I agree that it's quite frustrating. It used to be you could at a glance see if a particular node was to blame for repeatedly failing jobs, but now with that information so buried it's not so easy.

          It seems we have to take matters into our own hands. One approach I'm looking into is to write the node name to currentBuild.description so at least when looking at the build history by job you can see the different nodes that ran each build. Better than nothing.

          Justin Rainwater added a comment - I agree that it's quite frustrating. It used to be you could at a glance see if a particular node was to blame for repeatedly failing jobs, but now with that information so buried it's not so easy. It seems we have to take matters into our own hands. One approach I'm looking into is to write the node name to currentBuild.description so at least when looking at the build history by job you can see the different nodes that ran each build. Better than nothing.

          @Build history by job:
          Actually what we did is setting the build's display name. Doing that you even see the overview of used nodes in the stage view plugin. As a plus you can use the build filter provided by Jenkins to search for all builds which ran on a specific node.
          Works quite nice - but unfortunately no replacement for the node's build history.

          Joerg Schwaerzler added a comment - @Build history by job: Actually what we did is setting the build's display name. Doing that you even see the overview of used nodes in the stage view plugin. As a plus you can use the build filter provided by Jenkins to search for all builds which ran on a specific node. Works quite nice - but unfortunately no replacement for the node's build history.

          @Joerg just so I understand correctly, are you saying "build display name" like using the name setter plugin? https://wiki.jenkins-ci.org/display/JENKINS/Build+Name+Setter+Plugin

          Would you mind pasting in an example of what that end result name looks like? Since our pipeline jobs may run on 4-6 different nodes I can imagine having all of that information in a single build name getting a little unwieldy.

          Thanks,
          Justin

          Justin Rainwater added a comment - @Joerg just so I understand correctly, are you saying "build display name" like using the name setter plugin? https://wiki.jenkins-ci.org/display/JENKINS/Build+Name+Setter+Plugin Would you mind pasting in an example of what that end result name looks like? Since our pipeline jobs may run on 4-6 different nodes I can imagine having all of that information in a single build name getting a little unwieldy. Thanks, Justin

          Jesse Glick added a comment -

          You cannot even tell which nodes were used without look into the full log.

          This is already filed separately: JENKINS-26138

          Jesse Glick added a comment - You cannot even tell which nodes were used without look into the full log. This is already filed separately: JENKINS-26138

          rainwaj: For more than one node per job run this most probably will look messed up. Unfortunately I cannot paste any screenshots here because of security issues - and because I cannot setup a demonstrator instance using more than a single build agent.

          However setting the build display name is quite straight forward. You may give it a try like:

          currentBuild.setDisplayName(currentBuild.getDisplayName() + " (${agentName})")

          Afaik you won't need the build name setter plugin as it's using the currentBuild object.

          Joerg Schwaerzler added a comment - rainwaj : For more than one node per job run this most probably will look messed up. Unfortunately I cannot paste any screenshots here because of security issues - and because I cannot setup a demonstrator instance using more than a single build agent. However setting the build display name is quite straight forward. You may give it a try like: currentBuild.setDisplayName(currentBuild.getDisplayName() + " (${agentName})" ) Afaik you won't need the build name setter plugin as it's using the currentBuild object.

          Damien Jiang added a comment -

          It's been over a year since 

          There are various longer-term plans for analytic capabilities.

          Are there any updates on this?

          There are two annoyance with pipeline builds on multiple nodes:
          1. It's hard to tell which nodes were used for which pipeline steps (addressed by the "setting build display name" idea in the comments here)
          2. It's hard to tell which jobs/pipeline steps were run on a node, which is what this ticket was originally about.

           

          Are there any workarounds for the latter?

          Damien Jiang added a comment - It's been over a year since  There are various longer-term plans for analytic capabilities. Are there any updates on this? There are two annoyance with pipeline builds on multiple nodes: 1. It's hard to tell which nodes were used for which pipeline steps (addressed by the "setting build display name" idea in the comments here) 2. It's hard to tell which jobs/pipeline steps were run on a node, which is what this ticket was originally about.   Are there any workarounds for the latter?

          David Taylor added a comment -

          Why is this marked as Resolved? The original request was to be able to see build history of pipelines under each node's "Build History" section. Its May 2018 and pipelines still do not show up in this section. This ticket should not be marked as Resolved.

          David Taylor added a comment - Why is this marked as Resolved? The original request was to be able to see build history of pipelines under each node's "Build History" section. Its May 2018 and pipelines still do not show up in this section. This ticket should not be marked as Resolved.

          Jesse Glick added a comment -

          Resolved as “will not do”, which stands.

          Jesse Glick added a comment - Resolved as “will not do”, which stands.

          Noam Angel added a comment -

          any workaround for this?

          Noam Angel added a comment - any workaround for this?

          We have built our own solution. Post build results to a database or google sheet and view history by node there. CloudBees has no plans to take care of this so we have to take matters into our own hands.

          Justin Rainwater added a comment - We have built our own solution. Post build results to a database or google sheet and view history by node there. CloudBees has no plans to take care of this so we have to take matters into our own hands.

          Dylan Nicholson added a comment - - edited

          Agreed, this is an essential feature.  This shouldn't be marked as "Resolved" at all.  And it makes perfect sense to be able to see the pipeline stages that ran on each node as jobs, especially with any info about whether they failed.

          Dylan Nicholson added a comment - - edited Agreed, this is an essential feature.  This shouldn't be marked as "Resolved" at all.  And it makes perfect sense to be able to see the pipeline stages that ran on each node as jobs, especially with any info about whether they failed.

          It's "resolved" because they're not gonna do anything about it. Regardless of how essential it may be. Our team has rolled our own solution which gets the job done despite requiring a ton of work. There's a reason Mr. Glick is suggesting everyone use Jenkins less. https://www.youtube.com/watch?v=Zeqc6--0eQw

          Justin Rainwater added a comment - It's "resolved" because they're not gonna do anything about it. Regardless of how essential it may be. Our team has rolled our own solution which gets the job done despite requiring a ton of work. There's a reason Mr. Glick is suggesting everyone use Jenkins less.  https://www.youtube.com/watch?v=Zeqc6--0eQw

          Alex Dover added a comment - - edited

          Essential feature with no alternatives. Can't believe it's just abandoned

          Alex Dover added a comment - - edited Essential feature with no alternatives. Can't believe it's just abandoned

          jglick, are there any alternative ways to find out what builds were scheduled on a node?

          Andrey Babushkin added a comment - jglick , are there any alternative ways to find out what builds were scheduled on a node?

          Jesse Glick added a comment -

          Not currently, no.

          Jesse Glick added a comment - Not currently, no.

          Angelo Loria added a comment -

          Having this feature would be great.

          Angelo Loria added a comment - Having this feature would be great.

          rainwaj Hello. Since this hasn't been resolved, do you mind sharing the basic concept of how you got the job done? I'd really, really could use some help on this... jglick are you aware of any workarounds?

          Dan Dragomirescu added a comment - rainwaj Hello. Since this hasn't been resolved, do you mind sharing the basic concept of how you got the job done? I'd really, really could use some help on this... jglick are you aware of any workarounds?

          Jesse Glick added a comment -

          Well, it would be a few lines of Groovy scripting to calculate this…but as mentioned above, if you have a lot of historical builds, the performance will be abysmal (especially on HDDs) since an unbounded number of little XML files would need to be loaded from disk and deserialized into memory, then discarded shortly afterward once the system gets into a low memory condition. That is true of freestyle builds and would be worse still for Pipeline builds due to the more complex metadata. A supportable solution would need something like a database that gets updated when a build uses a node—not a difficult programming project at all, but logistically a problem since Jenkins currently lacks any kind of standardized database, so you would need to either initiate that effort, or do a one-off hack with some sort of text file stored in a node’s configuration directory.

          Jesse Glick added a comment - Well, it would be a few lines of Groovy scripting to calculate this…but as mentioned above, if you have a lot of historical builds, the performance will be abysmal (especially on HDDs) since an unbounded number of little XML files would need to be loaded from disk and deserialized into memory, then discarded shortly afterward once the system gets into a low memory condition. That is true of freestyle builds and would be worse still for Pipeline builds due to the more complex metadata. A supportable solution would need something like a database that gets updated when a build uses a node—not a difficult programming project at all, but logistically a problem since Jenkins currently lacks any kind of standardized database, so you would need to either initiate that effort, or do a one-off hack with some sort of text file stored in a node’s configuration directory.

          Nick added a comment - - edited

          Is it possible to trigger a clone freestyle job after the pipeline job completes and pass or fail it based on the pipeline job results? The result wouldn't have accurate info for time or etc but it would quickly show pass / fail results per node and the console output of the freestyle job would link to the pipeline output. I know how to trigger the freestyle clone but I'm not sure if its possible to make it pass or fail based on the pipeline job results.

          Update, it is possible to do this with the Node and Label parameter plugin. Needing the duplicate Freestyle job is a bit of a bummer though

          Nick added a comment - - edited Is it possible to trigger a clone freestyle job after the pipeline job completes and pass or fail it based on the pipeline job results? The result wouldn't have accurate info for time or etc but it would quickly show pass / fail results per node and the console output of the freestyle job would link to the pipeline output. I know how to trigger the freestyle clone but I'm not sure if its possible to make it pass or fail based on the pipeline job results. Update, it is possible to do this with the  Node and Label parameter plugin . Needing the duplicate Freestyle job is a bit of a bummer though

          Is there any chance of reopening this ticket and reconsidering the outcome? Not implementing it leaves setups primarily focused on using pipelines with no usable history of builds and gives no easy way of analyzing what was happening on connected nodes at certain point of time.

          Michał Machnicki added a comment - Is there any chance of reopening this ticket and reconsidering the outcome? Not implementing it leaves setups primarily focused on using pipelines with no usable history of builds and gives no easy way of analyzing what was happening on connected nodes at certain point of time.

          Tim Brown added a comment -

          > Is there any chance of reopening this ticket and reconsidering the outcome?
          +1

          Tim Brown added a comment - > Is there any chance of reopening this ticket and reconsidering the outcome? +1

          Adir added a comment -

          Supports issue re-opening as well. Very useful ability which gone for pipelines users (almost everyone).

          +1

          Adir added a comment - Supports issue re-opening as well. Very useful ability which gone for pipelines users (almost everyone). +1

          john added a comment -

          Likewise, as a heavy pipeline user (aren't we all ?), the Node History page is currently useless.
          +1

          john added a comment - Likewise, as a heavy pipeline user (aren't we all ?), the Node History page is currently useless. +1

          Rasmus added a comment -

          +1

          Rasmus added a comment - +1

          Darsh added a comment -

          +1

          Darsh added a comment - +1

          Dennis added a comment -

          This issue has been "resolved" since no action will be taken. Regardless of how crucial it undoubtedly is. Our group has developed a workable, albeit labor-intensive, approach. Mr. Glick has good reason to recommend reducing your use of Jenkins.

          coreball

           

          Dennis added a comment - This issue has been "resolved" since no action will be taken. Regardless of how crucial it undoubtedly is. Our group has developed a workable, albeit labor-intensive, approach. Mr. Glick has good reason to recommend reducing your use of Jenkins. coreball  

          Heiko Nardmann added a comment - - edited

          Maybe you can collect the basic information by running sth. asynchronously and populate some database with the results? I.e. looking for some '<node>' nodes inside the XML files inside the 'workflow' folder for a build?

           

          Heiko Nardmann added a comment - - edited Maybe you can collect the basic information by running sth. asynchronously and populate some database with the results? I.e. looking for some '<node>' nodes inside the XML files inside the 'workflow' folder for a build?  

          Anton added a comment -

          +1

          As an alternative I can think of using ELK for build analytics.

          Anton added a comment - +1 As an alternative I can think of using ELK for build analytics.

            Unassigned Unassigned
            rainwaj Justin Rainwater
            Votes:
            2 Vote for this issue
            Watchers:
            41 Start watching this issue

              Created:
              Updated:
              Resolved: