Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41897

Karaoke: steps and nodes are limited to 100 - need to increase limit

    • iapetus, 1.0

      tscherler the steps api is paginated with a page size of 100 - the front end code will need to change to fetch the next patch of data (cc vivek)

       In Scope: 

      Out of scope: 

      • steps api (separate ticket)
      • making graph cope with loading large graphs of nodes in a paginated fashion (this needs more thinking and design). 

       

      This could be addressed with a refactoring of Karaoke I guess?
      This also explains why only 100 parallel nodes are changed (in that case, it is pointless to show more, but 100 steps... you may want to show that).

      It seems that we only get a limited number of steps

      Using the following pipelineScript and https://github.com/scherler/blueocean-shared-library/commit/43d5620a42d7795b43bc829d694c3cae2a3c3919

      @Library('test-writer') import writeTest
      import longLog
      node {
          stage 'fin'
          def xml = writeTest()
          sh "echo '$xml' > TEST-some.xml"
          step([$class: 'JUnitResultArchiver', testResults: 'TEST-*.xml'])
          sh 'echo `date` fin;sleep 3; echo `date` fin;'
          sh 'echo yeah > foo.txt'
          archiveArtifacts 'foo.txt'
          longLog(10000, false)
          stage 'NoSteps'
      }
      

          [JENKINS-41897] Karaoke: steps and nodes are limited to 100 - need to increase limit

          Michael Neale added a comment -

          tscherler is this a limit in the data or what the UI shows?

          Michael Neale added a comment - tscherler is this a limit in the data or what the UI shows?

          Michael Neale added a comment -

          Karaoke needs some rework

          Michael Neale added a comment - Karaoke needs some rework

          Thorsten Scherler added a comment - - edited

          michaelneale the problem is lies in the server response, that is limiting to 100 for `nodes`. There is no easy solution for this in rendering the pipelineGraph since that needs ALL data to be returned by the rest backend and not  only a subset. That is due to the fact that the code is using the "edges" to determine the next step and if that is not present it will fail.

           

          Thorsten Scherler added a comment - - edited michaelneale the problem is lies in the server response, that is limiting to 100 for `nodes`. There is no easy solution for this in rendering the pipelineGraph since that needs ALL data to be returned by the rest backend and not  only a subset. That is due to the fact that the code is using the "edges" to determine the next step and if that is not present it will fail.  

          Michael Neale added a comment -

          tscherler understood. I guess we can bump up the limit, but know there is a limit. A more digorous solution is perhaps to change how data is returned in large cases, as I think it practically makes no sense to show > 100 nodes, probably right? 

          Ie in large cases, we have to flatten out the steps in the stage similar to how stage view shows it (or else we have UI work to fold some away...)

           

          jamesdumay I think we need to think on this some more on the practicalities. There are a few solutions, but just fetching all the data, pagination or not, obviously wont work for all cases. 

          Michael Neale added a comment - tscherler understood. I guess we can bump up the limit, but know there is a limit. A more digorous solution is perhaps to change how data is returned in large cases, as I think it practically makes no sense to show > 100 nodes, probably right?  Ie in large cases, we have to flatten out the steps in the stage similar to how stage view shows it (or else we have UI work to fold some away...)   jamesdumay I think we need to think on this some more on the practicalities. There are a few solutions, but just fetching all the data, pagination or not, obviously wont work for all cases. 

          Michael Neale added a comment - - edited

          Well as a medium term measure: https://github.com/jenkinsci/blueocean-plugin/compare/raiseNodeLimitTo1000 I think is ok IMO. 

          People with large numbers of nodes will have a not great time no matter what we do I think, so that is probably the best bet. 

          If you have 1000s of nodes, parallel or otherwise (firstly, parallel?) then a karaoke like visualisation is not appropriate anyway. So we raise the limit (in future we could recalculate things server side to collapse nodes, which makes the visualisation not representable, but manageable). 

           

          EDIT: tscherler vivek it would be nice to support a paginated list of steps under a node - but I am not sure if they are included in the total node count/graph? 

          Michael Neale added a comment - - edited Well as a medium term measure: https://github.com/jenkinsci/blueocean-plugin/compare/raiseNodeLimitTo1000  I think is ok IMO.  People with large numbers of nodes will have a not great time no matter what we do I think, so that is probably the best bet.  If you have 1000s of nodes, parallel or otherwise (firstly, parallel?) then a karaoke like visualisation is not appropriate anyway. So we raise the limit (in future we could recalculate things server side to collapse nodes, which makes the visualisation not representable, but manageable).    EDIT: tscherler vivek it would be nice to support a paginated list of steps under a node - but I am not sure if they are included in the total node count/graph? 

          Michael Neale added a comment -

          OK - lets keep the scope of this to do with increasing the limit for nodes - I will open another ticket for loading more steps as that can use pagination.  

          Michael Neale added a comment - OK - lets keep the scope of this to do with increasing the limit for nodes - I will open another ticket for loading more steps as that can use pagination.  

          James Dumay added a comment -

          michaelneale certainly we need to figure out what a practical limit is for how many nodes to display. There is a design TODO for that in the backlog.

          James Dumay added a comment - michaelneale certainly we need to figure out what a practical limit is for how many nodes to display. There is a design TODO for that in the backlog.

          Michael Neale added a comment -

          jamesdumay right - so we can bump some limit, but need to know what it would look like in future for these cases. I really don't think 500 dots across by 500 down is a great UX - but we can at least render something. 

           

          I am sure there is a longer term solution out there. 

          Michael Neale added a comment - jamesdumay right - so we can bump some limit, but need to know what it would look like in future for these cases. I really don't think 500 dots across by 500 down is a great UX - but we can at least render something.    I am sure there is a longer term solution out there. 

          Vivek Pandey added a comment -

          Maybe short term of 500 is fine. '/nodes/:id/steps/?start=0&limit=500' would fetch 0 to 500 on first and then on subsequent requests client can look at Link HTTP header, fetch next and simply use it.

          For example:

          https://ci.blueocean.io/blue/rest/organizations/jenkins/pipelines/i386/pipelines/app-store-demo/master/runs/1/nodes/11/steps/?start=0&limit=2
          

          HTTP link Header:

          link:</blue/rest/organizations/jenkins/pipelines/i386/pipelines/app-store-demo/master/runs/1/nodes/11/steps/?start=2&limit=2>; rel="next"
          

          Vivek Pandey added a comment - Maybe short term of 500 is fine. '/nodes/:id/steps/?start=0&limit=500' would fetch 0 to 500 on first and then on subsequent requests client can look at Link HTTP header, fetch next and simply use it. For example: https: //ci.blueocean.io/blue/ rest /organizations/jenkins/pipelines/i386/pipelines/app-store-demo/master/runs/1/nodes/11/steps/?start=0&limit=2 HTTP link Header: link:</blue/ rest /organizations/jenkins/pipelines/i386/pipelines/app-store-demo/master/runs/1/nodes/11/steps/?start=2&limit=2>; rel= "next"

          James Dumay added a comment -

          Closing this as a duplicate of JENKINS-39770.

          James Dumay added a comment - Closing this as a duplicate of JENKINS-39770 .

            tscherler Thorsten Scherler
            tscherler Thorsten Scherler
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: