Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39661

Investigate performance of Pipeline /activity and /runs REST endpoint

    • Icon: Task Task
    • Resolution: Fixed
    • Icon: Major Major
    • blueocean-plugin
    • None
    • arctic, tasman

      The /activity and /runs endpoints have been reported to take up to 28 seconds on some Jenkins masters.

      Tracing through the code, it looks like io.jenkins.blueocean.service.embedded.rest.RunContainerImpl#get will load all the runs using hudson.model.Job#getBuilds() for the Job then applies the pagination.

      However, there is a method hudson.model.Job#getBuilds(hudson.model.Fingerprint.RangeSet) that will allow us to fetch a range of data which would be suitable for pagination.

          [JENKINS-39661] Investigate performance of Pipeline /activity and /runs REST endpoint

          Vivek Pandey added a comment -

          tfennelly Great thanks. I will look at generalizing it. BTW, whats the best way to create 5000+ jobs? groovy script console?

          Vivek Pandey added a comment - tfennelly Great thanks. I will look at generalizing it. BTW, whats the best way to create 5000+ jobs? groovy script console?

          James Dumay added a comment -

          Thanks both for taking a look

          James Dumay added a comment - Thanks both for taking a look

          Tom FENNELLY added a comment -

          vivek Hey Vivek .... I'm guessing you mean 5000+ runs of a job. In that case, yes, I used the script console to do that e.g.

          def job = Jenkins.instance.getItem('pipeline_1');
          
          for (i = 0; i < 1000; i++) {
              job.scheduleBuild2(0).waitForStart();
          }
          

          If you mean Jobs then I have used the following script: https://gist.github.com/tfennelly/c53953b98850304deb405591063a6e85

          And then run that like ....

          ./create-jobs.sh --host=localhost:8080/jenkins --login=tfennelly:XXXXX --job=pipeline --count=500
          

          Tom FENNELLY added a comment - vivek Hey Vivek .... I'm guessing you mean 5000+ runs of a job. In that case, yes, I used the script console to do that e.g. def job = Jenkins.instance.getItem( 'pipeline_1' ); for (i = 0; i < 1000; i++) { job.scheduleBuild2(0).waitForStart(); } If you mean Jobs then I have used the following script: https://gist.github.com/tfennelly/c53953b98850304deb405591063a6e85 And then run that like .... ./create-jobs.sh --host=localhost:8080/jenkins --login=tfennelly:XXXXX --job=pipeline --count=500

          Michael Neale added a comment -

          I think that 5000 runs is what you would want to do to repro this, not jobs (that is a separate issue)

          Michael Neale added a comment - I think that 5000 runs is what you would want to do to repro this, not jobs (that is a separate issue)

          Michael Neale added a comment -

          FYI I think in at least the case of a lot of artifacts, blue ocean is broken here: https://issues.jenkins-ci.org/browse/JENKINS-39737 - am following it up on that ticket.

          Michael Neale added a comment - FYI I think in at least the case of a lot of artifacts, blue ocean is broken here: https://issues.jenkins-ci.org/browse/JENKINS-39737 - am following it up on that ticket.

          Vivek Pandey added a comment -

          tfennelly Yeah runs not jobs, thanks!

          Vivek Pandey added a comment - tfennelly Yeah runs not jobs , thanks!

          Michael Neale added a comment - - edited

          ok vivek we have opened other tickets for the artifact stuff, which will probably improve things a bit (in your testing you might want to comment out all artifacts and see if it is still slow).

          Based on my calculations looking at the json from https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fpipeline-model-definition-plugin/detail/PR-54/1/artifacts/ (ie the https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fpipeline-model-definition-plugin project) - it takes around 10 seconds to load one runs artifact flat listing (there are 1700 odd artifacts). Multiply that by a page worth of activity (26 items), and you get about 4 minutes latency.

          So this could be the cause of the slowness, at least in the case of abayer's pipeline.

          Still need to run a pressure test anyway as suggested by tom's script.

          Michael Neale added a comment - - edited ok vivek we have opened other tickets for the artifact stuff, which will probably improve things a bit (in your testing you might want to comment out all artifacts and see if it is still slow). Based on my calculations looking at the json from https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fpipeline-model-definition-plugin/detail/PR-54/1/artifacts/ (ie the https://ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fpipeline-model-definition-plugin project) - it takes around 10 seconds to load one runs artifact flat listing (there are 1700 odd artifacts). Multiply that by a page worth of activity (26 items), and you get about 4 minutes latency. So this could be the cause of the slowness, at least in the case of abayer 's pipeline. Still need to run a pressure test anyway as suggested by tom's script.

          Vivek Pandey added a comment -

          tfennelly I have create 6000 runs of a simple pipeline, fresh load time (on new browser is about 600ms). Maybe its my pipeline is too simple to reproduce 10s of sec initial load time? Its all on my laptop.

          Here is my pipeline:

          stage 'build'
          node{
            echo "Building..."
          }
          stage 'deploy'
          node{
            echo "Deploying"
          }
          

          Can you share your pipeline? Maybe that will help me diagnose. I am proceeding with a fix regardless, however need to validate it so need to reproduce.

          Thanks buddy!

          Vivek Pandey added a comment - tfennelly I have create 6000 runs of a simple pipeline, fresh load time (on new browser is about 600ms). Maybe its my pipeline is too simple to reproduce 10s of sec initial load time? Its all on my laptop. Here is my pipeline: stage 'build' node{ echo "Building..." } stage 'deploy' node{ echo "Deploying" } Can you share your pipeline? Maybe that will help me diagnose. I am proceeding with a fix regardless, however need to validate it so need to reproduce. Thanks buddy!

          Tom FENNELLY added a comment -

          vivek Hey Vivek ... afraid I wiped my test env, but it was a simple enough pipeline that just echoed to the log about 100 or 200 times (I forget exactly, but wasn't a lot).

          So something like...

          for (i = 0; i < 200; i++) {
              echo "Vivek is an awesome guy, except that he loves Donald Trump"
          }
          

          Tom FENNELLY added a comment - vivek Hey Vivek ... afraid I wiped my test env, but it was a simple enough pipeline that just echoed to the log about 100 or 200 times (I forget exactly, but wasn't a lot). So something like... for (i = 0; i < 200; i++) { echo "Vivek is an awesome guy, except that he loves Donald Trump" }

          Vivek Pandey added a comment -

          tfennelly Ha ha! That test is some evil fantasy of yours

          Ok so test is fine. For some reason I don't see that kind of slow performance, relatively speaking I am getting about 300% improvement with my change. 670ms to ~250ms. I am going to put a PR.

          Vivek Pandey added a comment - tfennelly Ha ha! That test is some evil fantasy of yours Ok so test is fine. For some reason I don't see that kind of slow performance, relatively speaking I am getting about 300% improvement with my change. 670ms to ~250ms. I am going to put a PR.

            vivek Vivek Pandey
            jamesdumay James Dumay
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: