Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64543

Jenkins slows down after upgrade to 2.249.2

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Background: So we were running Jenkins 2.107.3 previously and I upgraded Jenkins to 2.249.2 (A big update). After that we are seeing a lot of slowness in response time from Jenkins. 

      Plugin Version:

      • Amazon EC2 - 1.53
      • SSH Build Agents - 1.31.2

      Note: Our EC2-plugin usage is big, we currently have 1200 EC2 agents in use and we can easily use 1200 more but I think if I increase the instance cap, Jenkins will just slow down even more so I am limiting that. 

      Slowness:

      • When we trigger a Matrix Job which has Dynamic Axis (250 of them), it takes 2 hours to starts all of them. Everytime after I restart the server and if the load is low, its fine but after a couple of days it starts to increase i.e. the time to "just start" all the axis slows down. 
      • I also see the below message a lot as well:
      2021-01-04 17:15:22.067+0000 [id=254615] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started DockerContainerWatchdog Asynchronous Periodic Work

      Which plugin is this?

      • I also see the below error across the board:
      2021-01-05 17:44:17.316+0000 [id=105960] WARNING h.model.listeners.RunListener#report: RunListener failed2021-01-05 17:44:17.316+0000 [id=105960] WARNING h.model.listeners.RunListener#report: RunListener failedjava.lang.NullPointerException at io.jenkins.plugins.securepostscript.SecurePostScript.onCompleted(SecurePostScript.java:26) at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209) at hudson.model.Run.execute(Run.java:1939) at hudson.matrix.MatrixRun.run(MatrixRun.java:153) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:428)

       

      Notes:

      So after these issues:

      1. I updated the EC2 plugin to 1.56
      2. I also updated all the nodes created using EC2-plugin to have a minimum no of instances and the same number to be the capacity thinking it will not try creating more.
      3. But it seems from TOP that there is something still happening in background which is still doing some work related to the EC2-plugin. I also see the below line right now:
        1. 2021-01-05 17:47:38.034+0000 [id=156676] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started EC2 alive slaves monitor

       

       

        Attachments

          Activity

          Hide
          raihaan Raihaan Shouhell added a comment -

          If you are experiencing a lot of UI slow down you might want to capture several thread dumps in succession to find any possible lock contention

          Show
          raihaan Raihaan Shouhell added a comment - If you are experiencing a lot of UI slow down you might want to capture several thread dumps in succession to find any possible lock contention

            People

            Assignee:
            thoulen FABRIZIO MANFREDI
            Reporter:
            cliffhanger Jason Hunt
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: