Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64543

Jenkins slows down after upgrade to 2.249.2

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Background: So we were running Jenkins 2.107.3 previously and I upgraded Jenkins to 2.249.2 (A big update). After that we are seeing a lot of slowness in response time from Jenkins. 

      Plugin Version:

      • Amazon EC2 - 1.53
      • SSH Build Agents - 1.31.2

      Note: Our EC2-plugin usage is big, we currently have 1200 EC2 agents in use and we can easily use 1200 more but I think if I increase the instance cap, Jenkins will just slow down even more so I am limiting that. 

      Slowness:

      • When we trigger a Matrix Job which has Dynamic Axis (250 of them), it takes 2 hours to starts all of them. Everytime after I restart the server and if the load is low, its fine but after a couple of days it starts to increase i.e. the time to "just start" all the axis slows down. 
      • I also see the below message a lot as well:
      2021-01-04 17:15:22.067+0000 [id=254615] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started DockerContainerWatchdog Asynchronous Periodic Work

      Which plugin is this?

      • I also see the below error across the board:
      2021-01-05 17:44:17.316+0000 [id=105960] WARNING h.model.listeners.RunListener#report: RunListener failed2021-01-05 17:44:17.316+0000 [id=105960] WARNING h.model.listeners.RunListener#report: RunListener failedjava.lang.NullPointerException at io.jenkins.plugins.securepostscript.SecurePostScript.onCompleted(SecurePostScript.java:26) at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209) at hudson.model.Run.execute(Run.java:1939) at hudson.matrix.MatrixRun.run(MatrixRun.java:153) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:428)

       

      Notes:

      So after these issues:

      1. I updated the EC2 plugin to 1.56
      2. I also updated all the nodes created using EC2-plugin to have a minimum no of instances and the same number to be the capacity thinking it will not try creating more.
      3. But it seems from TOP that there is something still happening in background which is still doing some work related to the EC2-plugin. I also see the below line right now:
        1. 2021-01-05 17:47:38.034+0000 [id=156676] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started EC2 alive slaves monitor

       

       

        Attachments

          Issue Links

            Activity

            Hide
            raihaan Raihaan Shouhell added a comment -

            If you are experiencing a lot of UI slow down you might want to capture several thread dumps in succession to find any possible lock contention

            Show
            raihaan Raihaan Shouhell added a comment - If you are experiencing a lot of UI slow down you might want to capture several thread dumps in succession to find any possible lock contention

              People

              Assignee:
              thoulen FABRIZIO MANFREDI
              Reporter:
              cliffhanger Jason Hunt
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: