Background: So we were running Jenkins 2.107.3 previously and I upgraded Jenkins to 2.249.2 (A big update). After that we are seeing a lot of slowness in response time from Jenkins. 

Plugin Version:

• Amazon EC2 - 1.53
• SSH Build Agents - 1.31.2

Note: Our EC2-plugin usage is big, we currently have 1200 EC2 agents in use and we can easily use 1200 more but I think if I increase the instance cap, Jenkins will just slow down even more so I am limiting that. 

Slowness:

• When we trigger a Matrix Job which has Dynamic Axis (250 of them), it takes 2 hours to starts all of them. Everytime after I restart the server and if the load is low, its fine but after a couple of days it starts to increase i.e. the time to "just start" all the axis slows down. 
• I also see the below message a lot as well:

2021-01-04 17:15:22.067+0000 [id=254615] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started DockerContainerWatchdog Asynchronous Periodic Work

Which plugin is this?

• I also see the below error across the board:

2021-01-05 17:44:17.316+0000 [id=105960] WARNING h.model.listeners.RunListener#report: RunListener failed2021-01-05 17:44:17.316+0000 [id=105960] WARNING h.model.listeners.RunListener#report: RunListener failedjava.lang.NullPointerException at io.jenkins.plugins.securepostscript.SecurePostScript.onCompleted(SecurePostScript.java:26) at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209) at hudson.model.Run.execute(Run.java:1939) at hudson.matrix.MatrixRun.run(MatrixRun.java:153) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:428)

Notes:

So after these issues:

1. I updated the EC2 plugin to 1.56
2. I also updated all the nodes created using EC2-plugin to have a minimum no of instances and the same number to be the capacity thinking it will not try creating more.
3. But it seems from TOP that there is something still happening in background which is still doing some work related to the EC2-plugin. I also see the below line right now:

   1. 2021-01-05 17:47:38.034+0000 [id=156676] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started EC2 alive slaves monitor