Background: So we were running Jenkins 2.107.3 previously and I upgraded Jenkins to 2.249.2 (A big update). After that we are seeing a lot of slowness in response time from Jenkins.
- Amazon EC2 - 1.53
- SSH Build Agents - 1.31.2
Note: Our EC2-plugin usage is big, we currently have 1200 EC2 agents in use and we can easily use 1200 more but I think if I increase the instance cap, Jenkins will just slow down even more so I am limiting that.
- When we trigger a Matrix Job which has Dynamic Axis (250 of them), it takes 2 hours to starts all of them. Everytime after I restart the server and if the load is low, its fine but after a couple of days it starts to increase i.e. the time to "just start" all the axis slows down.
- I also see the below message a lot as well:
Which plugin is this?
- I also see the below error across the board:
So after these issues:
- I updated the EC2 plugin to 1.56
- I also updated all the nodes created using EC2-plugin to have a minimum no of instances and the same number to be the capacity thinking it will not try creating more.
- But it seems from TOP that there is something still happening in background which is still doing some work related to the EC2-plugin. I also see the below line right now: