I have a Jenkins server running on an EC2 instance with a number of workers running in containers in our data center. Most of my jobs are either triggered on a timed schedule or are triggered by the Bitbucket pr runner plugin.
This system has been running fine for months but recently I've been seeing an issue where the timed jobs and Bitbucket pr jobs have stopped being scheduled. During this time I can still launch jobs manually.
This happens 2 or 3 times a day and I don't see any errors in the jekins logs. Restarting Jenkins will fix it for a few hours but it always returns.
The jenkins master has 16GB of ram and I've checked that it is never close to using it all. Generally it uses 2-3G.
I've attached a threadDump taken when this was happening.
I'm not sure where else to look to debug this so I'd appreciate any advice on what to look for. Thanks