-
Bug
-
Resolution: Fixed
-
Major
-
None
-
jenkinsci/blueocean image, Core version 2.249.1
Docker plugin 1.2.1
Docker pipeline 1.24
Docker commons 1.17
Node with Docker CE 19.03.13
Jenkins Master host with Docker CE 19.03.13
I can build projects by hand individually. For the nightly builds, we throw probably 10-15 projects into the queue at a time, and wait for them to filter through. Upon a recent upgrade of both jenkinsci/blueocean and the plugins, these builds now hang indefinitely and no new builds can be started successfully. Restarting Jenkins master fixes the issue.
Connections are done over a Docker cloud, using the TCP connection and the "attach Docker container" option.
The errors on the node hosting the build instances is:
DockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:15:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:20:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:25:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted.
The console logs for each build on the Jenkins master show:
Started by timer Obtained Jenkinsfile from git git@gitlab.company.org:ns/repo.git Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Still waiting to schedule task 'Ubuntu 16.04 Kinetic-0006iph8pkfzg on docker' is offline
EDIT: Just verified as well that I can overload our build agents and the queue will eventually clear (overloaded by 1 extra build). So maybe it's the number of tasks? I try to push something like 16 builds at the same time, with the bandwidth to handle 4 at a time, and each build takes probably 10 minutes.
We're seeing a fairly similar problem. This cropped up in the last 5 or 6 days after using the docker agent model for about 6 months now with no issues.
(I used to see the aborted message in our logs, but have not seen it in the last week when this problem has started to occur)
Our issue is that the Jenkins master stops spinning up new container build agents. I'm not even sure yet where the failure point is in my logs.
Jenkins LTS 2.249.2
Docker plugin 1.2.1
Docker version 19.03.13
All my plugins are generally up to date. I usually update them all every week or two.
Our docker agents are using SSH to connect.