-
Bug
-
Resolution: Fixed
-
Major
-
None
-
jenkinsci/blueocean image, Core version 2.249.1
Docker plugin 1.2.1
Docker pipeline 1.24
Docker commons 1.17
Node with Docker CE 19.03.13
Jenkins Master host with Docker CE 19.03.13
I can build projects by hand individually. For the nightly builds, we throw probably 10-15 projects into the queue at a time, and wait for them to filter through. Upon a recent upgrade of both jenkinsci/blueocean and the plugins, these builds now hang indefinitely and no new builds can be started successfully. Restarting Jenkins master fixes the issue.
Connections are done over a Docker cloud, using the TCP connection and the "attach Docker container" option.
The errors on the node hosting the build instances is:
DockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:15:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:20:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:25:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted.
The console logs for each build on the Jenkins master show:
Started by timer Obtained Jenkinsfile from git git@gitlab.company.org:ns/repo.git Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Still waiting to schedule task 'Ubuntu 16.04 Kinetic-0006iph8pkfzg on docker' is offline
EDIT: Just verified as well that I can overload our build agents and the queue will eventually clear (overloaded by 1 extra build). So maybe it's the number of tasks? I try to push something like 16 builds at the same time, with the bandwidth to handle 4 at a time, and each build takes probably 10 minutes.
[JENKINS-63999] Build nodes stop responding with DockerContainerWatchdog error
| Description |
Original:
I can build projects by hand individually. For the nightly builds, we throw probably 10-15 projects into the queue at a time, and wait for them to filter through. Upon a recent upgrade of both jenkinsci/blueocean and the plugins, these builds now hang indefinitely and no new builds can be started successfully. Restarting Jenkins master fixes the issue. Connections are done over a Docker cloud, using the TCP connection and the "attach Docker container" option. The errors on the node hosting the build instances is: {noformat} DockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:15:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:20:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:25:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted.{noformat} The console logs for each build on the Jenkins master show: {noformat} Started by timer Obtained Jenkinsfile from git git@gitlab.company.org:ns/repo.git Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Still waiting to schedule task 'Ubuntu 16.04 Kinetic-0006iph8pkfzg on docker' is offline{noformat} |
New:
I can build projects by hand individually. For the nightly builds, we throw probably 10-15 projects into the queue at a time, and wait for them to filter through. Upon a recent upgrade of both jenkinsci/blueocean and the plugins, these builds now hang indefinitely and no new builds can be started successfully. Restarting Jenkins master fixes the issue. Connections are done over a Docker cloud, using the TCP connection and the "attach Docker container" option. The errors on the node hosting the build instances is: {noformat} DockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:15:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:20:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted. Oct 13, 2020 4:25:41 PM INFO hudson.model.AsyncPeriodicWork doRunDockerContainerWatchdog Asynchronous Periodic Work thread is still running. Execution aborted.{noformat} The console logs for each build on the Jenkins master show: {noformat} Started by timer Obtained Jenkinsfile from git git@gitlab.company.org:ns/repo.git Running in Durability level: MAX_SURVIVABILITY [Pipeline] Start of Pipeline [Pipeline] node Still waiting to schedule task 'Ubuntu 16.04 Kinetic-0006iph8pkfzg on docker' is offline{noformat} EDIT: Just verified as well that I can overload our build agents and the queue will eventually clear (overloaded by 1 extra build). So maybe it's the number of tasks? I try to push something like 16 builds at the same time, with the bandwidth to handle 4 at a time, and each build takes probably 10 minutes. |
Kevin Broselge
made changes -
| Attachment | New: Thread dump [Jenkins].html [ 58714 ] |
Kevin Broselge
made changes -
| Attachment | Original: Thread dump [Jenkins].html [ 58714 ] |
Kevin Broselge
made changes -
| Attachment | New: Thread dump [Jenkins].zip [ 58715 ] |
| Resolution | New: Fixed [ 1 ] | |
| Status | Original: Open [ 1 ] | New: Closed [ 6 ] |