I have just been investigating a problem in our jenkins setup that I think might be related to JENKINS-25218. We're using the EC2 plugin and running builds that generate quite large logs (230 MB). At some point during the build, the master loses track of the log and just starts logging the same block of text from the log over and over as long as I let it. The build completes successfully on the slave and nothing bad appears in the Node log in the jenkins UI. But the master continues to fill up the filesystem with the same repeated text forever. I changed the build to log much less and now this isn't happening. We're running 2.46.2. Could this potentially be one of the edge cases?
- depends on
-
JENKINS-38381 [JEP-210] Optimize log handling in Pipeline and Durable Task
-
- Resolved
-
- duplicates
-
JENKINS-37575 Delays in FileMonitoringTask.WriteLog can cause process output to be resent indefinitely
-
- Closed
-
[JENKINS-44483] Large console logs cause infinite loops in slave
Chris Phillips
created issue -
Chris Phillips
made changes -
| Environment | Original: We run Jenkins usin the official docker image version 2.46.2, docker version 17.03.1-ce, Amazon EC2 plugin 1.36, Ubuntu 14.04, Oracle java version 1.8.0_131 for the slave.jar process | New: We run Jenkins using the official docker image version 2.46.2, docker version 17.03.1-ce, Amazon EC2 plugin 1.36, Ubuntu 14.04, Oracle java version 1.8.0_131 for the slave.jar process |
Chris Phillips
added a comment - We do use pipeline. Another variable that might be in play is that we were using an EFS volume for jenkins home. We've since migrated away to using EBS. We were having pretty typical NFS type problems with the master getting hung up with super high load avg yet using no cpu and high network bandwidth.
Since we reduced the log verbosity we haven't had the problem (even before we switched off EFS). I didn't see anything in the system logs when it happened. The thread dump wasn't the same as 25218. It really appeared to be a livelock situation the threads weren't stuck outright. I'll try and reproduce and take some thread dumps.
Chris Phillips
added a comment - We do use pipeline. Another variable that might be in play is that we were using an EFS volume for jenkins home. We've since migrated away to using EBS. We were having pretty typical NFS type problems with the master getting hung up with super high load avg yet using no cpu and high network bandwidth.
Since we reduced the log verbosity we haven't had the problem (even before we switched off EFS). I didn't see anything in the system logs when it happened. The thread dump wasn't the same as 25218. It really appeared to be a livelock situation the threads weren't stuck outright. I'll try and reproduce and take some thread dumps. Pipeline has it's own log collection logic, hence I am not sure it is a completely remoting issue. CC jglick
| Component/s | New: pipeline [ 21692 ] |
| Labels | New: pipeline |
Probably a dupe of JENKINS-37575. Will be obsolete as soon as I can merge JENKINS-38381.
| Link |
New:
This issue duplicates |
Chris Phillips
added a comment - I was able to recreate the issue just now by flipping the verbosity of the build back higher. Nothing blocked in the thread dumps.
I'll keep an eye out for resolution of JENKINS-38381 and try again then. It sounds promising.
Chris Phillips
added a comment - I was able to recreate the issue just now by flipping the verbosity of the build back higher. Nothing blocked in the thread dumps.
I'll keep an eye out for resolution of JENKINS-38381 and try again then. It sounds promising.
Recently I had a job with about 25GB of logs, and remoting didn't hang.
It would be great to see more details about your setup...
JENKINS-25218?Any additional information would be useful, especially system logs and stacktraces