I have just been investigating a problem in our jenkins setup that I think might be related to JENKINS-25218. We're using the EC2 plugin and running builds that generate quite large logs (230 MB). At some point during the build, the master loses track of the log and just starts logging the same block of text from the log over and over as long as I let it. The build completes successfully on the slave and nothing bad appears in the Node log in the jenkins UI. But the master continues to fill up the filesystem with the same repeated text forever. I changed the build to log much less and now this isn't happening. We're running 2.46.2. Could this potentially be one of the edge cases?
- depends on
-
JENKINS-38381 [JEP-210] Optimize log handling in Pipeline and Durable Task
-
- Resolved
-
- duplicates
-
JENKINS-37575 Delays in FileMonitoringTask.WriteLog can cause process output to be resent indefinitely
-
- Closed
-
[JENKINS-44483] Large console logs cause infinite loops in slave
Chris Phillips
created issue -
Chris Phillips
made changes -
| Environment | Original: We run Jenkins usin the official docker image version 2.46.2, docker version 17.03.1-ce, Amazon EC2 plugin 1.36, Ubuntu 14.04, Oracle java version 1.8.0_131 for the slave.jar process | New: We run Jenkins using the official docker image version 2.46.2, docker version 17.03.1-ce, Amazon EC2 plugin 1.36, Ubuntu 14.04, Oracle java version 1.8.0_131 for the slave.jar process |
| Component/s | New: pipeline [ 21692 ] |
| Labels | New: pipeline |
| Link |
New:
This issue duplicates |
| Link |
New:
This issue depends on |
Dima Rudaev
made changes -
| Attachment | New: workaround.py [ 39704 ] |
Dima Rudaev
made changes -
| Comment |
[ Hi guys, I encountered the same issue, and I want to share my observations and a workaround for it : It less dependent on a size of a log and more depends on frequency of prints, it happens when it has a huge amount of prints in a very short time period e.g: 1000000 prints in less then 1 sec. The idea of workaround is to redirect the output and print in batches (in the attached example of workaround I redirect the stderr of my_script.py and I allow 1000 prints per second with frequency of one print of 10 lines each 0.01 second): [^workaround.py] Hope it helps !! ] |
Dima Rudaev
made changes -
| Attachment | Original: workaround.py [ 39704 ] |
| Resolution | New: Duplicate [ 3 ] | |
| Status | Original: Open [ 1 ] | New: Resolved [ 5 ] |