-
Bug
-
Resolution: Unresolved
-
Critical
-
Jenkins 2.303.3
log-parser 2.1.0
RH 7
JDK 11
We ran into a curious but deadly problem for our CI, involving the log parser.
One job created by accident a very, very, very big console file. (80 GByte)
Seems log-parser is creating "log-parser_......."-files in /tmp, based on console log.
So this really big console file, made /tmp FS (20 GByte) fill up completely and instantly.
Unfortunately Jenkins is also creating the jenkins<DIGITS>.sh files, containing the 'Execute Shell' section of freestyle jobs below /tmp. Because /tmp was full, this failed, bringing Jenkins to an immediate and grinding halt.
We have jobs, trying to clean up /tmp and warning us of the fill, but these too couldn't start => Catch 22.
Any ideas to prevent something like that ?
My immediate idea would be a setting to limit log parser to generate files larger than x Gbyte below /tmp - or y % of free space; obviously I prefer a missing parsed console to a halted system.
Note:
The "obvious" solution is to not creating such big console logs - but we see some strange bugs causing this maybe once a year or every two years.
We also currently work on moving all jobs away from the controller ("controller isolation") which would keep Jenkins itself running, but "only" "kill" one of the agents.