Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48344

Log files generated by Jenkins pipeline scripts are bloated

    • 1.9

      We recently had one of our production Jenkins masters nearly run out of disk space due to an over-allocation of log data by just a couple of jobs running on our farm. In fact, 2 such builds consumed nearly a half a terabyte for their build logs alone! Further, closer examination of those log files revealed that the data therein was being excessively bloated by what appears to be some sort of markup or metadata attached to each line in the build log that looks something like this:

       

      ^[[8mha:////4OsXDMICFQEJVGu5QN07bZyJAnhssncpc0tH8m8uvnrSAAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA^[[0m[Pipeline] node

       

      To illustrate the problem I created a super trivial pipeline script as follows:

      node {
        stage('first') {
          echo "hello world"
        }
      }
      

      This simple example produced 5kb worth of build logs! To make matters worse, it appears as though the raw text (ie: excluding the markup) is also duplicated among several other .log files in the same folder as the main build log, causing even further bloat.

      I am creating this issue in the hopes that (a) someone can explain what this extra log metadata is in the main build log of a pipeline build and (b) someone can suggest some way to either eliminate this bloat and superfluous duplication, or to at least offer some way to detect large build logs and perhaps truncate, purge or even prevent them by causing the build to fail. Really anything other than filling up the Jenkins home folder would be preferable.

          [JENKINS-48344] Log files generated by Jenkins pipeline scripts are bloated

          Jesse Glick added a comment -

          For anyone interested in trying this, use this build.

          Jesse Glick added a comment - For anyone interested in trying this, use this build .

          Nick Jones added a comment - - edited

          Running the custom Timestamper plugin build in jglick's previous comment, on our test Jenkins instance, with a timestamps block in a Scripted Pipeline job but with the global "Enabled for all Pipeline builds" checkbox checked, I'm seeing 6681 "deprecated*.log" files – each the same size as the "log" file (2328MB) – created on our master node by a single build, all within a few minutes of each other. This is consuming 14GB+ of space for this one build, which would be catastrophic for disk space if we applied the same updates to our production Jenkins instance (where there are hundreds of jobs building hundreds of times per day).

          This timestamps block is still in place because our production Jenkins system hasn't had its plugins changed, and uses the same Jenkinsfile for this particular job, and with the hundreds of jobs we have in place, coordinating the removal of a timestamps block in GitHub across all these Jenkinsfiles with the upgrade of the Timestamper (and/or other plugins) would be exceptionally difficult. I'm assuming (perhaps wrongly) that these "deprecated" files are being generated by some interaction between this block and the newer plugin(s) in this system, but perhaps it's something else.

          Is the only way forward to remove all the timestamps blocks or options from all our Jenkinsfiles first, then upgrading these plugins? Or is something else going on?

          Nick Jones added a comment - - edited Running the custom Timestamper plugin build in jglick 's previous comment, on our test Jenkins instance, with a timestamps block in a Scripted Pipeline job but with the global "Enabled for all Pipeline builds" checkbox checked, I'm seeing 6681 "deprecated*.log" files – each the same size as the "log" file (2328MB) – created on our master node by a single build , all within a few minutes of each other. This is consuming 14GB+ of space for this one build, which would be catastrophic for disk space if we applied the same updates to our production Jenkins instance (where there are hundreds of jobs building hundreds of times per day). This timestamps block is still in place because our production Jenkins system hasn't had its plugins changed, and uses the same Jenkinsfile for this particular job, and with the hundreds of jobs we have in place, coordinating the removal of a timestamps block in GitHub across all these Jenkinsfiles with the upgrade of the Timestamper (and/or other plugins) would be exceptionally difficult. I'm assuming (perhaps wrongly) that these "deprecated" files are being generated by some interaction between this block and the newer plugin(s) in this system, but perhaps it's something else. Is the only way forward to remove all the timestamps blocks or options from all our Jenkinsfiles first, then upgrading these plugins? Or is something else going on?

          Jesse Glick added a comment -

          medianick no, with the global setting enabled the timestamps step/option does nothing besides print a warning that it is doing nothing. Your issue is some instance of JENKINS-54128, with the exact cause visible by checking the stack trace in your system log.

          Jesse Glick added a comment - medianick no, with the global setting enabled the timestamps step/option does nothing besides print a warning that it is doing nothing. Your issue is some instance of JENKINS-54128 , with the exact cause visible by checking the stack trace in your system log.

          Ulli Hafner added a comment -

          jglick I just came around this issue due to JENKINS-56484. Before Timestamper 1.9 I used ConsoleNote::removeNotes to get rid of additional console decorations when parsing warnings. This does obviously not work anymore. Is there a general way to remove the timestamper plugin output from the console log?

          Ulli Hafner added a comment - jglick I just came around this issue due to JENKINS-56484 . Before Timestamper 1.9 I used ConsoleNote::removeNotes to get rid of additional console decorations when parsing warnings. This does obviously not work anymore. Is there a general way to remove the timestamper plugin output from the console log?

          Jesse Glick added a comment -

          Better to discuss there.

          Jesse Glick added a comment - Better to discuss there.

          Tim Oz added a comment -

          Please, help

          I have the same issue on Jenkins ver. 2.176.1, Timestamper 1.10 and AnsiColor 0.6.2 plugins.

          Tim Oz added a comment - Please, help I have the same issue on Jenkins ver. 2.176.1, Timestamper 1.10 and AnsiColor 0.6.2 plugins.

          Hari Dara added a comment -

          jglick: I am a unclear on what was fixed as part of this jira. I came looking for a solution to this same problem. We are using Jenkins version 2.176.4 and don't have any timestamps calls in our pipelines and have the configuration flag Enabled for all Pipeline builds unchecked. We don't have AnsiColor plugin installed at all, which seems to be another reason why this prefix gets added.

          leedega: I am wondering if you ever found the root cause and a solution. Thank you!

          Hari Dara added a comment - jglick : I am a unclear on what was fixed as part of this jira. I came looking for a solution to this same problem. We are using Jenkins version 2.176.4 and don't have any timestamps  calls in our pipelines and have the configuration flag Enabled for all Pipeline builds unchecked. We don't have AnsiColor plugin installed at all, which seems to be another reason why this prefix gets added. leedega : I am wondering if you ever found the root cause and a solution. Thank you!

          The root cause is the Jenkins core product was updated a while back such that it decorates every single line that is streamed to the log file with some metadata about the the line (not sure all the details of the contents but it’s probably stuff like time stamps and such). This metadata is often larger than the actual output produced by the build via stout, sometimes by several orders of magnitude. It is not caused by any third party plugins, and since it is a feature of the product there is no way to turn it off. Based on what I’ve heard, it is basically described as a “feature” or “enhancement” provided by Jenkins and thus is deemed as “working as expected” behaviour and is not likely going to be fixed. In the end you’ll just have to hack around the problem in some way. For example, one thing we tried which helped was enabling file system compression on the file system mount containing the Jenkins logs so it doesn’t overwhelm the storage medium. Sorry I don’t have better news for you.

          Kevin Phillips added a comment - The root cause is the Jenkins core product was updated a while back such that it decorates every single line that is streamed to the log file with some metadata about the the line (not sure all the details of the contents but it’s probably stuff like time stamps and such). This metadata is often larger than the actual output produced by the build via stout, sometimes by several orders of magnitude. It is not caused by any third party plugins, and since it is a feature of the product there is no way to turn it off. Based on what I’ve heard, it is basically described as a “feature” or “enhancement” provided by Jenkins and thus is deemed as “working as expected” behaviour and is not likely going to be fixed. In the end you’ll just have to hack around the problem in some way. For example, one thing we tried which helped was enabling file system compression on the file system mount containing the Jenkins logs so it doesn’t overwhelm the storage medium. Sorry I don’t have better news for you.

          Hari Dara added a comment -

          leedega: Thanks for the clarification. The current Fixed resolution then seems to be incorrect and will e confusing to anyone who comes looking for a solution to this problem.

          Hari Dara added a comment - leedega : Thanks for the clarification. The current Fixed  resolution then seems to be incorrect and will e confusing to anyone who comes looking for a solution to this problem.

          Jesse Glick added a comment -

          The timestamper used to dump a big Base64-encoded serialized Java object on every line, but now adds only a fairly compact ISO-formatted timestamp, making for much less overhead. So I considered the Fixed resolution to be correct. If you have some other issue, best to discuss it elsewhere.

          Jesse Glick added a comment - The timestamper used to dump a big Base64-encoded serialized Java object on every line, but now adds only a fairly compact ISO-formatted timestamp, making for much less overhead. So I considered the Fixed resolution to be correct. If you have some other issue, best to discuss it elsewhere.

            jglick Jesse Glick
            leedega Kevin Phillips
            Votes:
            11 Vote for this issue
            Watchers:
            31 Start watching this issue

              Created:
              Updated:
              Resolved: