Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48344

Log files generated by Jenkins pipeline scripts are bloated

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      1.9

      Description

      We recently had one of our production Jenkins masters nearly run out of disk space due to an over-allocation of log data by just a couple of jobs running on our farm. In fact, 2 such builds consumed nearly a half a terabyte for their build logs alone! Further, closer examination of those log files revealed that the data therein was being excessively bloated by what appears to be some sort of markup or metadata attached to each line in the build log that looks something like this:

       

      ^[[8mha:////4OsXDMICFQEJVGu5QN07bZyJAnhssncpc0tH8m8uvnrSAAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA^[[0m[Pipeline] node

       

      To illustrate the problem I created a super trivial pipeline script as follows:

      node {
        stage('first') {
          echo "hello world"
        }
      }
      

      This simple example produced 5kb worth of build logs! To make matters worse, it appears as though the raw text (ie: excluding the markup) is also duplicated among several other .log files in the same folder as the main build log, causing even further bloat.

      I am creating this issue in the hopes that (a) someone can explain what this extra log metadata is in the main build log of a pipeline build and (b) someone can suggest some way to either eliminate this bloat and superfluous duplication, or to at least offer some way to detect large build logs and perhaps truncate, purge or even prevent them by causing the build to fail. Really anything other than filling up the Jenkins home folder would be preferable.

        Attachments

          Issue Links

            Activity

            Hide
            timur87 Tim Oz added a comment -

            Please, help

            I have the same issue on Jenkins ver. 2.176.1, Timestamper 1.10 and AnsiColor 0.6.2 plugins.

            Show
            timur87 Tim Oz added a comment - Please, help I have the same issue on Jenkins ver. 2.176.1, Timestamper 1.10 and AnsiColor 0.6.2 plugins.
            Hide
            haridsv Hari Dara added a comment -

            Jesse Glick: I am a unclear on what was fixed as part of this jira. I came looking for a solution to this same problem. We are using Jenkins version 2.176.4 and don't have any timestamps calls in our pipelines and have the configuration flag Enabled for all Pipeline builds unchecked. We don't have AnsiColor plugin installed at all, which seems to be another reason why this prefix gets added.

            Kevin Phillips: I am wondering if you ever found the root cause and a solution. Thank you!

            Show
            haridsv Hari Dara added a comment - Jesse Glick : I am a unclear on what was fixed as part of this jira. I came looking for a solution to this same problem. We are using Jenkins version 2.176.4 and don't have any timestamps  calls in our pipelines and have the configuration flag Enabled for all Pipeline builds unchecked. We don't have AnsiColor plugin installed at all, which seems to be another reason why this prefix gets added. Kevin Phillips : I am wondering if you ever found the root cause and a solution. Thank you!
            Hide
            leedega Kevin Phillips added a comment -

            The root cause is the Jenkins core product was updated a while back such that it decorates every single line that is streamed to the log file with some metadata about the the line (not sure all the details of the contents but it’s probably stuff like time stamps and such). This metadata is often larger than the actual output produced by the build via stout, sometimes by several orders of magnitude. It is not caused by any third party plugins, and since it is a feature of the product there is no way to turn it off. Based on what I’ve heard, it is basically described as a “feature” or “enhancement” provided by Jenkins and thus is deemed as “working as expected” behaviour and is not likely going to be fixed. In the end you’ll just have to hack around the problem in some way. For example, one thing we tried which helped was enabling file system compression on the file system mount containing the Jenkins logs so it doesn’t overwhelm the storage medium. Sorry I don’t have better news for you.

            Show
            leedega Kevin Phillips added a comment - The root cause is the Jenkins core product was updated a while back such that it decorates every single line that is streamed to the log file with some metadata about the the line (not sure all the details of the contents but it’s probably stuff like time stamps and such). This metadata is often larger than the actual output produced by the build via stout, sometimes by several orders of magnitude. It is not caused by any third party plugins, and since it is a feature of the product there is no way to turn it off. Based on what I’ve heard, it is basically described as a “feature” or “enhancement” provided by Jenkins and thus is deemed as “working as expected” behaviour and is not likely going to be fixed. In the end you’ll just have to hack around the problem in some way. For example, one thing we tried which helped was enabling file system compression on the file system mount containing the Jenkins logs so it doesn’t overwhelm the storage medium. Sorry I don’t have better news for you.
            Hide
            haridsv Hari Dara added a comment -

            Kevin Phillips: Thanks for the clarification. The current Fixed resolution then seems to be incorrect and will e confusing to anyone who comes looking for a solution to this problem.

            Show
            haridsv Hari Dara added a comment - Kevin Phillips : Thanks for the clarification. The current Fixed  resolution then seems to be incorrect and will e confusing to anyone who comes looking for a solution to this problem.
            Hide
            jglick Jesse Glick added a comment -

            The timestamper used to dump a big Base64-encoded serialized Java object on every line, but now adds only a fairly compact ISO-formatted timestamp, making for much less overhead. So I considered the Fixed resolution to be correct. If you have some other issue, best to discuss it elsewhere.

            Show
            jglick Jesse Glick added a comment - The timestamper used to dump a big Base64-encoded serialized Java object on every line, but now adds only a fairly compact ISO-formatted timestamp, making for much less overhead. So I considered the Fixed resolution to be correct. If you have some other issue, best to discuss it elsewhere.

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              leedega Kevin Phillips
              Votes:
              11 Vote for this issue
              Watchers:
              31 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: