Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48344

Log files generated by Jenkins pipeline scripts are bloated

    • 1.9

      We recently had one of our production Jenkins masters nearly run out of disk space due to an over-allocation of log data by just a couple of jobs running on our farm. In fact, 2 such builds consumed nearly a half a terabyte for their build logs alone! Further, closer examination of those log files revealed that the data therein was being excessively bloated by what appears to be some sort of markup or metadata attached to each line in the build log that looks something like this:

       

      ^[[8mha:////4OsXDMICFQEJVGu5QN07bZyJAnhssncpc0tH8m8uvnrSAAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA^[[0m[Pipeline] node

       

      To illustrate the problem I created a super trivial pipeline script as follows:

      node {
        stage('first') {
          echo "hello world"
        }
      }
      

      This simple example produced 5kb worth of build logs! To make matters worse, it appears as though the raw text (ie: excluding the markup) is also duplicated among several other .log files in the same folder as the main build log, causing even further bloat.

      I am creating this issue in the hopes that (a) someone can explain what this extra log metadata is in the main build log of a pipeline build and (b) someone can suggest some way to either eliminate this bloat and superfluous duplication, or to at least offer some way to detect large build logs and perhaps truncate, purge or even prevent them by causing the build to fail. Really anything other than filling up the Jenkins home folder would be preferable.

          [JENKINS-48344] Log files generated by Jenkins pipeline scripts are bloated

          Kevin Phillips created issue -
          Kevin Phillips made changes -
          Comment [ Note: I did find the build log compression plugin found [here|https://plugins.jenkins.io/compress-buildlog] which sounds encouraging, but given that it changes the actual format of the log file itself it risks breaking other plugins that use the log. I haven't tested it out with our current production system but it seems overly fragile / risky to me. Also, it doesn't address the duplication of the independent .log files either apparently.  ]
          Kevin Phillips made changes -
          Environment New: jenkins core 2.46.3
          pipeline v2.6
          Kevin Phillips made changes -
          Environment Original: jenkins core 2.46.3
          pipeline v2.6
          New: jenkins core 2.60.2
          pipeline v2.6
          Kevin Phillips made changes -
          Environment Original: jenkins core 2.60.2
          pipeline v2.6
          New: jenkins core v2.43.3, v2.60.2, v2.89.2
          pipeline v2.6
          Kevin Phillips made changes -
          Attachment New: log [ 40709 ]
          Kevin Phillips made changes -
          Attachment New: support_2017-12-15_14.zip [ 40710 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-38381 [ JENKINS-38381 ]
          Jakub Bochenski made changes -
          Link New: This issue is blocked by JENKINS-44195 [ JENKINS-44195 ]
          Jesse Glick made changes -
          Component/s New: timestamper-plugin [ 15749 ]
          Component/s Original: pipeline [ 21692 ]
          Issue Type Original: Improvement [ 4 ] New: Bug [ 1 ]
          Labels New: performance pipeline
          Oleg Nenashev made changes -
          Link New: This issue relates to JENKINS-51229 [ JENKINS-51229 ]

            jglick Jesse Glick
            leedega Kevin Phillips
            Votes:
            11 Vote for this issue
            Watchers:
            31 Start watching this issue

              Created:
              Updated:
              Resolved: