Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48344

Log files generated by Jenkins pipeline scripts are bloated

    • 1.9

      We recently had one of our production Jenkins masters nearly run out of disk space due to an over-allocation of log data by just a couple of jobs running on our farm. In fact, 2 such builds consumed nearly a half a terabyte for their build logs alone! Further, closer examination of those log files revealed that the data therein was being excessively bloated by what appears to be some sort of markup or metadata attached to each line in the build log that looks something like this:

       

      ^[[8mha:////4OsXDMICFQEJVGu5QN07bZyJAnhssncpc0tH8m8uvnrSAAAAaB+LCAAAAAAAAP9b85aBtbiIwTG/KF0vKzUvOzOvODlTryCnNB3I0ivPL8pOy8kv18vKT9JLzs8rzs9J1QuHCgaV5jlDhPzyS1IZIICRiYGhoohBKqM0pTg/D64Hh8ICAFt0h+h/AAAA^[[0m[Pipeline] node

       

      To illustrate the problem I created a super trivial pipeline script as follows:

      node {
        stage('first') {
          echo "hello world"
        }
      }
      

      This simple example produced 5kb worth of build logs! To make matters worse, it appears as though the raw text (ie: excluding the markup) is also duplicated among several other .log files in the same folder as the main build log, causing even further bloat.

      I am creating this issue in the hopes that (a) someone can explain what this extra log metadata is in the main build log of a pipeline build and (b) someone can suggest some way to either eliminate this bloat and superfluous duplication, or to at least offer some way to detect large build logs and perhaps truncate, purge or even prevent them by causing the build to fail. Really anything other than filling up the Jenkins home folder would be preferable.

          [JENKINS-48344] Log files generated by Jenkins pipeline scripts are bloated

          Kevin Phillips created issue -

          Note: I did find the build log compression plugin found here which sounds encouraging, but given that it changes the actual format of the log file itself it risks breaking other plugins that use the log. I haven't tested it out with our current production system but it seems overly fragile / risky to me. Also, it doesn't address the duplication of the independent .log files either apparently. 

          Kevin Phillips added a comment - Note: I did find the build log compression plugin found here  which sounds encouraging, but given that it changes the actual format of the log file itself it risks breaking other plugins that use the log. I haven't tested it out with our current production system but it seems overly fragile / risky to me. Also, it doesn't address the duplication of the independent .log files either apparently. 
          Kevin Phillips made changes -
          Comment [ Note: I did find the build log compression plugin found [here|https://plugins.jenkins.io/compress-buildlog] which sounds encouraging, but given that it changes the actual format of the log file itself it risks breaking other plugins that use the log. I haven't tested it out with our current production system but it seems overly fragile / risky to me. Also, it doesn't address the duplication of the independent .log files either apparently.  ]
          Kevin Phillips made changes -
          Environment New: jenkins core 2.46.3
          pipeline v2.6

          James Dumay added a comment - - edited

          leedega what you are seeing in the log is known as a log annotation. You likely have some plugins installed that annotate log files, such as the ASCII color plugin, which are adding this data for each log line. Please review your plugins for any that annotate log files.

          If you generate a support bundle and post the plugins.txt here perhaps we can identify the issue.

          James Dumay added a comment - - edited leedega what you are seeing in the log is known as a log annotation. You likely have some plugins installed that annotate log files, such as the ASCII color plugin, which are adding this data for each log line. Please review your plugins for any that annotate log files. If you generate a support bundle and post the plugins.txt here perhaps we can identify the issue.

          I am going to try and isolate this behavior to a specific plugin, however in my sample test case I provided above I'm not actually making use of any third party plugins as part of the build and the logs are still bloated. If a plugin were to blame wouldn't they all be disabled in this case?

          Kevin Phillips added a comment - I am going to try and isolate this behavior to a specific plugin, however in my sample test case I provided above I'm not actually making use of any third party plugins as part of the build and the logs are still bloated. If a plugin were to blame wouldn't they all be disabled in this case?

          Andrew Bayer added a comment -

          leedega - if I remember correctly, it'd probably be one of the plugins listed here.

          Andrew Bayer added a comment - leedega - if I remember correctly, it'd probably be one of the plugins listed here .

          Thanks everyone for the suggestions.

          So, in order to help isolate the cause of this problem I created a new Jenkins sandbox that was a mirror configuration of our production server, and I started installing plugins one by one to try and isolate the root cause of this issue.

          So, just to summarize, I installed Jenkins core v2.60.2 with no plugins selected for the initial installation (ie: a clean slate). Then I started by installing 'just' the pipeline-* plugins (ie: those required to create the trivial pipeline job described above) and their dependencies. I then ran a test build of the sample job presented above. I had expected to see the log file would be small and clean since there were no third party plugins installed at all on this server. However, to my surprise the first build of my sample job resulted in a bloated log file! So this suggests to me that something in the core set of pipeline dependencies is causing this bloat.

          For reference, here's a list of the plugins I have installed along with their versions (reminder: every single one of these plugins was required in order for the base 'pipeline' plugin to run):

          matrix-project:
            version: "1.11"
          script-security:
            version: "1.29.1"
          mailer:
            version: "1.20"
          scm-api:
            version: "2.1.1"
          docker-commons:
            version: "1.6"
          ssh-credentials:
            version: "1.13"
          credentials:
            version: "2.1.13"
          junit:
            version: "1.20"
          git:
            version: "3.3.2"
          durable-task:
            version: "1.14"
          git-client:
            version: "2.4.6"
          git-server:
            version: "1.7"
          cloudbees-folder:
            version: "6.0.4"
          workflow-basic-steps:
            version: "2.6"
          workflow-durable-task-step:
            version: "2.12"
          workflow-cps-global-lib:
            version: "2.8"
          workflow-step-api:
            version: "2.12"
          workflow-api:
            version: "2.18"
          authentication-tokens:
            version: "1.3"
          jquery-detached:
            version: "1.2.1"
          workflow-scm-step:
            version: "2.5"
          ace-editor:
            version: "1.1"
          pipeline-rest-api:
            version: "2.8"
          workflow-support:
            version: "2.14"
          workflow-cps:
            version: "2.36.1"
          workflow-job:
            version: "2.11.1"
          docker-workflow:
            version: "1.10"
          display-url-api:
            version: "2.0"
          pipeline-github-lib:
            version: "1.0"
          structs:
            version: "1.9"
          pipeline-milestone-step:
            version: "1.3.1"
          pipeline-build-step:
            version: "2.5.1"
          pipeline-input-step:
            version: "2.7"
          pipeline-stage-step:
            version: "2.2"
          pipeline-graph-analysis:
            version: "1.4"
          handlebars:
            version: "1.1.1"
          momentjs:
            version: "1.1.1"
          pipeline-stage-view:
            version: "2.8"
          branch-api:
            version: "2.0.10"
          workflow-multibranch:
            version: "2.16"
          icon-shim:
            version: "2.0.3"
          plain-credentials:
            version: "1.4"
          credentials-binding:
            version: "1.11"
          pipeline-utility-steps:
            version: "1.3.0"
          pipeline-model-declarative-agent:
            version: "1.1.1"
          pipeline-stage-tags-metadata:
            version: "1.1.8"
          pipeline-model-api:
            version: "1.1.8"
          pipeline-model-extensions:
            version: "1.1.8"
          pipeline-model-definition:
            version: "1.1.8"

           

          Kevin Phillips added a comment - Thanks everyone for the suggestions. So, in order to help isolate the cause of this problem I created a new Jenkins sandbox that was a mirror configuration of our production server, and I started installing plugins one by one to try and isolate the root cause of this issue. So, just to summarize, I installed Jenkins core v2.60.2 with no plugins selected for the initial installation (ie: a clean slate). Then I started by installing 'just' the pipeline-* plugins (ie: those required to create the trivial pipeline job described above) and their dependencies. I then ran a test build of the sample job presented above. I had expected to see the log file would be small and clean since there were no third party plugins installed at all on this server. However, to my surprise the first build of my sample job resulted in a bloated log file! So this suggests to me that something in the core set of pipeline dependencies is causing this bloat. For reference, here's a list of the plugins I have installed along with their versions (reminder: every single one of these plugins was required in order for the base 'pipeline' plugin to run): matrix-project: version: "1.11" script-security: version: "1.29.1" mailer: version: "1.20" scm-api: version: "2.1.1" docker-commons: version: "1.6" ssh-credentials: version: "1.13" credentials: version: "2.1.13" junit: version: "1.20" git: version: "3.3.2" durable-task: version: "1.14" git-client: version: "2.4.6" git-server: version: "1.7" cloudbees-folder: version: "6.0.4" workflow-basic-steps: version: "2.6" workflow-durable-task-step: version: "2.12" workflow-cps-global-lib: version: "2.8" workflow-step-api: version: "2.12" workflow-api: version: "2.18" authentication-tokens: version: "1.3" jquery-detached: version: "1.2.1" workflow-scm-step: version: "2.5" ace-editor: version: "1.1" pipeline-rest-api: version: "2.8" workflow-support: version: "2.14" workflow-cps: version: "2.36.1" workflow-job: version: "2.11.1" docker-workflow: version: "1.10" display-url-api: version: "2.0" pipeline-github-lib: version: "1.0" structs: version: "1.9" pipeline-milestone-step: version: "1.3.1" pipeline-build-step: version: "2.5.1" pipeline-input-step: version: "2.7" pipeline-stage-step: version: "2.2" pipeline-graph-analysis: version: "1.4" handlebars: version: "1.1.1" momentjs: version: "1.1.1" pipeline-stage-view: version: "2.8" branch-api: version: "2.0.10" workflow-multibranch: version: "2.16" icon-shim: version: "2.0.3" plain-credentials: version: "1.4" credentials-binding: version: "1.11" pipeline-utility-steps: version: "1.3.0" pipeline-model-declarative-agent: version: "1.1.1" pipeline-stage-tags-metadata: version: "1.1.8" pipeline-model-api: version: "1.1.8" pipeline-model-extensions: version: "1.1.8" pipeline-model-definition: version: "1.1.8"  
          Kevin Phillips made changes -
          Environment Original: jenkins core 2.46.3
          pipeline v2.6
          New: jenkins core 2.60.2
          pipeline v2.6
          Kevin Phillips made changes -
          Environment Original: jenkins core 2.60.2
          pipeline v2.6
          New: jenkins core v2.43.3, v2.60.2, v2.89.2
          pipeline v2.6

            jglick Jesse Glick
            leedega Kevin Phillips
            Votes:
            11 Vote for this issue
            Watchers:
            31 Start watching this issue

              Created:
              Updated:
              Resolved: