Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64401

Performance issues in support-core plugin when task logs contains big text files

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      support-core:2.71

      Description

      In case under $JENKINS_HOME/logs/tasks/ we have big files - I can easily reproduce this with 5 text files with around 50MB each of them - the support-core plugin takes a lot of minutes to generate the bundle.

      To reproduce the issue:

      1. We can have Anonymize support bundle contents disabled. To have this enabled, the performance issue is even worse, but we are concentrating the efforts when the anonymization is disabled.
      2. Insert 5 text files with around 50Mb each of them under $JENKINS_HOME/logs/tasks/. The files must have this pattern

      -rw-r--r--   1 user staff 220688805 Dec  3 10:46 Workspace clean-up.log
      -rw-r--r--   1 user staff 220688241 Dec  3 10:46 Workspace clean-up.log.1
      -rw-r--r--   1 user staff 220980327 Dec  3 10:46 Workspace clean-up.log.2
      -rw-r--r--   1 user staff 441376347 Dec  3 10:46 Workspace clean-up.log.3
      -rw-r--r--   1 user staff 441375743 Dec  3 10:46 Workspace clean-up.log.4
      -rw-r--r--   1 user staff 220687588 Dec  3 10:46 Workspace clean-up.log.5
      

      3. Generate a support bundle by ONLY enabling Master Task Log Recorders to easily reproduce the issue and

      The key seems to be that we are using FilterOutputStream.write when we should be using a BufferedOutputStream.write to improve the writing process.

      In my case I have 1.7 Gb of data under $JENKINS_HOME/logs/tasks/ (6 files of 211 Mb each of them + rest of files are very small) and the generation takes 2 full CPUs of Mac Book Pro - so it also has high CPU consumption during this time.

      We will see a stacktrace like the one below in the thread dumps when this issue happens.

      "Handling POST /jenkins/support/generateAllBundles from 127.0.0.1 : qtp928487227-97" Id=97 Group=main RUNNABLE
      	at java.util.zip.Deflater.deflateBytes(Native Method)
      	at java.util.zip.Deflater.deflate(Deflater.java:442)
      	-  locked java.util.zip.ZStreamRef@446ac8b3
      	at java.util.zip.Deflater.deflate(Deflater.java:364)
      	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflate(StreamCompressor.java:262)
      	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflateUntilInputIsNeeded(StreamCompressor.java:257)
      	at org.apache.commons.compress.archivers.zip.StreamCompressor.writeDeflated(StreamCompressor.java:238)
      	at org.apache.commons.compress.archivers.zip.StreamCompressor.write(StreamCompressor.java:205)
      	at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.write(ZipArchiveOutputStream.java:926)
      	at org.apache.commons.compress.archivers.ArchiveOutputStream.write(ArchiveOutputStream.java:109)
      	at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
      	at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
      	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2315)
      	at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
      	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
      	at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
      	at com.cloudbees.jenkins.support.api.BaseFileContent.writeTo(BaseFileContent.java:79)
      	at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:72)
      	at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:351)
      	at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:298)
      	at com.cloudbees.jenkins.support.SupportAction.prepareBundle(SupportAction.java:356)
      	at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:306)
      

        Attachments

          Activity

          fbelzunc Félix Belzunce Arcos created issue -
          fbelzunc Félix Belzunce Arcos made changes -
          Field Original Value New Value
          Description In case under $JENKINS_HOME/logs/tasks/ we have big files - I can easily reproduce this with 5 text files with around 50MB each of them - the support-core plugin takes a lot of minutes to generate the bundle. In case under $JENKINS_HOME/logs/tasks/ we have big files - I can easily reproduce this with 5 text files with around 50MB each of them - the support-core plugin takes a lot of minutes to generate the bundle.

          To reproduce the issue:

          1. We can have {{Anonymize support bundle contents}} disabled. To have this enabled, the performance issue is even worse, but we are concentrating the efforts when the anonymization is disabled.
          2. Insert 5 text files with around 50Mb each of them under {{$JENKINS_HOME/logs/tasks/}}. The files must have this pattern
          {code:java}
          -rw-r--r-- 1 user staff 220688805 Dec 3 10:46 Workspace clean-up.log
          -rw-r--r-- 1 user staff 220688241 Dec 3 10:46 Workspace clean-up.log.1
          -rw-r--r-- 1 user staff 220980327 Dec 3 10:46 Workspace clean-up.log.2
          -rw-r--r-- 1 user staff 441376347 Dec 3 10:46 Workspace clean-up.log.3
          -rw-r--r-- 1 user staff 441375743 Dec 3 10:46 Workspace clean-up.log.4
          -rw-r--r-- 1 user staff 220687588 Dec 3 10:46 Workspace clean-up.log.5
          {code}
          3. Generate a support bundle by ONLY enabling {{Master Task Log Recorders}} to easily reproduce the issue and

          The key seems to be that we are using FilterOutputStream.write when we should be using a BufferedOutputStream.write to improve the writing process.

          In my case I have 1.7 Gb of data under $JENKINS_HOME/logs/tasks/ (6 files of 211 Mb each of them + rest of files are very small) and the generation takes 2 full CPUs of Mac Book Pro - so it also has high CPU consumption during this time.

          We will see a stacktrace like the one below in the thread dumps when this issue happens.

          {code:java}
          "Handling POST /jenkins/support/generateAllBundles from 127.0.0.1 : qtp928487227-97" Id=97 Group=main RUNNABLE
          at java.util.zip.Deflater.deflateBytes(Native Method)
          at java.util.zip.Deflater.deflate(Deflater.java:442)
          - locked java.util.zip.ZStreamRef@446ac8b3
          at java.util.zip.Deflater.deflate(Deflater.java:364)
          at org.apache.commons.compress.archivers.zip.StreamCompressor.deflate(StreamCompressor.java:262)
          at org.apache.commons.compress.archivers.zip.StreamCompressor.deflateUntilInputIsNeeded(StreamCompressor.java:257)
          at org.apache.commons.compress.archivers.zip.StreamCompressor.writeDeflated(StreamCompressor.java:238)
          at org.apache.commons.compress.archivers.zip.StreamCompressor.write(StreamCompressor.java:205)
          at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.write(ZipArchiveOutputStream.java:926)
          at org.apache.commons.compress.archivers.ArchiveOutputStream.write(ArchiveOutputStream.java:109)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
          at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2315)
          at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
          at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
          at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
          at com.cloudbees.jenkins.support.api.BaseFileContent.writeTo(BaseFileContent.java:79)
          at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:72)
          at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:351)
          at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:298)
          at com.cloudbees.jenkins.support.SupportAction.prepareBundle(SupportAction.java:356)
          at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:306)
          {code}
          fbelzunc Félix Belzunce Arcos made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          allan_burdajewicz Allan BURDAJEWICZ made changes -
          Status In Progress [ 3 ] In Review [ 10005 ]
          allan_burdajewicz Allan BURDAJEWICZ made changes -
          Resolution Fixed [ 1 ]
          Status In Review [ 10005 ] Fixed but Unreleased [ 10203 ]
          Show
          allan_burdajewicz Allan BURDAJEWICZ added a comment - Released in https://github.com/jenkinsci/support-core-plugin/releases/tag/support-core-2.71
          allan_burdajewicz Allan BURDAJEWICZ made changes -
          Released As support-core:2.71
          Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]

            People

            Assignee:
            fbelzunc Félix Belzunce Arcos
            Reporter:
            fbelzunc Félix Belzunce Arcos
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: