Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64401

Performance issues in support-core plugin when task logs contains big text files

    XMLWordPrintable

    Details

    • Similar Issues:
    • Released As:
      support-core:2.71

      Description

      In case under $JENKINS_HOME/logs/tasks/ we have big files - I can easily reproduce this with 5 text files with around 50MB each of them - the support-core plugin takes a lot of minutes to generate the bundle.

      To reproduce the issue:

      1. We can have Anonymize support bundle contents disabled. To have this enabled, the performance issue is even worse, but we are concentrating the efforts when the anonymization is disabled.
      2. Insert 5 text files with around 50Mb each of them under $JENKINS_HOME/logs/tasks/. The files must have this pattern

      -rw-r--r--   1 user staff 220688805 Dec  3 10:46 Workspace clean-up.log
      -rw-r--r--   1 user staff 220688241 Dec  3 10:46 Workspace clean-up.log.1
      -rw-r--r--   1 user staff 220980327 Dec  3 10:46 Workspace clean-up.log.2
      -rw-r--r--   1 user staff 441376347 Dec  3 10:46 Workspace clean-up.log.3
      -rw-r--r--   1 user staff 441375743 Dec  3 10:46 Workspace clean-up.log.4
      -rw-r--r--   1 user staff 220687588 Dec  3 10:46 Workspace clean-up.log.5
      

      3. Generate a support bundle by ONLY enabling Master Task Log Recorders to easily reproduce the issue and

      The key seems to be that we are using FilterOutputStream.write when we should be using a BufferedOutputStream.write to improve the writing process.

      In my case I have 1.7 Gb of data under $JENKINS_HOME/logs/tasks/ (6 files of 211 Mb each of them + rest of files are very small) and the generation takes 2 full CPUs of Mac Book Pro - so it also has high CPU consumption during this time.

      We will see a stacktrace like the one below in the thread dumps when this issue happens.

      "Handling POST /jenkins/support/generateAllBundles from 127.0.0.1 : qtp928487227-97" Id=97 Group=main RUNNABLE
      	at java.util.zip.Deflater.deflateBytes(Native Method)
      	at java.util.zip.Deflater.deflate(Deflater.java:442)
      	-  locked java.util.zip.ZStreamRef@446ac8b3
      	at java.util.zip.Deflater.deflate(Deflater.java:364)
      	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflate(StreamCompressor.java:262)
      	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflateUntilInputIsNeeded(StreamCompressor.java:257)
      	at org.apache.commons.compress.archivers.zip.StreamCompressor.writeDeflated(StreamCompressor.java:238)
      	at org.apache.commons.compress.archivers.zip.StreamCompressor.write(StreamCompressor.java:205)
      	at org.apache.commons.compress.archivers.zip.ZipArchiveOutputStream.write(ZipArchiveOutputStream.java:926)
      	at org.apache.commons.compress.archivers.ArchiveOutputStream.write(ArchiveOutputStream.java:109)
      	at java.io.FilterOutputStream.write(FilterOutputStream.java:77)
      	at java.io.FilterOutputStream.write(FilterOutputStream.java:125)
      	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2315)
      	at org.apache.commons.io.IOUtils.copy(IOUtils.java:2270)
      	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2291)
      	at org.apache.commons.io.IOUtils.copy(IOUtils.java:2246)
      	at com.cloudbees.jenkins.support.api.BaseFileContent.writeTo(BaseFileContent.java:79)
      	at com.cloudbees.jenkins.support.api.FileContent.writeTo(FileContent.java:72)
      	at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:351)
      	at com.cloudbees.jenkins.support.SupportPlugin.writeBundle(SupportPlugin.java:298)
      	at com.cloudbees.jenkins.support.SupportAction.prepareBundle(SupportAction.java:356)
      	at com.cloudbees.jenkins.support.SupportAction.doGenerateAllBundles(SupportAction.java:306)
      

        Attachments

          Activity

            People

            Assignee:
            fbelzunc Félix Belzunce Arcos
            Reporter:
            fbelzunc Félix Belzunce Arcos
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: