Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60907

Poor performance with jenkins.model.StandardArtifactManager.disableTrafficCompression=true

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Fixed
    • Component/s: core
    • Labels:
    • Environment:
      Jenkins 2.204.1
      Linux (Ubuntu 18.04)
      OpenJDK 8u242
    • Similar Issues:
    • Released As:
      Jenkins 2.220

      Description

      With the new jenkins.model.StandardArtifactManager.disableTrafficCompression property (JENKINS-26008 / PR #4205), I was expecting improved performances when archiving artifacts from slaves to master. Especially when there is no room for further compression (when the artifact is already a compressed archive).
      But my first tests with 2.204.1 and this option are disappointing. On my test setup:

      • with traffic compression (default setting), archiving a ~200 MB .tar.gz file consistently takes ~50 seconds (which is really not great already)
      • without traffic compression (the new option), it now takes ~240 seconds!

      I suspect this is because FilePath.TarCompression.NONE lacks a layer of buffering input/output streams, which is there in FilePath.TarCompression.GZIP:
      FilePath.java#L758

      • GZIPInputStream is a buffering InputStream (here with buffer size set to 8K)
      • GZIPOutputStream is a buffering OutputStream (with a default buffer size of 512), and there is also an additionnal BufferedOutputStream involved (with its default buffer size of 8K)

      I will submit a PR for FilePath.TarCompression.NONE to add the missing buffering layer, so that the difference with GZIP will really be only about compression.

        Attachments

          Activity

          tom_gl Thomas de Grenier de Latour created issue -
          tom_gl Thomas de Grenier de Latour made changes -
          Field Original Value New Value
          Description With the new {{jenkins.model.StandardArtifactManager.disableTrafficCompression}} property (JENKINS-26008 / [PR #4205|https://github.com/jenkinsci/jenkins/pull/4205]), I was expecting improved performances when archiving artifacts from slaves to master. Especially when there is no room for further compression (when the artifact is already a compressed archive).
          But my first tests with 2.204.1 and this option are disappointing. On my test setup:
          - with traffic compression (default setting), archiving a ~200 MB .tar.gz file consistently takes ~50 seconds (which is really not great already)
          - without traffic compression (the new option), it now takes ~240 seconds!

          I suspect this is because {{FilePath.TarCompression.NONE}} lacks a layer of buffering input/output streams, which is there in {{FilePath.TarCompression.GZIP}}:
          [FilePath.java#L758|https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/FilePath.java#L758]
           - {{GZIPInputStream}} is a buffering {{InputStream}} (here with buffer size set to 8K)
           - {{GZIPOutputStream}} is a buffering {{OutputStream}} (with a default buffer size of 512), and here there is also an addition {{BufferedOutputStream}} involved (with its default buffer size of 8K)

          I will submit a PR for {{FilePath.TarCompression.NONE}} to add the missing buffering layer, such that the difference with {{GZIP}} will really only be about compression.
          With the new {{jenkins.model.StandardArtifactManager.disableTrafficCompression}} property (JENKINS-26008 / [PR #4205|https://github.com/jenkinsci/jenkins/pull/4205]), I was expecting improved performances when archiving artifacts from slaves to master. Especially when there is no room for further compression (when the artifact is already a compressed archive).
           But my first tests with 2.204.1 and this option are disappointing. On my test setup:
           - with traffic compression (default setting), archiving a ~200 MB .tar.gz file consistently takes ~50 seconds (which is really not great already)
           - without traffic compression (the new option), it now takes ~240 seconds!

          I suspect this is because {{FilePath.TarCompression.NONE}} lacks a layer of buffering input/output streams, which is there in {{FilePath.TarCompression.GZIP}}:
           [FilePath.java#L758|https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/FilePath.java#L758]
           - {{GZIPInputStream}} is a buffering {{InputStream}} (here with buffer size set to 8K)
           - {{GZIPOutputStream}} is a buffering {{OutputStream}} (with a default buffer size of 512), and there is also an additionnal {{BufferedOutputStream}} involved (with its default buffer size of 8K)

          I will submit a PR for {{FilePath.TarCompression.NONE}} to add the missing buffering layer, so that the difference with {{GZIP}} will really be only about compression.
          Hide
          tom_gl Thomas de Grenier de Latour added a comment -

          With buffered streams (PR #4459), my test case with traffic compression disabled now completes in ~30 seconds (vs. ~50 seconds with compression).

          Show
          tom_gl Thomas de Grenier de Latour added a comment - With buffered streams ( PR #4459 ), my test case with traffic compression disabled now completes in ~30 seconds (vs. ~50 seconds with compression).
          Hide
          tom_gl Thomas de Grenier de Latour added a comment -

          PR #4459 is now merged

          Show
          tom_gl Thomas de Grenier de Latour added a comment - PR #4459 is now merged
          tom_gl Thomas de Grenier de Latour made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Fixed but Unreleased [ 10203 ]
          Hide
          tom_gl Thomas de Grenier de Latour added a comment -

          Tagging as a LTS candidate (for 2.204.3), to be considered after it is released in the next weekly. It's not a critical fix, but on the other hand the change is pretty simple, can hardly have any undesirable side effects, and is useful in some use cases.

          Show
          tom_gl Thomas de Grenier de Latour added a comment - Tagging as a LTS candidate (for 2.204.3), to be considered after it is released in the next weekly. It's not a critical fix, but on the other hand the change is pretty simple, can hardly have any undesirable side effects, and is useful in some use cases.
          tom_gl Thomas de Grenier de Latour made changes -
          Labels lts-candidate performance
          tom_gl Thomas de Grenier de Latour made changes -
          Released As https://jenkins.io/changelog/#v2.220
          Status Fixed but Unreleased [ 10203 ] Resolved [ 5 ]
          oleg_nenashev Oleg Nenashev made changes -
          Released As https://jenkins.io/changelog/#v2.220 Jenkins 2.220
          tom_gl Thomas de Grenier de Latour made changes -
          Assignee Thomas de Grenier de Latour [ tom_gl ]
          Status Resolved [ 5 ] Closed [ 6 ]
          danielbeck Daniel Beck made changes -
          Labels lts-candidate performance performance

            People

            Assignee:
            tom_gl Thomas de Grenier de Latour
            Reporter:
            tom_gl Thomas de Grenier de Latour
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: