Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60907

Poor performance with jenkins.model.StandardArtifactManager.disableTrafficCompression=true

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • core
    • Jenkins 2.204.1
      Linux (Ubuntu 18.04)
      OpenJDK 8u242
    • Jenkins 2.220

      With the new jenkins.model.StandardArtifactManager.disableTrafficCompression property (JENKINS-26008 / PR #4205), I was expecting improved performances when archiving artifacts from slaves to master. Especially when there is no room for further compression (when the artifact is already a compressed archive).
      But my first tests with 2.204.1 and this option are disappointing. On my test setup:

      • with traffic compression (default setting), archiving a ~200 MB .tar.gz file consistently takes ~50 seconds (which is really not great already)
      • without traffic compression (the new option), it now takes ~240 seconds!

      I suspect this is because FilePath.TarCompression.NONE lacks a layer of buffering input/output streams, which is there in FilePath.TarCompression.GZIP:
      FilePath.java#L758

      • GZIPInputStream is a buffering InputStream (here with buffer size set to 8K)
      • GZIPOutputStream is a buffering OutputStream (with a default buffer size of 512), and there is also an additionnal BufferedOutputStream involved (with its default buffer size of 8K)

      I will submit a PR for FilePath.TarCompression.NONE to add the missing buffering layer, so that the difference with GZIP will really be only about compression.

          [JENKINS-60907] Poor performance with jenkins.model.StandardArtifactManager.disableTrafficCompression=true

          Thomas de Grenier de Latour created issue -
          Thomas de Grenier de Latour made changes -
          Description Original: With the new {{jenkins.model.StandardArtifactManager.disableTrafficCompression}} property (JENKINS-26008 / [PR #4205|https://github.com/jenkinsci/jenkins/pull/4205]), I was expecting improved performances when archiving artifacts from slaves to master. Especially when there is no room for further compression (when the artifact is already a compressed archive).
          But my first tests with 2.204.1 and this option are disappointing. On my test setup:
          - with traffic compression (default setting), archiving a ~200 MB .tar.gz file consistently takes ~50 seconds (which is really not great already)
          - without traffic compression (the new option), it now takes ~240 seconds!

          I suspect this is because {{FilePath.TarCompression.NONE}} lacks a layer of buffering input/output streams, which is there in {{FilePath.TarCompression.GZIP}}:
          [FilePath.java#L758|https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/FilePath.java#L758]
           - {{GZIPInputStream}} is a buffering {{InputStream}} (here with buffer size set to 8K)
           - {{GZIPOutputStream}} is a buffering {{OutputStream}} (with a default buffer size of 512), and here there is also an addition {{BufferedOutputStream}} involved (with its default buffer size of 8K)

          I will submit a PR for {{FilePath.TarCompression.NONE}} to add the missing buffering layer, such that the difference with {{GZIP}} will really only be about compression.
          New: With the new {{jenkins.model.StandardArtifactManager.disableTrafficCompression}} property (JENKINS-26008 / [PR #4205|https://github.com/jenkinsci/jenkins/pull/4205]), I was expecting improved performances when archiving artifacts from slaves to master. Especially when there is no room for further compression (when the artifact is already a compressed archive).
           But my first tests with 2.204.1 and this option are disappointing. On my test setup:
           - with traffic compression (default setting), archiving a ~200 MB .tar.gz file consistently takes ~50 seconds (which is really not great already)
           - without traffic compression (the new option), it now takes ~240 seconds!

          I suspect this is because {{FilePath.TarCompression.NONE}} lacks a layer of buffering input/output streams, which is there in {{FilePath.TarCompression.GZIP}}:
           [FilePath.java#L758|https://github.com/jenkinsci/jenkins/blob/jenkins-2.204.1/core/src/main/java/hudson/FilePath.java#L758]
           - {{GZIPInputStream}} is a buffering {{InputStream}} (here with buffer size set to 8K)
           - {{GZIPOutputStream}} is a buffering {{OutputStream}} (with a default buffer size of 512), and there is also an additionnal {{BufferedOutputStream}} involved (with its default buffer size of 8K)

          I will submit a PR for {{FilePath.TarCompression.NONE}} to add the missing buffering layer, so that the difference with {{GZIP}} will really be only about compression.

          With buffered streams (PR #4459), my test case with traffic compression disabled now completes in ~30 seconds (vs. ~50 seconds with compression).

          Thomas de Grenier de Latour added a comment - With buffered streams ( PR #4459 ), my test case with traffic compression disabled now completes in ~30 seconds (vs. ~50 seconds with compression).

          PR #4459 is now merged

          Thomas de Grenier de Latour added a comment - PR #4459 is now merged
          Thomas de Grenier de Latour made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: Open [ 1 ] New: Fixed but Unreleased [ 10203 ]

          Tagging as a LTS candidate (for 2.204.3), to be considered after it is released in the next weekly. It's not a critical fix, but on the other hand the change is pretty simple, can hardly have any undesirable side effects, and is useful in some use cases.

          Thomas de Grenier de Latour added a comment - Tagging as a LTS candidate (for 2.204.3), to be considered after it is released in the next weekly. It's not a critical fix, but on the other hand the change is pretty simple, can hardly have any undesirable side effects, and is useful in some use cases.
          Thomas de Grenier de Latour made changes -
          Labels New: lts-candidate performance
          Thomas de Grenier de Latour made changes -
          Released As New: https://jenkins.io/changelog/#v2.220
          Status Original: Fixed but Unreleased [ 10203 ] New: Resolved [ 5 ]
          Oleg Nenashev made changes -
          Released As Original: https://jenkins.io/changelog/#v2.220 New: Jenkins 2.220

            tom_gl Thomas de Grenier de Latour
            tom_gl Thomas de Grenier de Latour
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: