Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10629

Tar implimentation can't handle > 8GB and doesn't error out.

    XMLWordPrintable

Details

    Description

      During archiving from a slave the untaring can fail with files > 8GB. This is because the tar creator makes a corrupt archive. I think it doesn't properly null terminate the size value when the file size is gets to the 12th digit - which is a violation of tar spec (from Wikipedia tar article: Numeric values are encoded in octal numbers using ASCII digits, with leading zeroes. For historical reasons, a final NUL or space character should be used. Thus although there are 12 bytes reserved for storing the file size, only 11 octal digits can be stored. This gives a maximum file size of 8 gigabytes on archived files.)
      Changing to gnutar might be nice to handle large files, or at least making the library error out when it hits large files rather than happily building corrupt archives.

      Error caused by this bug (due to reading the corrupt header):

      Archiving artifacts
      ERROR: Publisher hudson.tasks.ArtifactArchiver aborted due to exception
      java.lang.IllegalArgumentException: Negative time
      at java.io.File.setLastModified(File.java:1258)
      at hudson.FilePath.readFromTar(FilePath.java:1605)
      at hudson.FilePath.copyRecursiveTo(FilePath.java:1530)
      at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:117)
      at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:649)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:625)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:603)
      at hudson.model.Build$RunnerImpl.post2(Build.java:161)
      at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:572)
      at hudson.model.Run.run(Run.java:1386)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:145)
      Finished: FAILURE

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-25006 Archiving of huge files from remote workspace fails with exception

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-29140 Temporary file appears to leak data

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-28013 Archiving artifiacts fails with java.io.IOException: Truncated TAR archive

          • Blocker - Blocks development and/or testing work, production could not run.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-28012 "Failed to parse POMs" during archiving of poms when running maven job

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-21905 intermittent Failed to archive artifacts: Failed to extract caused by Unexpected end of ZLIB input stream

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          links to

          Web Link PR 1670

          Activity

            Ascending order - Click to sort in descending order
            View 48 older comments
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kanstantsin Shautsou
            Path:
            core/pom.xml
            core/src/main/java/hudson/FilePath.java
            core/src/main/java/hudson/org/apache/tools/tar/TarInputStream.java
            core/src/main/java/hudson/org/apache/tools/tar/TarOutputStream.java
            core/src/main/java/hudson/util/io/TarArchiver.java
            http://jenkins-ci.org/commit/jenkins/6e5740eb5be85ede0cbd01d900cb6d5fbf71ee62
            Log:
            Revert "Revert "JENKINS-10629 - Migrate the Tar archives handling code to commons-compress""

            This reverts commit 8f1280a85c54ea6150b15c38303464ab23b32e92.

            (cherry picked from commit 03bd5959cb94a0e626c5ace910d14c3083a9db2c)

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kanstantsin Shautsou Path: core/pom.xml core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/org/apache/tools/tar/TarInputStream.java core/src/main/java/hudson/org/apache/tools/tar/TarOutputStream.java core/src/main/java/hudson/util/io/TarArchiver.java http://jenkins-ci.org/commit/jenkins/6e5740eb5be85ede0cbd01d900cb6d5fbf71ee62 Log: Revert "Revert " JENKINS-10629 - Migrate the Tar archives handling code to commons-compress"" This reverts commit 8f1280a85c54ea6150b15c38303464ab23b32e92. (cherry picked from commit 03bd5959cb94a0e626c5ace910d14c3083a9db2c)
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kanstantsin Shautsou
            Path:
            core/src/main/java/hudson/util/io/TarArchiver.java
            http://jenkins-ci.org/commit/jenkins/2b63fea84739c1ae900e6997efc18658bcdbaee0
            Log:
            Revert "Revert "FIXED JENKINS-10629] - Enable BigNumber mode to support archiving of files with size >8Gb""

            This reverts commit ee57300963ca0137565f61fe314d459b627ad74b.

            (cherry picked from commit 2ef29215c2b311a9d5c2e515268285b5623ce833)

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kanstantsin Shautsou Path: core/src/main/java/hudson/util/io/TarArchiver.java http://jenkins-ci.org/commit/jenkins/2b63fea84739c1ae900e6997efc18658bcdbaee0 Log: Revert "Revert "FIXED JENKINS-10629 ] - Enable BigNumber mode to support archiving of files with size >8Gb"" This reverts commit ee57300963ca0137565f61fe314d459b627ad74b. (cherry picked from commit 2ef29215c2b311a9d5c2e515268285b5623ce833)
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kanstantsin Shautsou
            Path:
            core/src/main/java/hudson/util/io/TarArchiver.java
            core/src/test/java/hudson/FilePathTest.java
            http://jenkins-ci.org/commit/jenkins/fd1e392fd4cb9d9feade9da7540c00921d32d816
            Log:
            [FIXED JENKINS-10629] Unbroke stream with new tar implementation.

            Causes bytes lost and truncated tar archive.
            TarBuffer not used in TarArchiveOutputStream.

            (cherry picked from commit 3e187a026408d5ca74202f7e26dd565cde2e87d0)

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kanstantsin Shautsou Path: core/src/main/java/hudson/util/io/TarArchiver.java core/src/test/java/hudson/FilePathTest.java http://jenkins-ci.org/commit/jenkins/fd1e392fd4cb9d9feade9da7540c00921d32d816 Log: [FIXED JENKINS-10629] Unbroke stream with new tar implementation. Causes bytes lost and truncated tar archive. TarBuffer not used in TarArchiveOutputStream. (cherry picked from commit 3e187a026408d5ca74202f7e26dd565cde2e87d0)
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #4358
            Revert "Revert "JENKINS-10629 - Migrate the Tar archives handling code to commons-compress"" (Revision 6e5740eb5be85ede0cbd01d900cb6d5fbf71ee62)
            Revert "Revert "FIXED JENKINS-10629] - Enable BigNumber mode to support archiving of files with size >8Gb"" (Revision 2b63fea84739c1ae900e6997efc18658bcdbaee0)
            [FIXED JENKINS-10629] Unbroke stream with new tar implementation. (Revision fd1e392fd4cb9d9feade9da7540c00921d32d816)

            Result = UNSTABLE
            ogondza : 6e5740eb5be85ede0cbd01d900cb6d5fbf71ee62
            Files :

            • core/src/main/java/hudson/org/apache/tools/tar/TarInputStream.java
            • core/src/main/java/hudson/FilePath.java
            • core/src/main/java/hudson/util/io/TarArchiver.java
            • core/src/main/java/hudson/org/apache/tools/tar/TarOutputStream.java
            • core/pom.xml

            ogondza : 2b63fea84739c1ae900e6997efc18658bcdbaee0
            Files :

            • core/src/main/java/hudson/util/io/TarArchiver.java

            ogondza : fd1e392fd4cb9d9feade9da7540c00921d32d816
            Files :

            • core/src/test/java/hudson/FilePathTest.java
            • core/src/main/java/hudson/util/io/TarArchiver.java
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #4358 Revert "Revert " JENKINS-10629 - Migrate the Tar archives handling code to commons-compress"" (Revision 6e5740eb5be85ede0cbd01d900cb6d5fbf71ee62) Revert "Revert "FIXED JENKINS-10629 ] - Enable BigNumber mode to support archiving of files with size >8Gb"" (Revision 2b63fea84739c1ae900e6997efc18658bcdbaee0) [FIXED JENKINS-10629] Unbroke stream with new tar implementation. (Revision fd1e392fd4cb9d9feade9da7540c00921d32d816) Result = UNSTABLE ogondza : 6e5740eb5be85ede0cbd01d900cb6d5fbf71ee62 Files : core/src/main/java/hudson/org/apache/tools/tar/TarInputStream.java core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/util/io/TarArchiver.java core/src/main/java/hudson/org/apache/tools/tar/TarOutputStream.java core/pom.xml ogondza : 2b63fea84739c1ae900e6997efc18658bcdbaee0 Files : core/src/main/java/hudson/util/io/TarArchiver.java ogondza : fd1e392fd4cb9d9feade9da7540c00921d32d816 Files : core/src/test/java/hudson/FilePathTest.java core/src/main/java/hudson/util/io/TarArchiver.java
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: tfennelly
            Path:
            test/src/main/java/hudson/core/PluginManagerOverrideTest.java
            test/src/main/resources/hudson/core/PluginManagerOverrideTest/BasicPluginManagerOverride/newview.jelly
            http://jenkins-ci.org/commit/jenkins-test-harness/9823d43e2d9ebd9cccad7efca8eb965bc4aafedf
            Log:
            Merge branch 'master' into upgrade-htmlunit

            • master:
              commons-compress -> 1.10
              Noting #1788
              Remove hardcode in tar test
              [FIXED JENKINS-10629] Unbroke stream with new tar implementation.
              Revert "Revert "FIXED JENKINS-10629] - Enable BigNumber mode to support archiving of files with size >8Gb""
              Revert "Revert "JENKINS-10629 - Migrate the Tar archives handling code to commons-compress""
              Fix at-since from PR #1788
              Merge PR #1788: Make plugin manager pluggable

            Originally-Committed-As: 3f98011bd4aae0ee6956d264c952782452389f79

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: tfennelly Path: test/src/main/java/hudson/core/PluginManagerOverrideTest.java test/src/main/resources/hudson/core/PluginManagerOverrideTest/BasicPluginManagerOverride/newview.jelly http://jenkins-ci.org/commit/jenkins-test-harness/9823d43e2d9ebd9cccad7efca8eb965bc4aafedf Log: Merge branch 'master' into upgrade-htmlunit master: commons-compress -> 1.10 Noting #1788 Remove hardcode in tar test [FIXED JENKINS-10629] Unbroke stream with new tar implementation. Revert "Revert "FIXED JENKINS-10629 ] - Enable BigNumber mode to support archiving of files with size >8Gb"" Revert "Revert " JENKINS-10629 - Migrate the Tar archives handling code to commons-compress"" Fix at-since from PR #1788 Merge PR #1788: Make plugin manager pluggable Originally-Committed-As: 3f98011bd4aae0ee6956d264c952782452389f79

            People

              integer Kanstantsin Shautsou
              keibaker keith baker
              Votes:
              6 Vote for this issue
              Watchers:
              26 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: