[JENKINS-21905] intermittent Failed to archive artifacts: Failed to extract caused by Unexpected end of ZLIB input stream

Type: Bug
Resolution: Fixed
Priority: Critical
Component/s: core
Labels:
None

Similar Issues:
Powered by SuggestiMate

Show

Since approximately 1/30 we have seen 15 very intermittent archiving failures in Jenkins, of the form:

Archiving artifacts
ERROR: Failed to archive artifacts: versions/**, archive/**
java.io.IOException: Failed to extract /foo/bar/slave/workspace/someJob/transfer of 7 files
	at hudson.FilePath.readFromTar(FilePath.java:2088)
	at hudson.FilePath.copyRecursiveTo(FilePath.java:2000)
	at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:57)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:140)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:784)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:756)
	at hudson.model.Build$BuildExecution.post2(Build.java:183)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:705)
	at hudson.model.Run.execute(Run.java:1695)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:231)
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
	at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:186)
	at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106)
	at org.apache.tools.tar.TarBuffer.readBlock(TarBuffer.java:257)
	at org.apache.tools.tar.TarBuffer.readRecord(TarBuffer.java:223)
	at hudson.org.apache.tools.tar.TarInputStream.getNextEntry(TarInputStream.java:228)
	at hudson.FilePath.readFromTar(FilePath.java:2066)
	... 12 more
Build step 'Archive the artifacts' changed build result to FAILURE

We definitely have seen this running 1.548 and later, especially with 1.549. I cannot confirm if we saw it with 1.547 or earlier, but if so, I think it was less frequent. It seems to be a relatively new issue we are seeing.

is related to

JENKINS-10629 Tar implimentation can't handle > 8GB and doesn't error out.

Resolved

JENKINS-19473 failed to archive slave artifacts. Unexpected end of ZLIB input stream

Closed

links to

PR #2976

Steve Roth created issue - 2014-02-21 00:39

Steve Roth added a comment - 2014-02-21 00:41

I saw ~~JENKINS-19473~~, but this seems to be a different issue. The caused by clause is different. [in that case, it was a java.lang.ArrayIndexOutOfBoundsException]

Steve Roth added a comment - 2014-02-21 00:41 I saw JENKINS-19473 , but this seems to be a different issue. The caused by clause is different. [in that case, it was a java.lang.ArrayIndexOutOfBoundsException]

Steve Roth made changes - 2014-02-21 00:42

Description

Original: Since approximately 1/30 we have seen 15 very intermittent archiving failures in Jenkins, of the form:

{noformat}
Archiving artifacts
ERROR: Failed to archive artifacts: versions/**, archive/**
java.io.IOException: Failed to extract /foo/bar/slave/workspace/someJob/transfer of 7 files
at hudson.FilePath.readFromTar(FilePath.java:2088)
at hudson.FilePath.copyRecursiveTo(FilePath.java:2000)
at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:57)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:140)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:784)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:756)
at hudson.model.Build$BuildExecution.post2(Build.java:183)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:705)
at hudson.model.Run.execute(Run.java:1695)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:231)
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:186)
at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106)
at org.apache.tools.tar.TarBuffer.readBlock(TarBuffer.java:257)
at org.apache.tools.tar.TarBuffer.readRecord(TarBuffer.java:223)
at hudson.org.apache.tools.tar.TarInputStream.getNextEntry(TarInputStream.java:228)
at hudson.FilePath.readFromTar(FilePath.java:2066)
... 12 more
Build step 'Archive the artifacts' changed build result to FAILURE
{noformat}

We definitely have seen this running 1.549 and later. I cannot confirm if we saw it with 1.548 or earlier, but I think it was less frequent. It seems to be a relatively new issue we are seeing.

New: Since approximately 1/30 we have seen 15 very intermittent archiving failures in Jenkins, of the form:

{noformat}
Archiving artifacts
ERROR: Failed to archive artifacts: versions/**, archive/**
java.io.IOException: Failed to extract /foo/bar/slave/workspace/someJob/transfer of 7 files
at hudson.FilePath.readFromTar(FilePath.java:2088)
at hudson.FilePath.copyRecursiveTo(FilePath.java:2000)
at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:57)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:140)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:784)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:756)
at hudson.model.Build$BuildExecution.post2(Build.java:183)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:705)
at hudson.model.Run.execute(Run.java:1695)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:231)
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
at com.jcraft.jzlib.InflaterInputStream.fill(InflaterInputStream.java:186)
at com.jcraft.jzlib.InflaterInputStream.read(InflaterInputStream.java:106)
at org.apache.tools.tar.TarBuffer.readBlock(TarBuffer.java:257)
at org.apache.tools.tar.TarBuffer.readRecord(TarBuffer.java:223)
at hudson.org.apache.tools.tar.TarInputStream.getNextEntry(TarInputStream.java:228)
at hudson.FilePath.readFromTar(FilePath.java:2066)
... 12 more
Build step 'Archive the artifacts' changed build result to FAILURE
{noformat}

We definitely have seen this running 1.548 and later, especially with 1.549. I cannot confirm if we saw it with 1.547 or earlier, but if so, I think it was less frequent. It seems to be a relatively new issue we are seeing.

Steve Roth added a comment - 2014-02-21 02:05

Forgot to mention, we are seeing this even in the latest Jenkins version (1.551)

Steve Roth added a comment - 2014-02-21 02:05 Forgot to mention, we are seeing this even in the latest Jenkins version (1.551)

Jesse Glick made changes - 2014-02-21 14:45

Link

New: This issue is related to ~~JENKINS-19473~~ [ ~~JENKINS-19473~~ ]

Jesse Glick added a comment - 2014-02-21 14:48

If it is reproducible, it can probably be fixed. Otherwise there is not much to go on from the stack trace. Best would be to capture a complete copy of the files which you were attempting to archive, and check whether a fresh job can just unzip those into its workspace, archive all files, and reproduce the bug.

Jesse Glick added a comment - 2014-02-21 14:48 If it is reproducible, it can probably be fixed. Otherwise there is not much to go on from the stack trace. Best would be to capture a complete copy of the files which you were attempting to archive, and check whether a fresh job can just unzip those into its workspace, archive all files, and reproduce the bug.

Steve Roth added a comment - 2014-03-16 03:17 - edited

Thanks Jesse. I've been able to capture artifacts for two cases in which archiving subsequently failed with this exception. Unfortunately, I cannot repro the issue in a test job by unpacking and archiving. So it seems like there might be more to this issue than just the file-to-archive structure.

Is there any logging I can enable which might be helpful in identifying the underlying issue?

Steve Roth added a comment - 2014-03-16 03:17 - edited Thanks Jesse. I've been able to capture artifacts for two cases in which archiving subsequently failed with this exception. Unfortunately, I cannot repro the issue in a test job by unpacking and archiving. So it seems like there might be more to this issue than just the file-to-archive structure. Is there any logging I can enable which might be helpful in identifying the underlying issue?

Jesse Glick added a comment - 2014-03-17 14:21

There is no relevant logging. Fixing this would really hinge on finding a fully reproducible test case.

Or, someone with detailed knowledge of jzlib would need to inject low-level diagnostics, as we did for ~~JENKINS-20618~~ and ~~JENKINS-20114~~. Unfortunately an EOFException is very generic and I cannot think of a diagnostic other than capturing the entire stream to that point, which would be unacceptable from a performance standpoint. (How large are your artifacts?)

It would be nice to use the standard gzip support in the JRE. Unfortunately, that seems to cause occasional 100% CPU freezes that were a major issue for many Jenkins users (worse than issues like this).

Jesse Glick added a comment - 2014-03-17 14:21 There is no relevant logging. Fixing this would really hinge on finding a fully reproducible test case. Or, someone with detailed knowledge of jzlib would need to inject low-level diagnostics, as we did for JENKINS-20618 and JENKINS-20114 . Unfortunately an EOFException is very generic and I cannot think of a diagnostic other than capturing the entire stream to that point, which would be unacceptable from a performance standpoint. (How large are your artifacts?) It would be nice to use the standard gzip support in the JRE. Unfortunately, that seems to cause occasional 100% CPU freezes that were a major issue for many Jenkins users (worse than issues like this).

Steve Roth added a comment - 2014-05-04 06:23

FWIW, I have not seen this in quite awhile. We are currently running 1.558 so it is possible this issue is no longer occurring. I will monitor for the next couple of weeks and close if I do not see it again.

Steve Roth added a comment - 2014-05-04 06:23 FWIW, I have not seen this in quite awhile. We are currently running 1.558 so it is possible this issue is no longer occurring. I will monitor for the next couple of weeks and close if I do not see it again.

Steve Roth added a comment - 2014-07-09 03:00

I have not seen this in more than a month – closing. Thanks!

Steve Roth added a comment - 2014-07-09 03:00 I have not seen this in more than a month – closing. Thanks!

Assignee:: Oliver Gondža

Reporter:: Steve Roth

Votes:: 10 Vote for this issue

Watchers:: 25 Start watching this issue

Created:: 2014-02-21 00:39

Updated:: 2017-09-24 18:15

Resolved:: 2017-09-24 18:13

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Steve Roth added a comment - 2014-02-21 00:41

Expand comment: Steve Roth added a comment - 2014-02-21 00:41

Collapse comment: Steve Roth added a comment - 2014-02-21 02:05

Expand comment: Steve Roth added a comment - 2014-02-21 02:05

Collapse comment: Jesse Glick added a comment - 2014-02-21 14:48

Expand comment: Jesse Glick added a comment - 2014-02-21 14:48

Collapse comment: Steve Roth added a comment - 2014-03-16 03:17, Edited by Steve Roth - 2014-03-16 03:18

Expand comment: Steve Roth added a comment - 2014-03-16 03:17, Edited by Steve Roth - 2014-03-16 03:18

Collapse comment: Jesse Glick added a comment - 2014-03-17 14:21

Expand comment: Jesse Glick added a comment - 2014-03-17 14:21

Collapse comment: Steve Roth added a comment - 2014-05-04 06:23

Expand comment: Steve Roth added a comment - 2014-05-04 06:23

Collapse comment: Steve Roth added a comment - 2014-07-09 03:00

Expand comment: Steve Roth added a comment - 2014-07-09 03:00

People

Dates