-
Bug
-
Resolution: Fixed
-
Major
-
None
-
-
2.449
I'm experiencing this problem with pipeline-maven-plugin when it is running 'Generated Artifacts Publisher' when running maven on a z/OS agent (line numbers are as of Jenkins version 2.426.2):
ERROR: [withMaven] WARNING Exception executing Maven reporter 'Generated Artifacts Publisher' / org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher. Please report a bug associated for the component 'pipeline-maven-plugin' at https://issues.jenkins-ci.org java.io.IOException: Exception archiving { map content removed } at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:145) at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:172) at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution2$WithMavenStepExecutionCallBack.finished(WithMavenStepExecution2.java:1359) at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution$TailCall.lambda$onSuccess$0(GeneralNonBlockingStepExecution.java:140) at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:77) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.io.IOException: Failed to extract /path removed/transfer of 8 files at hudson.FilePath.readFromTar(FilePath.java:3068) at hudson.FilePath.copyRecursiveTo(FilePath.java:2828) at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:73) at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:143) ... 9 more Caused by: java.nio.charset.UnmappableCharacterException: Input length = 1 at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:275) at java.base/java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:820) at org.apache.commons.compress.archivers.zip.NioZipEncoding.decode(NioZipEncoding.java:130) at org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:524) at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1543) at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1531) at org.apache.commons.compress.archivers.tar.TarArchiveEntry.<init>(TarArchiveEntry.java:553) at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:464) at hudson.FilePath.readFromTar(FilePath.java:3043) ... 12 more
Code review turned out that this is most likely a problem in hudson.FilePath#copyRecursiveTo implementation (line numbers are as of Jenkins version 2.440):
- Assumption is that the remote part is the agent with a default character set of CP1047 (EBCDIC) and local is a Windows or Linux with something ASCII/UTF-8 based.
- At FilePath.java#L2864 CopyRecursiveRemoteToLocal is instantiated and effectively CopyRecursiveRemoteToLocal#invoke is executed at the agent:
- Calling FilePath#writeToTar() which effectively calls hudson.util.io.ArchiverFactory.TarArchiverFactory#create() which instantiates and returns a TarArchiver.
- The TarArchiver instatiates in its constructor at TarArchiver.java#L53 a TarArchiveOutputStream with constructor org.apache.commons.compress.archivers.tar.TarArchiveOutputStream#TarArchiveOutputStream(final OutputStream os). Using this constructor results at TarArchiveOutputStream.java#L221 in initializing member zipEncoding from call to ZipEncodingHelper.getZipEncoding(null).
- ZipEncodingHelper#getZipEncoding(null) returns effectively the Charset.defaultCharset() being wrapped into a NioZipEncoding object which is stored in TarArchiveOutputStream#zipEncoding.
- TarArchiveOutputStream.java#zipEncoding is used then to encode names written into the tar archive at TarArchiveOutputStream.java#handleLongName(), TarArchiveOutputStream#putArchiveEntry(), and TarArchiveOutputStream.java#L614
- So per above file names are written to the tar archive with agents default character set which is CP1047 in our case.
- At controller side which reads the tar stream from FilePath.java#L2866 calling FilePath#readFromTar() which instantiates TarArchiveInputStream. That has a zipEncoding member, too. It is initialized at TarArchiveInputStream.java#L211 the same way as TarArchiveOutputStream's zipEncoding member. However, since this is executed at the controller it receives whatever the default character set is there.
- Due to the mismatch of character sets used for writing and reading file names in tar stream the reading fails.
I suspect that JENKINS-18558 is related to this issue.
- is related to
-
JENKINS-18558 FilePath's copyRecursiveTo doesn't handle Chinese characters
-
- Open
-
- links to
[JENKINS-72540] hudson.FilePath#copyRecursiveTo() failing if remote and local having different default character sets
Description |
Original:
I'm experiencing this problem with pipeline-maven-plugin when it is running 'Generated Artifacts Publisher' when running maven on a z/OS agent (line numbers are as of Jenkins version 2.426.2):
{code:java} ERROR: [withMaven] WARNING Exception executing Maven reporter 'Generated Artifacts Publisher' / org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher. Please report a bug associated for the component 'pipeline-maven-plugin' at https://issues.jenkins-ci.org java.io.IOException: Exception archiving { map content removed } at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:145) at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:172) at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution2$WithMavenStepExecutionCallBack.finished(WithMavenStepExecution2.java:1359) at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution$TailCall.lambda$onSuccess$0(GeneralNonBlockingStepExecution.java:140) at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:77) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.io.IOException: Failed to extract /path removed/transfer of 8 files at hudson.FilePath.readFromTar(FilePath.java:3068) at hudson.FilePath.copyRecursiveTo(FilePath.java:2828) at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:73) at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:143) ... 9 more Caused by: java.nio.charset.UnmappableCharacterException: Input length = 1 at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:275) at java.base/java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:820) at org.apache.commons.compress.archivers.zip.NioZipEncoding.decode(NioZipEncoding.java:130) at org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:524) at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1543) at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1531) at org.apache.commons.compress.archivers.tar.TarArchiveEntry.<init>(TarArchiveEntry.java:553) at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:464) at hudson.FilePath.readFromTar(FilePath.java:3043) ... 12 more {code} Code review turned out that this is most likely a problem in [hudson.FilePath#copyRecursiveTo |https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2842] implementation: * Assumption is that the remote part is the agent with a default character set of CP1047 (EBCDIC) and local is a Windows or Linux with something ASCII/UTF-8 based. * At [FilePath.java#L2864|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2864] CopyRecursiveRemoteToLocal is instantiated and effectively [CopyRecursiveRemoteToLocal#invoke|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3030%20at%20FilePath.java#L3030] is executed at the agent: ## Calling [FilePath#writeToTar()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3064] which effectively calls [hudson.util.io.ArchiverFactory.TarArchiverFactory#create()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/util/io/ArchiverFactory.java#L88] which instantiates and returns a TarArchiver. ## The TarArchiver instatiates in its constructor at [TarArchiver.java#L53|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/util/io/TarArchiver.java#L53] a TarArchiveOutputStream with constructor [org.apache.commons.compress.archivers.tar.TarArchiveOutputStream#TarArchiveOutputStream(final OutputStream os)|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L146]. Using this constructor results at [TarArchiveOutputStream.java#L221|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L221] in initializing member zipEncoding from call to ZipEncodingHelper.getZipEncoding(null). ## [ZipEncodingHelper#getZipEncoding(null)|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/zip/ZipEncodingHelper.java#L51C42-L51C42] returns effectively the Charset.defaultCharset() being wrapped into a NioZipEncoding object which is stored in TarArchiveOutputStream#zipEncoding. ## [TarArchiveOutputStream.java#zipEncoding|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L130] is used then to encode names written into the tar archive at [TarArchiveOutputStream.java#handleLongName()|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L512], [TarArchiveOutputStream#putArchiveEntry()|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L575], and [TarArchiveOutputStream.java#L614|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L614] * So per above file names are written to the tar archive with agents default character set which is CP1047 in our case. * At controller side which reads the tar stream from [FilePath.java#L2866|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2866] calling [FilePath#readFromTar()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3076] which instantiates [TarArchiveInputStream|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveInputStream.java#L50]. That has a zipEncoding member, too. It is initialized at [TarArchiveInputStream.java#L211|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveInputStream.java#L211] the same way as TarArchiveOutputStream's zipEncoding member. However, since this is executed at the controller it receives whatever the default character set is there. * Due to the mismatch of character sets used for writing and reading file names in tar stream the reading fails. I suspect that JENKINS-18558 is related to this issue. |
New:
I'm experiencing this problem with pipeline-maven-plugin when it is running 'Generated Artifacts Publisher' when running maven on a z/OS agent (line numbers are as of Jenkins version 2.426.2):
{code:java} ERROR: [withMaven] WARNING Exception executing Maven reporter 'Generated Artifacts Publisher' / org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher. Please report a bug associated for the component 'pipeline-maven-plugin' at https://issues.jenkins-ci.org java.io.IOException: Exception archiving { map content removed } at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:145) at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:172) at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution2$WithMavenStepExecutionCallBack.finished(WithMavenStepExecution2.java:1359) at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution$TailCall.lambda$onSuccess$0(GeneralNonBlockingStepExecution.java:140) at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:77) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.io.IOException: Failed to extract /path removed/transfer of 8 files at hudson.FilePath.readFromTar(FilePath.java:3068) at hudson.FilePath.copyRecursiveTo(FilePath.java:2828) at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:73) at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:143) ... 9 more Caused by: java.nio.charset.UnmappableCharacterException: Input length = 1 at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:275) at java.base/java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:820) at org.apache.commons.compress.archivers.zip.NioZipEncoding.decode(NioZipEncoding.java:130) at org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:524) at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1543) at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1531) at org.apache.commons.compress.archivers.tar.TarArchiveEntry.<init>(TarArchiveEntry.java:553) at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:464) at hudson.FilePath.readFromTar(FilePath.java:3043) ... 12 more {code} Code review turned out that this is most likely a problem in [hudson.FilePath#copyRecursiveTo |https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2842] implementation (line numbers are as of Jenkins version 2.440): * Assumption is that the remote part is the agent with a default character set of CP1047 (EBCDIC) and local is a Windows or Linux with something ASCII/UTF-8 based. * At [FilePath.java#L2864|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2864] CopyRecursiveRemoteToLocal is instantiated and effectively [CopyRecursiveRemoteToLocal#invoke|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3030%20at%20FilePath.java#L3030] is executed at the agent: ## Calling [FilePath#writeToTar()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3064] which effectively calls [hudson.util.io.ArchiverFactory.TarArchiverFactory#create()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/util/io/ArchiverFactory.java#L88] which instantiates and returns a TarArchiver. ## The TarArchiver instatiates in its constructor at [TarArchiver.java#L53|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/util/io/TarArchiver.java#L53] a TarArchiveOutputStream with constructor [org.apache.commons.compress.archivers.tar.TarArchiveOutputStream#TarArchiveOutputStream(final OutputStream os)|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L146]. Using this constructor results at [TarArchiveOutputStream.java#L221|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L221] in initializing member zipEncoding from call to ZipEncodingHelper.getZipEncoding(null). ## [ZipEncodingHelper#getZipEncoding(null)|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/zip/ZipEncodingHelper.java#L51C42-L51C42] returns effectively the Charset.defaultCharset() being wrapped into a NioZipEncoding object which is stored in TarArchiveOutputStream#zipEncoding. ## [TarArchiveOutputStream.java#zipEncoding|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L130] is used then to encode names written into the tar archive at [TarArchiveOutputStream.java#handleLongName()|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L512], [TarArchiveOutputStream#putArchiveEntry()|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L575], and [TarArchiveOutputStream.java#L614|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L614] * So per above file names are written to the tar archive with agents default character set which is CP1047 in our case. * At controller side which reads the tar stream from [FilePath.java#L2866|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2866] calling [FilePath#readFromTar()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3076] which instantiates [TarArchiveInputStream|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveInputStream.java#L50]. That has a zipEncoding member, too. It is initialized at [TarArchiveInputStream.java#L211|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveInputStream.java#L211] the same way as TarArchiveOutputStream's zipEncoding member. However, since this is executed at the controller it receives whatever the default character set is there. * Due to the mismatch of character sets used for writing and reading file names in tar stream the reading fails. I suspect that JENKINS-18558 is related to this issue. |
Link | New: This issue is related to JENKINS-18558 [ JENKINS-18558 ] |
Remote Link | New: This issue links to "https://github.com/jenkinsci/jenkins/pull/8860 (Web Link)" [ 29506 ] |
Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
Status | Original: In Progress [ 3 ] | New: In Review [ 10005 ] |
Resolution | New: Fixed [ 1 ] | |
Status | Original: In Review [ 10005 ] | New: Fixed but Unreleased [ 10203 ] |
Released As | New: https://www.jenkins.io/changelog/2.449/ | |
Status | Original: Fixed but Unreleased [ 10203 ] | New: Resolved [ 5 ] |
Released As | Original: https://www.jenkins.io/changelog/2.449/ | New: 2.449 |