Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72540

hudson.FilePath#copyRecursiveTo() failing if remote and local having different default character sets

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • core
    • None
    • 2.449

      I'm experiencing this problem with pipeline-maven-plugin when it is running 'Generated Artifacts Publisher' when running maven on a z/OS agent (line numbers are as of Jenkins version 2.426.2):

      ERROR: [withMaven] WARNING Exception executing Maven reporter 'Generated Artifacts Publisher' / org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher. Please report a bug associated for the component 'pipeline-maven-plugin' at https://issues.jenkins-ci.org 
       java.io.IOException: Exception archiving { map content removed }
       	at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:145)
       	at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:172)
       	at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution2$WithMavenStepExecutionCallBack.finished(WithMavenStepExecution2.java:1359)
       	at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution$TailCall.lambda$onSuccess$0(GeneralNonBlockingStepExecution.java:140)
       	at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:77)
       	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
       	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
       	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
       	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
       	at java.base/java.lang.Thread.run(Thread.java:840)
       Caused by: java.io.IOException: Failed to extract /path removed/transfer of 8 files
       	at hudson.FilePath.readFromTar(FilePath.java:3068)
       	at hudson.FilePath.copyRecursiveTo(FilePath.java:2828)
       	at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:73)
       	at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:143)
       	... 9 more
       Caused by: java.nio.charset.UnmappableCharacterException: Input length = 1
       	at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:275)
       	at java.base/java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:820)
       	at org.apache.commons.compress.archivers.zip.NioZipEncoding.decode(NioZipEncoding.java:130)
       	at org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:524)
       	at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1543)
       	at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1531)
       	at org.apache.commons.compress.archivers.tar.TarArchiveEntry.<init>(TarArchiveEntry.java:553)
       	at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:464)
       	at hudson.FilePath.readFromTar(FilePath.java:3043)
       	... 12 more
      

       

      Code review turned out that this is most likely a problem in hudson.FilePath#copyRecursiveTo implementation (line numbers are as of Jenkins version 2.440):

       

      I suspect that JENKINS-18558 is related to this issue.

          [JENKINS-72540] hudson.FilePath#copyRecursiveTo() failing if remote and local having different default character sets

          Lutz Neugebauer created issue -
          Lutz Neugebauer made changes -
          Description Original: I'm experiencing this problem with pipeline-maven-plugin when it is running 'Generated Artifacts Publisher' when running maven on a z/OS agent (line numbers are as of Jenkins version 2.426.2):
          {code:java}
          ERROR: [withMaven] WARNING Exception executing Maven reporter 'Generated Artifacts Publisher' / org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher. Please report a bug associated for the component 'pipeline-maven-plugin' at https://issues.jenkins-ci.org
           java.io.IOException: Exception archiving { map content removed }
            at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:145)
            at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:172)
            at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution2$WithMavenStepExecutionCallBack.finished(WithMavenStepExecution2.java:1359)
            at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution$TailCall.lambda$onSuccess$0(GeneralNonBlockingStepExecution.java:140)
            at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:77)
            at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
            at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
            at java.base/java.lang.Thread.run(Thread.java:840)
           Caused by: java.io.IOException: Failed to extract /path removed/transfer of 8 files
            at hudson.FilePath.readFromTar(FilePath.java:3068)
            at hudson.FilePath.copyRecursiveTo(FilePath.java:2828)
            at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:73)
            at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:143)
            ... 9 more
           Caused by: java.nio.charset.UnmappableCharacterException: Input length = 1
            at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:275)
            at java.base/java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:820)
            at org.apache.commons.compress.archivers.zip.NioZipEncoding.decode(NioZipEncoding.java:130)
            at org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:524)
            at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1543)
            at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1531)
            at org.apache.commons.compress.archivers.tar.TarArchiveEntry.<init>(TarArchiveEntry.java:553)
            at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:464)
            at hudson.FilePath.readFromTar(FilePath.java:3043)
            ... 12 more
          {code}
           

          Code review turned out that this is most likely a problem in [hudson.FilePath#copyRecursiveTo |https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2842] implementation:
           * Assumption is that the remote part is the agent with a default character set of CP1047 (EBCDIC) and local is a Windows or Linux with something ASCII/UTF-8 based.
           * At [FilePath.java#L2864|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2864] CopyRecursiveRemoteToLocal is instantiated and effectively [CopyRecursiveRemoteToLocal#invoke|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3030%20at%20FilePath.java#L3030] is executed at the agent:
           ## Calling [FilePath#writeToTar()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3064] which effectively calls [hudson.util.io.ArchiverFactory.TarArchiverFactory#create()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/util/io/ArchiverFactory.java#L88] which instantiates and returns a TarArchiver.
           ## The TarArchiver instatiates in its constructor at [TarArchiver.java#L53|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/util/io/TarArchiver.java#L53] a TarArchiveOutputStream with constructor [org.apache.commons.compress.archivers.tar.TarArchiveOutputStream#TarArchiveOutputStream(final OutputStream os)|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L146]. Using this constructor results at [TarArchiveOutputStream.java#L221|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L221] in initializing member zipEncoding from call to ZipEncodingHelper.getZipEncoding(null).
           ## [ZipEncodingHelper#getZipEncoding(null)|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/zip/ZipEncodingHelper.java#L51C42-L51C42] returns effectively the Charset.defaultCharset() being wrapped into a NioZipEncoding object which is stored in TarArchiveOutputStream#zipEncoding.
           ## [TarArchiveOutputStream.java#zipEncoding|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L130] is used then to encode names written into the tar archive at [TarArchiveOutputStream.java#handleLongName()|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L512], [TarArchiveOutputStream#putArchiveEntry()|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L575], and [TarArchiveOutputStream.java#L614|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L614]
           * So per above file names are written to the tar archive with agents default character set which is CP1047 in our case.
           * At controller side which reads the tar stream from [FilePath.java#L2866|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2866] calling [FilePath#readFromTar()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3076] which instantiates [TarArchiveInputStream|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveInputStream.java#L50]. That has a zipEncoding member, too. It is initialized at [TarArchiveInputStream.java#L211|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveInputStream.java#L211] the same way as TarArchiveOutputStream's zipEncoding member. However, since this is executed at the controller it receives whatever the default character set is there.
           * Due to the mismatch of character sets used for writing and reading file names in tar stream the reading fails.

           

          I suspect that JENKINS-18558 is related to this issue.
          New: I'm experiencing this problem with pipeline-maven-plugin when it is running 'Generated Artifacts Publisher' when running maven on a z/OS agent (line numbers are as of Jenkins version 2.426.2):
          {code:java}
          ERROR: [withMaven] WARNING Exception executing Maven reporter 'Generated Artifacts Publisher' / org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher. Please report a bug associated for the component 'pipeline-maven-plugin' at https://issues.jenkins-ci.org
           java.io.IOException: Exception archiving { map content removed }
            at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:145)
            at org.jenkinsci.plugins.pipeline.maven.MavenSpyLogProcessor.processMavenSpyLogs(MavenSpyLogProcessor.java:172)
            at org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution2$WithMavenStepExecutionCallBack.finished(WithMavenStepExecution2.java:1359)
            at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution$TailCall.lambda$onSuccess$0(GeneralNonBlockingStepExecution.java:140)
            at org.jenkinsci.plugins.workflow.steps.GeneralNonBlockingStepExecution.lambda$run$0(GeneralNonBlockingStepExecution.java:77)
            at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
            at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
            at java.base/java.lang.Thread.run(Thread.java:840)
           Caused by: java.io.IOException: Failed to extract /path removed/transfer of 8 files
            at hudson.FilePath.readFromTar(FilePath.java:3068)
            at hudson.FilePath.copyRecursiveTo(FilePath.java:2828)
            at jenkins.model.StandardArtifactManager.archive(StandardArtifactManager.java:73)
            at org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher.process(GeneratedArtifactsPublisher.java:143)
            ... 9 more
           Caused by: java.nio.charset.UnmappableCharacterException: Input length = 1
            at java.base/java.nio.charset.CoderResult.throwException(CoderResult.java:275)
            at java.base/java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:820)
            at org.apache.commons.compress.archivers.zip.NioZipEncoding.decode(NioZipEncoding.java:130)
            at org.apache.commons.compress.archivers.tar.TarUtils.parseName(TarUtils.java:524)
            at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeaderUnwrapped(TarArchiveEntry.java:1543)
            at org.apache.commons.compress.archivers.tar.TarArchiveEntry.parseTarHeader(TarArchiveEntry.java:1531)
            at org.apache.commons.compress.archivers.tar.TarArchiveEntry.<init>(TarArchiveEntry.java:553)
            at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.getNextTarEntry(TarArchiveInputStream.java:464)
            at hudson.FilePath.readFromTar(FilePath.java:3043)
            ... 12 more
          {code}
           

          Code review turned out that this is most likely a problem in [hudson.FilePath#copyRecursiveTo |https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2842] implementation (line numbers are as of Jenkins version 2.440):
           * Assumption is that the remote part is the agent with a default character set of CP1047 (EBCDIC) and local is a Windows or Linux with something ASCII/UTF-8 based.
           * At [FilePath.java#L2864|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2864] CopyRecursiveRemoteToLocal is instantiated and effectively [CopyRecursiveRemoteToLocal#invoke|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3030%20at%20FilePath.java#L3030] is executed at the agent:
           ## Calling [FilePath#writeToTar()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3064] which effectively calls [hudson.util.io.ArchiverFactory.TarArchiverFactory#create()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/util/io/ArchiverFactory.java#L88] which instantiates and returns a TarArchiver.
           ## The TarArchiver instatiates in its constructor at [TarArchiver.java#L53|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/util/io/TarArchiver.java#L53] a TarArchiveOutputStream with constructor [org.apache.commons.compress.archivers.tar.TarArchiveOutputStream#TarArchiveOutputStream(final OutputStream os)|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L146]. Using this constructor results at [TarArchiveOutputStream.java#L221|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L221] in initializing member zipEncoding from call to ZipEncodingHelper.getZipEncoding(null).
           ## [ZipEncodingHelper#getZipEncoding(null)|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/zip/ZipEncodingHelper.java#L51C42-L51C42] returns effectively the Charset.defaultCharset() being wrapped into a NioZipEncoding object which is stored in TarArchiveOutputStream#zipEncoding.
           ## [TarArchiveOutputStream.java#zipEncoding|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L130] is used then to encode names written into the tar archive at [TarArchiveOutputStream.java#handleLongName()|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L512], [TarArchiveOutputStream#putArchiveEntry()|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L575], and [TarArchiveOutputStream.java#L614|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveOutputStream.java#L614]
           * So per above file names are written to the tar archive with agents default character set which is CP1047 in our case.
           * At controller side which reads the tar stream from [FilePath.java#L2866|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L2866] calling [FilePath#readFromTar()|https://github.com/jenkinsci/jenkins/blob/f1e9bbc410604bd2f82c65df3d92d02826312a84/core/src/main/java/hudson/FilePath.java#L3076] which instantiates [TarArchiveInputStream|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveInputStream.java#L50]. That has a zipEncoding member, too. It is initialized at [TarArchiveInputStream.java#L211|https://github.com/apache/commons-compress/blob/38dec7fb4690c023ef0af9661e0e1ae4f254a769/src/main/java/org/apache/commons/compress/archivers/tar/TarArchiveInputStream.java#L211] the same way as TarArchiveOutputStream's zipEncoding member. However, since this is executed at the controller it receives whatever the default character set is there.
           * Due to the mismatch of character sets used for writing and reading file names in tar stream the reading fails.

           

          I suspect that JENKINS-18558 is related to this issue.
          Lutz Neugebauer made changes -
          Link New: This issue is related to JENKINS-18558 [ JENKINS-18558 ]
          Alexander Brandes made changes -
          Remote Link New: This issue links to "https://github.com/jenkinsci/jenkins/pull/8860 (Web Link)" [ 29506 ]
          Alexander Brandes made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Alexander Brandes made changes -
          Status Original: In Progress [ 3 ] New: In Review [ 10005 ]
          Mark Waite made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Review [ 10005 ] New: Fixed but Unreleased [ 10203 ]
          Mark Waite made changes -
          Released As New: https://www.jenkins.io/changelog/2.449/
          Status Original: Fixed but Unreleased [ 10203 ] New: Resolved [ 5 ]
          Alexander Brandes made changes -
          Released As Original: https://www.jenkins.io/changelog/2.449/ New: 2.449

            Unassigned Unassigned
            lne Lutz Neugebauer
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: