Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22434

jgit clean after checkout fails with unmappable chars in path

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Major Major
    • git-client-plugin
    • None
    • Ubuntu 12.04 LTS amd64, using jdk7 from the distribution, jenkins v1.556

      Started by user anonymous
      Building in workspace /var/lib/jenkins/jobs/ext-jgit-3.3.0/workspace
      Cloning the remote Git repository
      Cloning repository https://github.com/eclipse/jgit.git
      ERROR: Failed to clean the workspace
      java.io.IOException: java.lang.reflect.InvocationTargetException
      at hudson.Util.isSymlinkJava7(Util.java:360)
      at hudson.Util.isSymlink(Util.java:325)
      at hudson.Util.deleteRecursive(Util.java:291)
      at hudson.Util.deleteContentsRecursive(Util.java:203)
      at hudson.Util.deleteRecursive(Util.java:292)
      at hudson.Util.deleteContentsRecursive(Util.java:203)
      at hudson.Util.deleteRecursive(Util.java:292)
      at hudson.Util.deleteContentsRecursive(Util.java:203)
      at hudson.Util.deleteRecursive(Util.java:292)
      at hudson.Util.deleteContentsRecursive(Util.java:203)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:327)
      at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:845)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:878)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1320)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
      at hudson.model.Run.execute(Run.java:1688)
      at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:519)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:231)
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at hudson.Util.isSymlinkJava7(Util.java:355)
      ... 20 more
      Caused by: java.nio.file.InvalidPathException: Malformed input or input contains unmappable chacraters: /var/lib/jenkins/jobs/ext-jgit-3.3.0/workspace/org.eclipse.jgit.java7.test/target/tmp_7536346264520663349/??
      at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
      at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
      at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
      at java.io.File.toPath(File.java:2186)
      ... 24 more
      ERROR: Error cloning remote repo 'origin'
      hudson.plugins.git.GitException: Failed to delete workspace
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:330)
      at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:845)
      at hudson.plugins.git.GitSCM.checkout(GitSCM.java:878)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1320)
      at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:609)
      at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:518)
      at hudson.model.Run.execute(Run.java:1688)
      at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:519)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:231)
      Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
      at hudson.Util.isSymlinkJava7(Util.java:360)
      at hudson.Util.isSymlink(Util.java:325)
      at hudson.Util.deleteRecursive(Util.java:291)
      at hudson.Util.deleteContentsRecursive(Util.java:203)
      at hudson.Util.deleteRecursive(Util.java:292)
      at hudson.Util.deleteContentsRecursive(Util.java:203)
      at hudson.Util.deleteRecursive(Util.java:292)
      at hudson.Util.deleteContentsRecursive(Util.java:203)
      at hudson.Util.deleteRecursive(Util.java:292)
      at hudson.Util.deleteContentsRecursive(Util.java:203)
      at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:327)
      ... 10 more
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:606)
      at hudson.Util.isSymlinkJava7(Util.java:355)
      ... 20 more
      Caused by: java.nio.file.InvalidPathException: Malformed input or input contains unmappable chacraters: /var/lib/jenkins/jobs/ext-jgit-3.3.0/workspace/org.eclipse.jgit.java7.test/target/tmp_7536346264520663349/??
      at sun.nio.fs.UnixPath.encode(UnixPath.java:147)
      at sun.nio.fs.UnixPath.<init>(UnixPath.java:71)
      at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:281)
      at java.io.File.toPath(File.java:2186)
      ... 24 more
      ERROR: null
      Retrying after 10 seconds

        1. org.eclipse.jgit.java7.test.tar
          280 kB

          [JENKINS-22434] jgit clean after checkout fails with unmappable chars in path

          Mr Cinquero created issue -

          Mark Waite added a comment -

          Can you isolate those unmappable characters? Are they the same as the NFD characters described in JENKINS-20410?

          Does the same failure occur for command line git, or only for JGit?

          Mark Waite added a comment - Can you isolate those unmappable characters? Are they the same as the NFD characters described in JENKINS-20410 ? Does the same failure occur for command line git, or only for JGit?
          Mark Waite made changes -
          Link New: This issue is related to JENKINS-20410 [ JENKINS-20410 ]

          Mr Cinquero added a comment -

          There is no .git so I don't know. My guess is that it is strictly java.io.File related. I had such issues before where Java strictly refuses to process files with filenames not encoded according to locale settings. Obviously this is one such case: I always let Jenkins run with LC_ALL=C and the umlaut 'ä' in this case is obviously encoded as UTF-8. See the attached tar file.

          Mr Cinquero added a comment - There is no .git so I don't know. My guess is that it is strictly java.io.File related. I had such issues before where Java strictly refuses to process files with filenames not encoded according to locale settings. Obviously this is one such case: I always let Jenkins run with LC_ALL=C and the umlaut 'ä' in this case is obviously encoded as UTF-8. See the attached tar file.
          Mr Cinquero made changes -
          Attachment New: org.eclipse.jgit.java7.test.tar [ 25649 ]

          Mr Cinquero added a comment - - edited

          Yeah, I just checked it again, and it seems java.io.File always operates on the basis of a java.lang.String, ie. with an internal representation-independent form and it always uses the environment settings (or file.encoding sys prop) when choosing the actual filename repesentation when interacting with the filesystem layer. That is nuts because it seemingly cannot handle files with bad name encodings, ie. one could think of falling back to a generic charset like iso8859-1 to handle such cases, at least when the filename's correct encoding is not an issue.....

          Mr Cinquero added a comment - - edited Yeah, I just checked it again, and it seems java.io.File always operates on the basis of a java.lang.String, ie. with an internal representation-independent form and it always uses the environment settings (or file.encoding sys prop) when choosing the actual filename repesentation when interacting with the filesystem layer. That is nuts because it seemingly cannot handle files with bad name encodings, ie. one could think of falling back to a generic charset like iso8859-1 to handle such cases, at least when the filename's correct encoding is not an issue.....

          Mr Cinquero added a comment -

          I have created a test case to verify the issue that is possibly at hand here:

          https://github.com/jjYBdx4IL/filenameenc

          There is also a topic at stackoverflow, but the people there don't get it yet, hopefully they do with the example at github.

          Mr Cinquero added a comment - I have created a test case to verify the issue that is possibly at hand here: https://github.com/jjYBdx4IL/filenameenc There is also a topic at stackoverflow, but the people there don't get it yet, hopefully they do with the example at github.

          Mark Waite added a comment -

          Doesn't your test case show that the problem is in the JDK, rather than in Jenkins or in JGit or the git-client-plugin?

          Mark Waite added a comment - Doesn't your test case show that the problem is in the JDK, rather than in Jenkins or in JGit or the git-client-plugin?

          Mr Cinquero added a comment - - edited

          Maybe. Or it may just be an indication that we need a workaround because otherwise the problem won't go away, possibly never because the JLS guys may decide it to be a feature

          I propose to use some platform-specific means for removing such badly encoded files, ie. run "rm -f $strange-filename" on Linux, and the equivalent on Windows. I guess that there are probably already libraries around doing exactly that, though I'm not aware of any.

          Another solution would be to let Jenkins always run with a full 8-bit charset as the default charset, ie. one that has no invalid characters/holes in the binary representation per se, though I'm not sure what undesirable side-effects that would bring about. Thinking about it further, I guess that this was no problem in the old days before Unicode, and it is rather a side-effect now that had to be introduced through the introducation of Unicode...

          Mr Cinquero added a comment - - edited Maybe. Or it may just be an indication that we need a workaround because otherwise the problem won't go away, possibly never because the JLS guys may decide it to be a feature I propose to use some platform-specific means for removing such badly encoded files, ie. run "rm -f $strange-filename" on Linux, and the equivalent on Windows. I guess that there are probably already libraries around doing exactly that, though I'm not aware of any. Another solution would be to let Jenkins always run with a full 8-bit charset as the default charset, ie. one that has no invalid characters/holes in the binary representation per se, though I'm not sure what undesirable side-effects that would bring about. Thinking about it further, I guess that this was no problem in the old days before Unicode, and it is rather a side-effect now that had to be introduced through the introducation of Unicode...

          Mr Cinquero added a comment - - edited

          OK, I found a possible solution: is the old java.io.* API still being used? If so, upgrade to java.nio.*, especially when reading filenames from disk the java.io.File.listFiles() method replaces invalid chars with the default unknown char character of the active charset, whereas java.nio.Files.newDirectoryStream() does not do such a thing (at least apparently, see https://github.com/jjYBdx4IL/filenameenc/blob/master/src/main/java/filenameenc/Test.java).

          Mr Cinquero added a comment - - edited OK, I found a possible solution: is the old java.io.* API still being used? If so, upgrade to java.nio.*, especially when reading filenames from disk the java.io.File.listFiles() method replaces invalid chars with the default unknown char character of the active charset, whereas java.nio.Files.newDirectoryStream() does not do such a thing (at least apparently, see https://github.com/jjYBdx4IL/filenameenc/blob/master/src/main/java/filenameenc/Test.java ).

            ndeloof Nicolas De Loof
            marc321 Mr Cinquero
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: