Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-18414

Jenkins Subversion plugin checkouts in something NON-UTF-8

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • subversion-plugin
    • None

      Subversion checkouts of a name with some UTF-8 chars are done incorrectly from the jenkins console ONLY.

      For instance "Descripción.txt" gets downloaded as "Descripci?n.txt"

      If you do them with svn on the command line everything is fine, you get the proper filename.
      If you do them with jsvn (svnkit cli ofrom Ubunut) everything is fine, you get the proper filename.

      Workaround:
      1) You have to write a bash script to be executed just after the subversion checkout BUT before the maven build.

      2) This script must remove all badly checked out files like "Descripci?n.txt"

      3) Finally you do an "svn update" within the checked out project directory, so the files are checked out correctly now "Descripción.txt".

      4) Everything else from here works as expected.

          [JENKINS-18414] Jenkins Subversion plugin checkouts in something NON-UTF-8

          Jose L Vazquez created issue -

          kutzi added a comment -

          Can you provide a minimal subversion repository to reproduce the problem?

          kutzi added a comment - Can you provide a minimal subversion repository to reproduce the problem?

          The problem is that this is configuration issue (yesterday I hadn't time to explain it all)

          I have another Jenkins installation (that I need to switch off) on a Ubuntu Linux 10.04 working against the same repository and it works fine with the same repo.

          The problem is that, for some reason, the Subversion plugin within Jenkins, is working on an incorrect encoding, contrary to the OS environment and Jenkins settings. I have managed to pass proper values ('UTF-8') to "file.encoding" and "sun.jnu.encoding" at Jenkins command line and it now displays accented words properly (like my name "José Luis Vázquez González") BUT the subversion plugin does NOT use this config and checkouts the repository in something non UTF-8 (Latin1 maybe?)

          It is NOT a Subversion plugin version problem either. I tested the same version that was working on the other server and it failed as well.

          I am sure it is a subversion plugin configuration or environment problem, and JUST in that plugin. (Remember, the svn and jsvn command line tools executed as the jenkins user will work just fine, checking out "descripción.txt" as expected)

          How can I FORCE the subversion plugin to use UTF-8 instead of whatever it is using now?

          Jose L Vazquez added a comment - The problem is that this is configuration issue (yesterday I hadn't time to explain it all) I have another Jenkins installation (that I need to switch off) on a Ubuntu Linux 10.04 working against the same repository and it works fine with the same repo. The problem is that, for some reason, the Subversion plugin within Jenkins, is working on an incorrect encoding, contrary to the OS environment and Jenkins settings. I have managed to pass proper values ('UTF-8') to "file.encoding" and "sun.jnu.encoding" at Jenkins command line and it now displays accented words properly (like my name "José Luis Vázquez González") BUT the subversion plugin does NOT use this config and checkouts the repository in something non UTF-8 (Latin1 maybe?) It is NOT a Subversion plugin version problem either. I tested the same version that was working on the other server and it failed as well. I am sure it is a subversion plugin configuration or environment problem, and JUST in that plugin. (Remember, the svn and jsvn command line tools executed as the jenkins user will work just fine, checking out "descripción.txt" as expected) How can I FORCE the subversion plugin to use UTF-8 instead of whatever it is using now?

          kutzi added a comment -

          AFAIK the subversion-plugin should use the system encoding.
          Have you checked that that the node (slave or master) which actually checks out the code has really UTF8 set?

          In any case this sounds more like a configuration problem on your side than a bug in the plugin. So I advise you to ask for help on the users mailing list.

          kutzi added a comment - AFAIK the subversion-plugin should use the system encoding. Have you checked that that the node (slave or master) which actually checks out the code has really UTF8 set? In any case this sounds more like a configuration problem on your side than a bug in the plugin. So I advise you to ask for help on the users mailing list.

          Yes, it is a config problem, but I still don't get it...

          What do you mean "that node (slave or master) which actually checks out the code"?

          1) The code is checked out locally, on the same server Jenkins is installed.

          2) I have logged as "jenkins" into the workspace directory that it is working on. And I know it is correct cause:

          • It tells me the local path ("Building in workspace /var/lib/jenkins/workspace/modcaja-a-test")
          • If I remove the workspace files from command line at that location it disappears from jenkins.
          • I can see the incorrect checkouts from jenkins and...
          • Jenkins can see proper checkouts if I do them on the command line instead.

          3) If I go to the workspace directory as jenkins I should have the same environment as the jenkins webapp, right?
          If there I do locale I get this:

          $ locale
          LANG=es_ES.UTF-8
          LANGUAGE=en_US.UTF-8
          LC_CTYPE="en_US.UTF-8"
          LC_NUMERIC="en_US.UTF-8"
          LC_TIME="en_US.UTF-8"
          LC_COLLATE="en_US.UTF-8"
          LC_MONETARY="en_US.UTF-8"
          LC_MESSAGES="en_US.UTF-8"
          LC_PAPER="en_US.UTF-8"
          LC_NAME="en_US.UTF-8"
          LC_ADDRESS="en_US.UTF-8"
          LC_TELEPHONE="en_US.UTF-8"
          LC_MEASUREMENT="en_US.UTF-8"
          LC_IDENTIFICATION="en_US.UTF-8"
          LC_ALL=en_US.UTF-8

          Everything is UTF-8, yes LANG is different from the rest, but the point is that ALL is UTF-8!
          right?

          4) If I do a svn co ... from there in the command line it is done correctly. So the environment seems ok right?

          5) EVEN if I do jsvn co ... from there it is done correctly. And jsvn is the command line of svnkit, which is what the jenkins plugin is using. right?

          6) SO how come that, with jenkins having file.encoding=UTF-8 and sun.jnu.encoding=UTF-8, the subversion plugin DOES NOT use UTF-8?
          What environment is it seen? (the jenkins linux user sees a different utf-8 one for sure)
          Can I override it?

          Jose L Vazquez added a comment - Yes, it is a config problem, but I still don't get it... What do you mean "that node (slave or master) which actually checks out the code"? 1) The code is checked out locally, on the same server Jenkins is installed. 2) I have logged as "jenkins" into the workspace directory that it is working on. And I know it is correct cause: It tells me the local path ("Building in workspace /var/lib/jenkins/workspace/modcaja-a-test") If I remove the workspace files from command line at that location it disappears from jenkins. I can see the incorrect checkouts from jenkins and... Jenkins can see proper checkouts if I do them on the command line instead. 3) If I go to the workspace directory as jenkins I should have the same environment as the jenkins webapp, right? If there I do locale I get this: $ locale LANG=es_ES.UTF-8 LANGUAGE=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=en_US.UTF-8 Everything is UTF-8, yes LANG is different from the rest, but the point is that ALL is UTF-8! right? 4) If I do a svn co ... from there in the command line it is done correctly. So the environment seems ok right? 5) EVEN if I do jsvn co ... from there it is done correctly. And jsvn is the command line of svnkit, which is what the jenkins plugin is using. right? 6) SO how come that, with jenkins having file.encoding=UTF-8 and sun.jnu.encoding=UTF-8, the subversion plugin DOES NOT use UTF-8? What environment is it seen? (the jenkins linux user sees a different utf-8 one for sure) Can I override it?

          kutzi added a comment -

          Maybe related to http://issues.tmatesoft.com/issue/SVNKIT-301

          Which jsvn version have you used to try it on the commandline?

          kutzi added a comment - Maybe related to http://issues.tmatesoft.com/issue/SVNKIT-301 Which jsvn version have you used to try it on the commandline?

          Thanks,

          It is the one that comes with Linux 12.04:

          $ dpkg -l |grep svnkit
          ii libsvnkit-java 1.3.5+dfsg-1 pure Java Subversion client library
          ii svnkit 1.3.5+dfsg-1 pure Java Subversion client

          1.3.5 according to the command line.

          Jose L Vazquez added a comment - Thanks, It is the one that comes with Linux 12.04: $ dpkg -l |grep svnkit ii libsvnkit-java 1.3.5+dfsg-1 pure Java Subversion client library ii svnkit 1.3.5+dfsg-1 pure Java Subversion client 1.3.5 according to the command line.

          kutzi added a comment -

          Can yout try it with 1.7.5?

          kutzi added a comment - Can yout try it with 1.7.5?

          Yes, I downloaded it with wget like this:

          wget http://www.svnkit.com/org.tmatesoft.svn_1.7.5.standalone.zip
          unzip org.tmatesoft.svn_1.7.5.standalone.zip

          Then I tested it on a workspace tmp directory as the jenkins user in the command line like this:

          /home/admin/svnkit-1.7.5/bin/jsvn co svn+ssh://svn.rflabs.com/data/subversion/modcaja/trunk/modcaja

          And it checked out the file correctly:
          $ ls modcaja/especs/ -l
          total 9968
          ...
          rw-rw-r- 1 jenkins jenkins 2577064 Jun 21 15:34 WG S2S - Descripción Protocolo 3GS.PDF

          As expected.

          But withint Jenkins the plugin is unable!

          Jose L Vazquez added a comment - Yes, I downloaded it with wget like this: wget http://www.svnkit.com/org.tmatesoft.svn_1.7.5.standalone.zip unzip org.tmatesoft.svn_1.7.5.standalone.zip Then I tested it on a workspace tmp directory as the jenkins user in the command line like this: /home/admin/svnkit-1.7.5/bin/jsvn co svn+ssh://svn.rflabs.com/data/subversion/modcaja/trunk/modcaja And it checked out the file correctly: $ ls modcaja/especs/ -l total 9968 ... rw-rw-r - 1 jenkins jenkins 2577064 Jun 21 15:34 WG S2S - Descripción Protocolo 3GS.PDF As expected. But withint Jenkins the plugin is unable!

          Tobias Peters added a comment -

          I had the same problem and the issue was with environment variables
          the slave was running on a linux machine, started via ssh from master but the jvm process on the slave didn't get the environment parameter set up for the slave
          you can check by doing

          sudo cat /proc/[slave_pid]/environ

          so I added 'Prefix Start Slave Command' of

          export LANG=en_US.UTF-8;

          restarted slave, and checked to see if it was set in /proc/?/environ

          after that it did do the svn co correctly

          Tobias Peters added a comment - I had the same problem and the issue was with environment variables the slave was running on a linux machine, started via ssh from master but the jvm process on the slave didn't get the environment parameter set up for the slave you can check by doing sudo cat /proc/ [slave_pid] /environ so I added 'Prefix Start Slave Command' of export LANG=en_US.UTF-8; restarted slave, and checked to see if it was set in /proc/?/environ after that it did do the svn co correctly

            Unassigned Unassigned
            josvazg Jose L Vazquez
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: