Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62987

Wrong information about UTF-8 url decoding

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Component/s: core
    • Labels:
      None
    • Environment:
      WebSphere Liberty Profile or OpenLiberty, any version
    • Similar Issues:

      Description

      Problem description

      Jenkins displays this banner in the settings:

      Ihr Servlet-Container verwendet kein UTF-8, um URLs zu dekodieren. Falls Sie Nicht-ASCII-Zeichen in Elementnamen usw. verwenden, kann dies Probleme mit sich bringen. Beachten Sie bitte die Hinweise zu Servlet-Containern bzw. Tomcat i18N).

      However, this is not true. Liberty Profile uses UTF-8 by default for decoding URIs.

      Sources:

       

      How to reproduce

      1. use ./bin/server create jenkins to create a new serer.
      2. Download the jenkins .war file to the apps directory of the server ($WLP_INSTALL_DIR/usr/servers/jenkins/apps).
      3. Modify the server.xml to contain these config items:
      <featureManager>
       <feature>servlet-4.0</feature>
       <feature>websocket-1.1</feature>
       <feature>localConnector-1.0</feature>
       <feature>jaxb-2.2</feature>
       <feature>jaxws-2.2</feature>
      </featureManager>
      
      <httpEndpoint id="defaultHttpEndpoint" host="*" httpPort="8080" httpsPort="-1" />
      <application id="jenkins" location="jenkins.war" name="jenkins" type="war" context-root="/" />
      
      1. Modify the file server.env to contain a JENKINS_HOME
      2. Start Liberty Profile (./bin/server start jenkins)

      Go to the settings and see the misleading banner

      Expected behaviour

      Jenkins does not display this banner, as OpenLiberty uses UTF-8 to decode URLs.

      Actual behaviour

      Jenkins does display this banner, which is misleading.

      Suggested fix

      Rework the URL check to display a proper message and how it got this information.

       

      I guess the issue might be related to my reverse proxy, which is set up in front of liberty profile. But as the message does not display why jenkins thinks the container is set up in a wrong way, I cannot confirm this behaviour.

       

       

        Attachments

          Activity

          Hide
          danielbeck Daniel Beck added a comment -

          Jenkins has the browser send a request expected to contain nontrivial Unicode chars in the query string. If that doesn't arrive correctly, this message is shown.

          I would recommend you create a view in Jenkins whose name contains nontrivial Unicode characters and see whether that behaves correctly.

          Show
          danielbeck Daniel Beck added a comment - Jenkins has the browser send a request expected to contain nontrivial Unicode chars in the query string. If that doesn't arrive correctly, this message is shown. I would recommend you create a view in Jenkins whose name contains nontrivial Unicode characters and see whether that behaves correctly.
          Hide
          danielbeck Daniel Beck added a comment -

          Nothing here indicates there's a bug in Jenkins. Perhaps the message can be improved to mention reverse proxies, load balancers, firewalls, and whatever else could mess with requests, but I'm not sure how that would be helpful.

          Show
          danielbeck Daniel Beck added a comment - Nothing here indicates there's a bug in Jenkins. Perhaps the message can be improved to mention reverse proxies, load balancers, firewalls, and whatever else could mess with requests, but I'm not sure how that would be helpful.
          Hide
          bmarwell Ben M added a comment - - edited

          > I would recommend you create a view in Jenkins whose name contains nontrivial Unicode characters and see whether that behaves correctly.

          > Closed: Could not reproduce
           
          So you didn't even try? You mean "too lazy to reproduce"? Wow.
           
          I tried again without a proxy and it is 100% reproducible.
           
          In chrome (latest), I see this URL:
          Request URL:
          http://192.168.53.34:8080/administrativeMonitor/jenkins.diagnostics.URICheckEncodingMonitor/checkURIEncoding?value=%E5%9F%B7%E4%BA%8B
          Response: 

          // Edit

          The message was cut out here because of Jira filtering things

           

          I created a project with a treble clef.

          While the creation page said "invalid character", clicking OK created the project successfully.

          So this must be a jenkins issue, since the URL is displayed correctly.

           

           

          The next time  you need more information, please do not close it as "could not reproduce", when you did not even try! I am willing to give as much hints as possible and help and to support you with solving this issue.

          However, just reading the text, not trying to reproduce it (not even bothering) and close as "could not reproduce" is not helpful at all.

          Setting up a liberty server with the instructions I wrote in this issue took me about 5 minutes.

           


           

          Show
          bmarwell Ben M added a comment - - edited > I would recommend you create a view in Jenkins whose name contains nontrivial Unicode characters and see whether that behaves correctly. … > Closed: Could not reproduce   So you didn't even try? You mean "too lazy to reproduce"? Wow.   I tried again without a proxy and it is 100% reproducible.   In chrome (latest), I see this URL: Request URL: http://192.168.53.34:8080/administrativeMonitor/jenkins.diagnostics.URICheckEncodingMonitor/checkURIEncoding?value=%E5%9F%B7%E4%BA%8B Response:  // Edit The message was cut out here because of Jira filtering things   I created a project with a treble clef. While the creation page said "invalid character", clicking OK created the project successfully. So this must be a jenkins issue, since the URL is displayed correctly.   –   The next time  you need more information, please do not close it as "could not reproduce", when you did not even try! I am willing to give as much hints as possible and help and to support you with solving this issue. However, just reading the text, not trying to reproduce it (not even bothering) and close as "could not reproduce" is not helpful at all. Setting up a liberty server with the instructions I wrote in this issue took me about 5 minutes.    
          Hide
          bmarwell Ben M added a comment -

          By the way, the project is browsed though the URL "http://192.168.53.34:8080/job/%C2%B5%F0%9D%84%9E/" in unicode. So this is a proof there is nothing wrong with liberty.

          Show
          bmarwell Ben M added a comment - By the way, the project is browsed though the URL " http://192.168.53.34:8080/job/%C2%B5%F0%9D%84%9E/ " in unicode. So this is a proof there is nothing wrong with liberty.
          Hide
          danielbeck Daniel Beck added a comment -

           So you didn't even try? You mean "too lazy to reproduce"? Wow.

          There is no need to insult others.

          In this case, given the fact that I don't see the warning on several Jenkins instances operated by the Jenkins project, and run it locally every day during development, indicates that it doesn't spuriously appear in some configurations, indicating this is a problem with yours.

          Show
          danielbeck Daniel Beck added a comment -  So you didn't even try? You mean "too lazy to reproduce"? Wow. There is no need to insult others. In this case, given the fact that I don't see the warning on several Jenkins instances operated by the Jenkins project, and run it locally every day during development, indicates that it doesn't spuriously appear in some configurations, indicating this is a problem with yours.
          Hide
          danielbeck Daniel Beck added a comment -

          While the creation page said "invalid character", clicking OK created the project successfully.

          This is probably the difference between values submitted in the URL (form validation), perhaps even only the query string, and values submitted in request bodies (project creation). For me, the "clef" project name produces no validation error. So my recommended steps are probably meaningless.

          OTOH, we still don't know what's wrong. For better diagnoseability I submitted https://github.com/jenkinsci/jenkins/pull/4871

          Show
          danielbeck Daniel Beck added a comment - While the creation page said "invalid character", clicking OK created the project successfully. This is probably the difference between values submitted in the URL (form validation), perhaps even only the query string, and values submitted in request bodies (project creation). For me, the "clef" project name produces no validation error. So my recommended steps are probably meaningless. OTOH, we still don't know what's wrong. For better diagnoseability I submitted https://github.com/jenkinsci/jenkins/pull/4871
          Hide
          bmarwell Ben M added a comment -

          > There is no need to insult others.

          Sorry! It was not meant as an insult, but I just was not able to see how you concluded that this was not a bug in jenkins. I failed to phrase it properly, and I am very sorry this happened. But true, I was a little bit disappointed in closing this ticket early.


          > In this case, given the fact that I don't see the warning on several Jenkins instances operated by the Jenkins project, and run it locally every day during development, indicates that it doesn't spuriously appear in some configurations, indicating this is a problem with yours.

          I should point out (which I forgot to mention when parts of my comment were swallowed by JIRA): 

          • This time there is not proxy involved! OpenLiberty does not act as a proxy, it acts as a container, just like tomcat. Thanks Mark Waite for pointing out I forgot to explicitly mention this.
          • If you need a test container, I'd be happy to provide a Dockerfile or similar via Github.

          Thanks for looking into it and thanks for the PR!  

          Show
          bmarwell Ben M added a comment - > There is no need to insult others. Sorry! It was not meant as an insult, but I just was not able to see how you concluded that this was not a bug in jenkins. I failed to phrase it properly, and I am very sorry this happened. But true, I was a little bit disappointed in closing this ticket early. > In this case, given the fact that I don't see the warning on several Jenkins instances operated by the Jenkins project, and run it locally every day during development, indicates that it doesn't spuriously appear in some configurations, indicating this is a problem with yours. I should point out (which I forgot to mention when parts of my comment were swallowed by JIRA):  This time there is not proxy involved! OpenLiberty does not act as a proxy, it acts as a container, just like tomcat. Thanks Mark Waite for pointing out I forgot to explicitly mention this. If you need a test container, I'd be happy to provide a Dockerfile or similar via Github. Thanks for looking into it and thanks for the PR!  
          Hide
          bmarwell Ben M added a comment -

          Daniel Beck thanks for https://github.com/jenkinsci/jenkins/pull/4871#event-3617187919

          Debug output

          The log for OpenLiberty yields:

          Aug 17, 2020 1:46:05 PM CONFIG jenkins.diagnostics.URICheckEncodingMonitorExpected to receive: 執事 (e59fb7e4ba8b) but got: 執事 (c3a5c29fc2b7c3a4c2bac28b)

           
          But I found this IBM support thread:
          https://www.ibm.com/mysupport/s/question/0D50z00005pgf5xCAA/how-to-set-url-encoding-to-utf8?language=en_US
           
          It seems that OpenLiberty is implementing everything correctly, as jenkins should call request.setCharacterEncoding first. But then, the other solution (specifying -Dclient.encoding.override=UTF-8) also works.
           
          But then there is another Stack Overflow thread I stumbled upon: https://stackoverflow.com/a/52393944/1549977
          So I tried it by adding -H 'Content-Type: text/plain;charset=UTF-8' to my curl request. Although there was no content (not a POST request), it did in fact work: no warning in the curl response!  
           

          Suggested solution

          1. Add Content-Type: text/plain;charset=UTF-8 to the request.
            According to the HTTP specification, the default is ISO-8859-1.
          2. Detect if running on LibertyProfile or OpenLiberty. If so, suggest the user to add the following parameters to the jvm.options file:
            {{-Dfile.encoding=UTF-8 }}
            -Dclient.encoding.override=UTF-8
          3. Or maybe try with the request.setCharacterEncoding() option?

          I'd vote for the first solution. It is easy to do and backwards-compatible and will also possibly be a remedy for any future application servers.

          Solution 2 is somewhat misleading and seems to be a workaround imho, because liberty DOES encode URLs property, as can be seen in my screenshots.

          Solution 3 is worth a shot. It looks like it is the most "correct" solution, but I haven't checked it any further.

          Show
          bmarwell Ben M added a comment - Daniel Beck thanks for  https://github.com/jenkinsci/jenkins/pull/4871#event-3617187919 Debug output The log for OpenLiberty yields: Aug 17, 2020 1:46:05 PM CONFIG jenkins.diagnostics.URICheckEncodingMonitorExpected to receive: 執事 (e59fb7e4ba8b) but got: 執事 (c3a5c29fc2b7c3a4c2bac28b)   But I found this IBM support thread: https://www.ibm.com/mysupport/s/question/0D50z00005pgf5xCAA/how-to-set-url-encoding-to-utf8?language=en_US   It seems that OpenLiberty is implementing everything correctly, as jenkins should call  request.setCharacterEncoding first. But then, the other solution (specifying  -Dclient.encoding.override=UTF-8 ) also works.   But then there is another Stack Overflow thread I stumbled upon: https://stackoverflow.com/a/52393944/1549977 So I tried it by adding  -H 'Content-Type: text/plain;charset=UTF-8' to my curl request. Although there was no content (not a POST request), it did in fact work: no warning in the curl response!     Suggested solution Add  Content-Type: text/plain;charset=UTF-8 to the request. According to the HTTP specification, the default is ISO-8859-1. Detect if running on LibertyProfile or OpenLiberty. If so, suggest the user to add the following parameters to the jvm.options file: {{-Dfile.encoding=UTF-8 }} -Dclient.encoding.override=UTF-8 Or maybe try with the request.setCharacterEncoding() option? I'd vote for the first solution. It is easy to do and backwards-compatible and will also possibly be a remedy for any future application servers. Solution 2 is somewhat misleading and seems to be a workaround imho, because liberty DOES encode URLs property, as can be seen in my screenshots. Solution 3 is worth a shot. It looks like it is the most "correct" solution, but I haven't checked it any further.
          Hide
          danielbeck Daniel Beck added a comment -

          I'd vote for the first solution. It is easy to do and backwards-compatible and will also possibly be a remedy for any future application servers.

          The more robust we make this check, the less useful it is to identify configurations that can result in problems.

          Unless this is very specifically only ever going to be a problem for the request sent to trigger (or not) the warning – something I cannot off hand say is the case or not –, this approach would basically accomplish the opposite of what the monitor attempts to do. Because similar requests would remain mangled, and now it's not even easily diagnoseable, because Jenkins says everything is fine.

          Show
          danielbeck Daniel Beck added a comment - I'd vote for the first solution. It is easy to do and backwards-compatible and will also possibly be a remedy for any future application servers. The more robust we make this check, the less useful it is to identify configurations that can result in problems. Unless this is very specifically only ever going to be a problem for the request sent to trigger (or not) the warning – something I cannot off hand say is the case or not –, this approach would basically accomplish the opposite of what the monitor attempts to do. Because similar requests would remain mangled, and now it's not even easily diagnoseable, because Jenkins says everything is fine.
          Hide
          bmarwell Ben M added a comment -

          Very good point! As long as *all* calls do the same, it is fine. But it is hard to track, and plugins may break that rule. I haven't tried the third option, though. But the same restriction would probably still apply – every call and every plugin had to make these changes.

          There is also the idea to add a wiki link which explains how to configure various application servers.

          Show
          bmarwell Ben M added a comment - Very good point! As long as * all * calls do the same, it is fine. But it is hard to track, and plugins may break that rule. I haven't tried the third option, though. But the same restriction would probably still apply – every call and every plugin had to make these changes. There is also the idea to add a wiki link which explains how to configure various application servers.
          Hide
          danielbeck Daniel Beck added a comment -

          There is also the idea to add a wiki link which explains how to configure various application servers.

          Already exists (see the original report), unfortunately the wiki is currently read-only for most users.

          Show
          danielbeck Daniel Beck added a comment - There is also the idea to add a wiki link which explains how to configure various application servers. Already exists (see the original report), unfortunately the wiki is currently read-only for most users.
          Hide
          bmarwell Ben M added a comment -

          Alright, then please add:

          # existing
          -DJENKINS_HOME=/path/to/jenkins_home/
          # new
          -Dfile.encoding=UTF-8
          -Dclient.encoding.override=UTF-8
          

          to https://wiki.jenkins.io/display/JENKINS/Liberty+profile

          That should actually suffice. Thanks!
          Maybe Jenkins could also check the container implementation  and add the hint directly to the warning.

          Show
          bmarwell Ben M added a comment - Alright, then please add: # existing -DJENKINS_HOME=/path/to/jenkins_home/ # new -Dfile.encoding=UTF-8 -Dclient.encoding.override=UTF-8 to  https://wiki.jenkins.io/display/JENKINS/Liberty+profile That should actually suffice. Thanks! Maybe Jenkins could also check the container implementation  and add the hint directly to the warning.
          Hide
          danielbeck Daniel Beck added a comment -

           Maybe Jenkins could also check the container implementation and add the hint directly to the warning.

          Maybe, since we know what the mangled iso-8859-1 value is we could change the warning specifically for that. I like that.

          I updated the wiki page a bit differently, please check to see whether it's correct.

          Show
          danielbeck Daniel Beck added a comment -  Maybe Jenkins could also check the container implementation and add the hint directly to the warning. Maybe, since we know what the mangled iso-8859-1 value is we could change the warning specifically for that. I like that. I updated the wiki page a bit differently, please check to see whether it's correct.
          Hide
          bmarwell Ben M added a comment -

          I'd like to request some more changes.

          Jenkins works on all released versions of the Liberty profile, however it works best with the latest version (at the time of writing that was the copy shipped as a part of WebSphere Application Server v8.5.5.) and these instructions are based on 8.5.5.

          So, there is also the OpenSource fork in the meantime, https://openliberty.io/. Most non-enterprise users will likely use OpenLiberty instead. The current version of both IBM LibertyProfile and OpenLiberty is 20.0.0.8.

          Also, all wiki container pages should make the distinction between container-based security and application-based security. It is not clear enough that you can use either one. 

          Also, you can add the features from above (jaxws, jaxb) for Java 11, 14, 17+ compatibility. 

          The wiki page should also mention that Jenkins does not support Java 11 because of destroying its thread context: https://issues.jenkins-ci.org/browse/JENKINS-60979

          Thanks! You can close this issue after updating all the information.

          Show
          bmarwell Ben M added a comment - I'd like to request some more changes. Jenkins works on all released versions of the Liberty profile, however it works best with the  latest version  (at the time of writing that was the copy shipped as a part of WebSphere Application Server v8.5.5.) and these instructions are based on 8.5.5. So, there is also the OpenSource fork in the meantime,  https://openliberty.io/ . Most non-enterprise users will likely use OpenLiberty instead. The current version of both IBM LibertyProfile and OpenLiberty is 20.0.0.8. Also, all wiki container pages should make the distinction between container-based security and application-based security. It is not clear enough that you can use either one.  Also, you can add the features from above (jaxws, jaxb) for Java 11, 14, 17+ compatibility.  The wiki page should also mention that Jenkins does not support Java 11 because of destroying its thread context: https://issues.jenkins-ci.org/browse/JENKINS-60979 Thanks! You can close this issue after updating all the information.
          Hide
          danielbeck Daniel Beck added a comment -

          This one edit (plus corrections of mistakes) is all I could spare time for, anything else needs https://github.com/jenkins-infra/jenkins.io/blob/master/CONTRIBUTING.adoc#moving-documentation-from-jenkins-wiki

          Show
          danielbeck Daniel Beck added a comment - This one edit (plus corrections of mistakes) is all I could spare time for, anything else needs https://github.com/jenkins-infra/jenkins.io/blob/master/CONTRIBUTING.adoc#moving-documentation-from-jenkins-wiki
          Hide
          bmarwell Ben M added a comment -

          Daniel Beck please close this, I created  this issue: https://github.com/jenkins-infra/jenkins.io/issues/3645

          Please kindly take a look at the JDK 11 issue I linked. I do not know why such a blocker gets no attention

          Show
          bmarwell Ben M added a comment - Daniel Beck please close this, I created  this issue:  https://github.com/jenkins-infra/jenkins.io/issues/3645 Please kindly take a look at the JDK 11 issue I linked. I do not know why such a blocker gets no attention
          Hide
          danielbeck Daniel Beck added a comment -

          I do not know why such a blocker gets no attention

          In case that's not rhetorical: Based on our usage stats, just over 0.1% of Jenkins instances are running on JDK 11, despite us having multiple supported Docker images with that. Guess how many of those instances are also on Liberty. I would not be surprised to learn that you're the only one running Jenkins that way.

          Show
          danielbeck Daniel Beck added a comment - I do not know why such a blocker gets no attention In case that's not rhetorical: Based on our usage stats, just over 0.1% of Jenkins instances are running on JDK 11, despite us having multiple supported Docker images with that. Guess how many of those instances are also on Liberty. I would not be surprised to learn that you're the only one running Jenkins that way.
          Hide
          bmarwell Ben M added a comment -

          The referenced issue will result in an error on most application servers and affects all java versions since 9+.

          You can close this issue now, if you like.

          Show
          bmarwell Ben M added a comment - The referenced issue will result in an error on most application servers and affects all java versions since 9+. You can close this issue now, if you like.
          Hide
          danielbeck Daniel Beck added a comment -

          Ben M Thanks for the clarification.

          Show
          danielbeck Daniel Beck added a comment - Ben M Thanks for the clarification.
          Hide
          bmarwell Ben M added a comment -

          Wiki was updated.

          Show
          bmarwell Ben M added a comment - Wiki was updated.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            bmarwell Ben M
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: