Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11543

UTF-8 encoding issue of build parameters as soon as including File parameter

      String parameter UTF-8 encoding crash issue when include File paramemter

      for example,

      setting up File parameter and String parameter. > String parameter is "가나다라마바" (in korean) > build

      parameter result is "가나다라" (encoding crashed)

      please check it
      thanks

          [JENKINS-11543] UTF-8 encoding issue of build parameters as soon as including File parameter

          Ryo Amano added a comment -

          I encountered similar problem.
          My case was 'Select parameter' with 'File parameter' instead of 'String parameter'.

          It seems to be crashed when input value includes non-ascii string if job has 'File parameter'.

          Ryo Amano added a comment - I encountered similar problem. My case was 'Select parameter' with 'File parameter' instead of 'String parameter'. It seems to be crashed when input value includes non-ascii string if job has 'File parameter'.

          krizleebear added a comment -

          The issue still exists with the newest version (1.577) today. This makes File parameters practically unusable for internationalized use cases!

          I've attached a minimal repro case config.xml. When you build it with default parameters (String parameter is defaulted to "äöü"), then this build will fail. Removing the File parameter restores the build.

          Please fix this issue. I was pretty surprised when I found this issue being known for 3 years.

          krizleebear added a comment - The issue still exists with the newest version (1.577) today. This makes File parameters practically unusable for internationalized use cases! I've attached a minimal repro case config.xml. When you build it with default parameters (String parameter is defaulted to "äöü"), then this build will fail. Removing the File parameter restores the build. Please fix this issue. I was pretty surprised when I found this issue being known for 3 years.

          Daniel Beck added a comment -

          When a file upload form field is included, it changes the request content type from application/x-www-form-urlencoded to multipart/form-data.

          Daniel Beck added a comment - When a file upload form field is included, it changes the request content type from application/x-www-form-urlencoded to multipart/form-data.

          Daniel Beck added a comment -

          hudson-behavior.js seems to build a UTF-8 encoded JSON string, while the browser sets no charset, making it to default to Latin 1 when parsed on the server.

          Not sure who's to blame here (although I'd tentatively guess YUI/behavior should make sure the browser sets a charset for all the multipart text parts), but the following patch in Stapler works for me by simply parsing the JSON string in multipart form data as UTF-8 if no other charset is set:

          $ git diff
          diff --git a/core/src/main/java/org/kohsuke/stapler/RequestImpl.java b/core/src/main/java/org/kohsuke/stapler/RequestImpl.java
          index b8fb81c..230cef2 100644
          --- a/core/src/main/java/org/kohsuke/stapler/RequestImpl.java
          +++ b/core/src/main/java/org/kohsuke/stapler/RequestImpl.java
          @@ -852,8 +852,19 @@ public class RequestImpl extends HttpServletRequestWrapper implements StaplerReq
                           isSubmission=true;
                           parseMultipartFormData();
                           FileItem item = parsedFormData.get("json");
          -                if(item!=null)
          -                    p = item.getString();
          +                if(item!=null) {
          +                    if (item.getContentType() == null) {
          +                        // JENKINS-11543: Client doesn't set charset per part, so
          +                        // default to UTF-8 if no other charset is specified.
          +                        try {
          +                            p = item.getString("UTF-8");
          +                        } catch (java.io.UnsupportedEncodingException uee) {
          +                            throw new AssertionError("UTF-8 unsupported")
          +                        }
          +                    } else {
          +                        p = item.getString();
          +                    }
          +                }
                       } else {
                           p = getParameter("json");
                           isSubmission = !getParameterMap().isEmpty();

          Of course anything not relying on structured form submission will continue to be broken.

          Asked KK on IRC where to file issues with Stapler, as the GitHub issue tracker seems abandoned.

          Daniel Beck added a comment - hudson-behavior.js seems to build a UTF-8 encoded JSON string, while the browser sets no charset, making it to default to Latin 1 when parsed on the server. Not sure who's to blame here (although I'd tentatively guess YUI/behavior should make sure the browser sets a charset for all the multipart text parts), but the following patch in Stapler works for me by simply parsing the JSON string in multipart form data as UTF-8 if no other charset is set: $ git diff diff --git a/core/src/main/java/org/kohsuke/stapler/RequestImpl.java b/core/src/main/java/org/kohsuke/stapler/RequestImpl.java index b8fb81c..230cef2 100644 --- a/core/src/main/java/org/kohsuke/stapler/RequestImpl.java +++ b/core/src/main/java/org/kohsuke/stapler/RequestImpl.java @@ -852,8 +852,19 @@ public class RequestImpl extends HttpServletRequestWrapper implements StaplerReq isSubmission= true ; parseMultipartFormData(); FileItem item = parsedFormData.get( "json" ); - if (item!= null ) - p = item.getString(); + if (item!= null ) { + if (item.getContentType() == null ) { + // JENKINS-11543: Client doesn't set charset per part, so + // default to UTF-8 if no other charset is specified. + try { + p = item.getString( "UTF-8" ); + } catch (java.io.UnsupportedEncodingException uee) { + throw new AssertionError( "UTF-8 unsupported" ) + } + } else { + p = item.getString(); + } + } } else { p = getParameter( "json" ); isSubmission = !getParameterMap().isEmpty(); Of course anything not relying on structured form submission will continue to be broken. Asked KK on IRC where to file issues with Stapler, as the GitHub issue tracker seems abandoned.

          krizleebear added a comment -

          Thanks for the quick reply!
          To be honest I didn't expect to get an update on a topic this old.
          I'm looking forward to see further updates to this issue and of course a fix.

          krizleebear added a comment - Thanks for the quick reply! To be honest I didn't expect to get an update on a topic this old. I'm looking forward to see further updates to this issue and of course a fix.

          Daniel Beck added a comment -

          krizleebear: It was easy to reproduce and someone confirmed it's still a problem. What's not to love?


          Possibly related library issue: https://issues.apache.org/jira/browse/FILEUPLOAD-206

          FileUpload fails to handle requests with a form enctype of multipart/form-data;charset=UTF-8, so I doubt this can be resolved client-side (happy to be proven wrong!), so the Stapler fix above seems to be the correct solution for now.

          Daniel Beck added a comment - krizleebear : It was easy to reproduce and someone confirmed it's still a problem. What's not to love? Possibly related library issue: https://issues.apache.org/jira/browse/FILEUPLOAD-206 FileUpload fails to handle requests with a form enctype of multipart/form-data;charset=UTF-8 , so I doubt this can be resolved client-side (happy to be proven wrong!), so the Stapler fix above seems to be the correct solution for now.

          krizleebear added a comment -

          Well - thanks a lot, I'm really thrilled about your fast response and take of direct action!
          I'm a fan of hard-to-guess issues. I even browsed github and had a swift look at two classes (FileParameterDefinition and FileParameterValue) in Jenkins source code on my own... but as complete stranger to the code I didn't realize where the error had its origin.

          krizleebear added a comment - Well - thanks a lot, I'm really thrilled about your fast response and take of direct action! I'm a fan of hard-to-guess issues. I even browsed github and had a swift look at two classes (FileParameterDefinition and FileParameterValue) in Jenkins source code on my own... but as complete stranger to the code I didn't realize where the error had its origin.

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          changelog.html
          core/pom.xml
          http://jenkins-ci.org/commit/jenkins/c57f9ed577b9635f01aa5affabf1f770b689a115
          Log:
          [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/pom.xml http://jenkins-ci.org/commit/jenkins/c57f9ed577b9635f01aa5affabf1f770b689a115 Log: [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3689
          [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (Revision c57f9ed577b9635f01aa5affabf1f770b689a115)

          Result = SUCCESS
          kohsuke : c57f9ed577b9635f01aa5affabf1f770b689a115
          Files :

          • changelog.html
          • core/pom.xml

          dogfood added a comment - Integrated in jenkins_main_trunk #3689 [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (Revision c57f9ed577b9635f01aa5affabf1f770b689a115) Result = SUCCESS kohsuke : c57f9ed577b9635f01aa5affabf1f770b689a115 Files : changelog.html core/pom.xml

          Daniel Beck added a comment -

          Fix scheduled for 1.582.

          Daniel Beck added a comment - Fix scheduled for 1.582.

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/pom.xml
          http://jenkins-ci.org/commit/jenkins/8ae29e50c7356a8e2ec9be1126bad957dab0117a
          Log:
          [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level

          (cherry picked from commit c57f9ed577b9635f01aa5affabf1f770b689a115)

          Conflicts:
          changelog.html

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/8ae29e50c7356a8e2ec9be1126bad957dab0117a Log: [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (cherry picked from commit c57f9ed577b9635f01aa5affabf1f770b689a115) Conflicts: changelog.html

          Code changed in jenkins
          User: Oliver Gondža
          Path:
          test/src/test/java/hudson/model/ParametersTest.java
          http://jenkins-ci.org/commit/jenkins/75f5d2180f7340ce2efd4f41c8670afb049eb530
          Log:
          JENKINS-11543 Unit test

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: test/src/test/java/hudson/model/ParametersTest.java http://jenkins-ci.org/commit/jenkins/75f5d2180f7340ce2efd4f41c8670afb049eb530 Log: JENKINS-11543 Unit test

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3746
          JENKINS-11543 Unit test (Revision 75f5d2180f7340ce2efd4f41c8670afb049eb530)

          Result = SUCCESS
          ogondza : 75f5d2180f7340ce2efd4f41c8670afb049eb530
          Files :

          • test/src/test/java/hudson/model/ParametersTest.java

          dogfood added a comment - Integrated in jenkins_main_trunk #3746 JENKINS-11543 Unit test (Revision 75f5d2180f7340ce2efd4f41c8670afb049eb530) Result = SUCCESS ogondza : 75f5d2180f7340ce2efd4f41c8670afb049eb530 Files : test/src/test/java/hudson/model/ParametersTest.java

          Code changed in jenkins
          User: Oliver Gondža
          Path:
          test/src/test/java/hudson/model/ParametersTest.java
          http://jenkins-ci.org/commit/jenkins/30715dfd46d8b9987635c6a39cdd21c55d0bcc87
          Log:
          JENKINS-11543 Unit test

          (cherry picked from commit 75f5d2180f7340ce2efd4f41c8670afb049eb530)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: test/src/test/java/hudson/model/ParametersTest.java http://jenkins-ci.org/commit/jenkins/30715dfd46d8b9987635c6a39cdd21c55d0bcc87 Log: JENKINS-11543 Unit test (cherry picked from commit 75f5d2180f7340ce2efd4f41c8670afb049eb530)

          Code changed in jenkins
          User: Oliver Gondža
          Path:
          test/src/test/java/hudson/model/ParametersTest.java
          http://jenkins-ci.org/commit/jenkins/66dce647d1d7df919f9707b4ce631cd49dc242f7
          Log:
          JENKINS-11543 Unit test

          (cherry picked from commit 75f5d2180f7340ce2efd4f41c8670afb049eb530)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: test/src/test/java/hudson/model/ParametersTest.java http://jenkins-ci.org/commit/jenkins/66dce647d1d7df919f9707b4ce631cd49dc242f7 Log: JENKINS-11543 Unit test (cherry picked from commit 75f5d2180f7340ce2efd4f41c8670afb049eb530)

          dogfood added a comment -

          Integrated in jenkins_main_trunk #4292
          [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (Revision 8ae29e50c7356a8e2ec9be1126bad957dab0117a)
          JENKINS-11543 Unit test (Revision 66dce647d1d7df919f9707b4ce631cd49dc242f7)

          Result = UNSTABLE
          ogondza : 8ae29e50c7356a8e2ec9be1126bad957dab0117a
          Files :

          • core/pom.xml

          ogondza : 66dce647d1d7df919f9707b4ce631cd49dc242f7
          Files :

          • test/src/test/java/hudson/model/ParametersTest.java

          dogfood added a comment - Integrated in jenkins_main_trunk #4292 [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (Revision 8ae29e50c7356a8e2ec9be1126bad957dab0117a) JENKINS-11543 Unit test (Revision 66dce647d1d7df919f9707b4ce631cd49dc242f7) Result = UNSTABLE ogondza : 8ae29e50c7356a8e2ec9be1126bad957dab0117a Files : core/pom.xml ogondza : 66dce647d1d7df919f9707b4ce631cd49dc242f7 Files : test/src/test/java/hudson/model/ParametersTest.java

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/pom.xml
          http://jenkins-ci.org/commit/jenkins/12a4b444a96d30a57e78c2427d07a46172aeb8e8
          Log:
          [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level

          (cherry picked from commit c57f9ed577b9635f01aa5affabf1f770b689a115)

          Conflicts:
          changelog.html

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/12a4b444a96d30a57e78c2427d07a46172aeb8e8 Log: [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (cherry picked from commit c57f9ed577b9635f01aa5affabf1f770b689a115) Conflicts: changelog.html

            danielbeck Daniel Beck
            senicy Han SeungHoon
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: