Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-11543

UTF-8 encoding issue of build parameters as soon as including File parameter

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      String parameter UTF-8 encoding crash issue when include File paramemter

      for example,

      setting up File parameter and String parameter. > String parameter is "가나다라마바" (in korean) > build

      parameter result is "가나다라" (encoding crashed)

      please check it
      thanks

        Attachments

          Issue Links

            Activity

            senicy Han SeungHoon created issue -
            Hide
            beakmark Ryo Amano added a comment -

            I encountered similar problem.
            My case was 'Select parameter' with 'File parameter' instead of 'String parameter'.

            It seems to be crashed when input value includes non-ascii string if job has 'File parameter'.

            Show
            beakmark Ryo Amano added a comment - I encountered similar problem. My case was 'Select parameter' with 'File parameter' instead of 'String parameter'. It seems to be crashed when input value includes non-ascii string if job has 'File parameter'.
            Hide
            krizleebear krizleebear added a comment -

            The issue still exists with the newest version (1.577) today. This makes File parameters practically unusable for internationalized use cases!

            I've attached a minimal repro case config.xml. When you build it with default parameters (String parameter is defaulted to "äöü"), then this build will fail. Removing the File parameter restores the build.

            Please fix this issue. I was pretty surprised when I found this issue being known for 3 years.

            Show
            krizleebear krizleebear added a comment - The issue still exists with the newest version (1.577) today. This makes File parameters practically unusable for internationalized use cases! I've attached a minimal repro case config.xml. When you build it with default parameters (String parameter is defaulted to "äöü"), then this build will fail. Removing the File parameter restores the build. Please fix this issue. I was pretty surprised when I found this issue being known for 3 years.
            krizleebear krizleebear made changes -
            Field Original Value New Value
            Attachment config.xml [ 26725 ]
            krizleebear krizleebear made changes -
            Description String parameter UTF-8 encoding crash issue when include File paramemter

            for example,

            setting up File paramemter and String parameter. > String parameter is "가나다라마바" (in korean) > build

            parameter result is "가나다라" (encoding crashed)

            please check it
            thanks
            String parameter UTF-8 encoding crash issue when include File paramemter

            for example,

            setting up File parameter and String parameter. > String parameter is "가나다라마바" (in korean) > build

            parameter result is "가나다라" (encoding crashed)

            please check it
            thanks
            Environment none Linux
            Summary UTF-8 encoding crash issue when include File paramemter UTF-8 encoding issue of build parameters as soon as including File parameter
            danielbeck Daniel Beck made changes -
            Component/s core [ 15593 ]
            Affects Version/s current [ 10162 ]
            Due Date 2011-11-11
            Hide
            danielbeck Daniel Beck added a comment -

            When a file upload form field is included, it changes the request content type from application/x-www-form-urlencoded to multipart/form-data.

            Show
            danielbeck Daniel Beck added a comment - When a file upload form field is included, it changes the request content type from application/x-www-form-urlencoded to multipart/form-data.
            Hide
            danielbeck Daniel Beck added a comment -

            hudson-behavior.js seems to build a UTF-8 encoded JSON string, while the browser sets no charset, making it to default to Latin 1 when parsed on the server.

            Not sure who's to blame here (although I'd tentatively guess YUI/behavior should make sure the browser sets a charset for all the multipart text parts), but the following patch in Stapler works for me by simply parsing the JSON string in multipart form data as UTF-8 if no other charset is set:

            $ git diff
            diff --git a/core/src/main/java/org/kohsuke/stapler/RequestImpl.java b/core/src/main/java/org/kohsuke/stapler/RequestImpl.java
            index b8fb81c..230cef2 100644
            --- a/core/src/main/java/org/kohsuke/stapler/RequestImpl.java
            +++ b/core/src/main/java/org/kohsuke/stapler/RequestImpl.java
            @@ -852,8 +852,19 @@ public class RequestImpl extends HttpServletRequestWrapper implements StaplerReq
                             isSubmission=true;
                             parseMultipartFormData();
                             FileItem item = parsedFormData.get("json");
            -                if(item!=null)
            -                    p = item.getString();
            +                if(item!=null) {
            +                    if (item.getContentType() == null) {
            +                        // JENKINS-11543: Client doesn't set charset per part, so
            +                        // default to UTF-8 if no other charset is specified.
            +                        try {
            +                            p = item.getString("UTF-8");
            +                        } catch (java.io.UnsupportedEncodingException uee) {
            +                            throw new AssertionError("UTF-8 unsupported")
            +                        }
            +                    } else {
            +                        p = item.getString();
            +                    }
            +                }
                         } else {
                             p = getParameter("json");
                             isSubmission = !getParameterMap().isEmpty();

            Of course anything not relying on structured form submission will continue to be broken.

            Asked KK on IRC where to file issues with Stapler, as the GitHub issue tracker seems abandoned.

            Show
            danielbeck Daniel Beck added a comment - hudson-behavior.js seems to build a UTF-8 encoded JSON string, while the browser sets no charset, making it to default to Latin 1 when parsed on the server. Not sure who's to blame here (although I'd tentatively guess YUI/behavior should make sure the browser sets a charset for all the multipart text parts), but the following patch in Stapler works for me by simply parsing the JSON string in multipart form data as UTF-8 if no other charset is set: $ git diff diff --git a/core/src/main/java/org/kohsuke/stapler/RequestImpl.java b/core/src/main/java/org/kohsuke/stapler/RequestImpl.java index b8fb81c..230cef2 100644 --- a/core/src/main/java/org/kohsuke/stapler/RequestImpl.java +++ b/core/src/main/java/org/kohsuke/stapler/RequestImpl.java @@ -852,8 +852,19 @@ public class RequestImpl extends HttpServletRequestWrapper implements StaplerReq isSubmission= true ; parseMultipartFormData(); FileItem item = parsedFormData.get( "json" ); - if (item!= null ) - p = item.getString(); + if (item!= null ) { + if (item.getContentType() == null ) { + // JENKINS-11543: Client doesn't set charset per part, so + // default to UTF-8 if no other charset is specified. + try { + p = item.getString( "UTF-8" ); + } catch (java.io.UnsupportedEncodingException uee) { + throw new AssertionError( "UTF-8 unsupported" ) + } + } else { + p = item.getString(); + } + } } else { p = getParameter( "json" ); isSubmission = !getParameterMap().isEmpty(); Of course anything not relying on structured form submission will continue to be broken. Asked KK on IRC where to file issues with Stapler, as the GitHub issue tracker seems abandoned.
            Hide
            krizleebear krizleebear added a comment -

            Thanks for the quick reply!
            To be honest I didn't expect to get an update on a topic this old.
            I'm looking forward to see further updates to this issue and of course a fix.

            Show
            krizleebear krizleebear added a comment - Thanks for the quick reply! To be honest I didn't expect to get an update on a topic this old. I'm looking forward to see further updates to this issue and of course a fix.
            Hide
            danielbeck Daniel Beck added a comment -

            krizleebear: It was easy to reproduce and someone confirmed it's still a problem. What's not to love?


            Possibly related library issue: https://issues.apache.org/jira/browse/FILEUPLOAD-206

            FileUpload fails to handle requests with a form enctype of multipart/form-data;charset=UTF-8, so I doubt this can be resolved client-side (happy to be proven wrong!), so the Stapler fix above seems to be the correct solution for now.

            Show
            danielbeck Daniel Beck added a comment - krizleebear : It was easy to reproduce and someone confirmed it's still a problem. What's not to love? Possibly related library issue: https://issues.apache.org/jira/browse/FILEUPLOAD-206 FileUpload fails to handle requests with a form enctype of multipart/form-data;charset=UTF-8 , so I doubt this can be resolved client-side (happy to be proven wrong!), so the Stapler fix above seems to be the correct solution for now.
            Hide
            krizleebear krizleebear added a comment -

            Well - thanks a lot, I'm really thrilled about your fast response and take of direct action!
            I'm a fan of hard-to-guess issues. I even browsed github and had a swift look at two classes (FileParameterDefinition and FileParameterValue) in Jenkins source code on my own... but as complete stranger to the code I didn't realize where the error had its origin.

            Show
            krizleebear krizleebear added a comment - Well - thanks a lot, I'm really thrilled about your fast response and take of direct action! I'm a fan of hard-to-guess issues. I even browsed github and had a swift look at two classes (FileParameterDefinition and FileParameterValue) in Jenkins source code on my own... but as complete stranger to the code I didn't realize where the error had its origin.
            danielbeck Daniel Beck made changes -
            Assignee Daniel Beck [ danielbeck ]
            danielbeck Daniel Beck made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            danielbeck Daniel Beck made changes -
            Remote Link This issue links to "Stapler PR 38 (Web Link)" [ 11406 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            changelog.html
            core/pom.xml
            http://jenkins-ci.org/commit/jenkins/c57f9ed577b9635f01aa5affabf1f770b689a115
            Log:
            [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/pom.xml http://jenkins-ci.org/commit/jenkins/c57f9ed577b9635f01aa5affabf1f770b689a115 Log: [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3689
            [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (Revision c57f9ed577b9635f01aa5affabf1f770b689a115)

            Result = SUCCESS
            kohsuke : c57f9ed577b9635f01aa5affabf1f770b689a115
            Files :

            • changelog.html
            • core/pom.xml
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3689 [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (Revision c57f9ed577b9635f01aa5affabf1f770b689a115) Result = SUCCESS kohsuke : c57f9ed577b9635f01aa5affabf1f770b689a115 Files : changelog.html core/pom.xml
            Hide
            danielbeck Daniel Beck added a comment -

            Fix scheduled for 1.582.

            Show
            danielbeck Daniel Beck added a comment - Fix scheduled for 1.582.
            danielbeck Daniel Beck made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            ircbot Jenkins IRC Bot made changes -
            Component/s parameters [ 15594 ]
            danielbeck Daniel Beck made changes -
            Labels lts-candidate
            danielbeck Daniel Beck made changes -
            Labels lts-candidate 1.565.3-fixed lts-candidate
            olivergondza Oliver Gondža made changes -
            Labels 1.565.3-fixed lts-candidate 1.565.3-fixed 1.580.1-fixed
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/pom.xml
            http://jenkins-ci.org/commit/jenkins/8ae29e50c7356a8e2ec9be1126bad957dab0117a
            Log:
            [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level

            (cherry picked from commit c57f9ed577b9635f01aa5affabf1f770b689a115)

            Conflicts:
            changelog.html

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/8ae29e50c7356a8e2ec9be1126bad957dab0117a Log: [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (cherry picked from commit c57f9ed577b9635f01aa5affabf1f770b689a115) Conflicts: changelog.html
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            test/src/test/java/hudson/model/ParametersTest.java
            http://jenkins-ci.org/commit/jenkins/75f5d2180f7340ce2efd4f41c8670afb049eb530
            Log:
            JENKINS-11543 Unit test

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: test/src/test/java/hudson/model/ParametersTest.java http://jenkins-ci.org/commit/jenkins/75f5d2180f7340ce2efd4f41c8670afb049eb530 Log: JENKINS-11543 Unit test
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3746
            JENKINS-11543 Unit test (Revision 75f5d2180f7340ce2efd4f41c8670afb049eb530)

            Result = SUCCESS
            ogondza : 75f5d2180f7340ce2efd4f41c8670afb049eb530
            Files :

            • test/src/test/java/hudson/model/ParametersTest.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3746 JENKINS-11543 Unit test (Revision 75f5d2180f7340ce2efd4f41c8670afb049eb530) Result = SUCCESS ogondza : 75f5d2180f7340ce2efd4f41c8670afb049eb530 Files : test/src/test/java/hudson/model/ParametersTest.java
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            test/src/test/java/hudson/model/ParametersTest.java
            http://jenkins-ci.org/commit/jenkins/30715dfd46d8b9987635c6a39cdd21c55d0bcc87
            Log:
            JENKINS-11543 Unit test

            (cherry picked from commit 75f5d2180f7340ce2efd4f41c8670afb049eb530)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: test/src/test/java/hudson/model/ParametersTest.java http://jenkins-ci.org/commit/jenkins/30715dfd46d8b9987635c6a39cdd21c55d0bcc87 Log: JENKINS-11543 Unit test (cherry picked from commit 75f5d2180f7340ce2efd4f41c8670afb049eb530)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oliver Gondža
            Path:
            test/src/test/java/hudson/model/ParametersTest.java
            http://jenkins-ci.org/commit/jenkins/66dce647d1d7df919f9707b4ce631cd49dc242f7
            Log:
            JENKINS-11543 Unit test

            (cherry picked from commit 75f5d2180f7340ce2efd4f41c8670afb049eb530)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: test/src/test/java/hudson/model/ParametersTest.java http://jenkins-ci.org/commit/jenkins/66dce647d1d7df919f9707b4ce631cd49dc242f7 Log: JENKINS-11543 Unit test (cherry picked from commit 75f5d2180f7340ce2efd4f41c8670afb049eb530)
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #4292
            [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (Revision 8ae29e50c7356a8e2ec9be1126bad957dab0117a)
            JENKINS-11543 Unit test (Revision 66dce647d1d7df919f9707b4ce631cd49dc242f7)

            Result = UNSTABLE
            ogondza : 8ae29e50c7356a8e2ec9be1126bad957dab0117a
            Files :

            • core/pom.xml

            ogondza : 66dce647d1d7df919f9707b4ce631cd49dc242f7
            Files :

            • test/src/test/java/hudson/model/ParametersTest.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #4292 [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (Revision 8ae29e50c7356a8e2ec9be1126bad957dab0117a) JENKINS-11543 Unit test (Revision 66dce647d1d7df919f9707b4ce631cd49dc242f7) Result = UNSTABLE ogondza : 8ae29e50c7356a8e2ec9be1126bad957dab0117a Files : core/pom.xml ogondza : 66dce647d1d7df919f9707b4ce631cd49dc242f7 Files : test/src/test/java/hudson/model/ParametersTest.java
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 141770 ] JNJira + In-Review [ 189811 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Kohsuke Kawaguchi
            Path:
            core/pom.xml
            http://jenkins-ci.org/commit/jenkins/12a4b444a96d30a57e78c2427d07a46172aeb8e8
            Log:
            [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level

            (cherry picked from commit c57f9ed577b9635f01aa5affabf1f770b689a115)

            Conflicts:
            changelog.html

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/12a4b444a96d30a57e78c2427d07a46172aeb8e8 Log: [JENKINS-24458 JENKINS-11543] integrated the newer version of stapler with reduced log level (cherry picked from commit c57f9ed577b9635f01aa5affabf1f770b689a115) Conflicts: changelog.html

              People

              Assignee:
              danielbeck Daniel Beck
              Reporter:
              senicy Han SeungHoon
              Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: