Jenkins.instance.getExtensionList(hudson.plugins.warnings.parser.AbstractWarningsParser)

in Script Console returns the javac parser twice, and here's the reason:

https://github.com/jenkinsci/warnings-plugin/blob/master/src/main/java/hudson/plugins/warnings/parser/AntJavacParser.java#L29
https://github.com/jenkinsci/warnings-plugin/blob/master/src/main/java/hudson/plugins/warnings/parser/JavacParser.java#L26
https://github.com/jenkinsci/warnings-plugin/blob/master/src/main/resources/hudson/plugins/warnings/parser/Messages.properties#L47

Both javac and Ant call themselves 'Java Compiler (javac)', making the actual 'Ant' one not selectable.

Code changed in jenkins
User: Ulli Hafner
Path:
src/test/java/hudson/plugins/warnings/parser/ParserRegistryIntegrationTest.java
src/test/resources/hudson/plugins/warnings/parser/issue24611.txt
http://jenkins-ci.org/commit/warnings-plugin/70e7048ef9ecc3d09be4b5c676e617c987e3b5a6
Log:
JENKINS-24611 Added testcase.

> You can't directly select the ant javac parser. It is part of the sub-set of the javac parsers.
I see the proper settings for ant javac is to select 'Java Compiler (javac)' parser.

Then, I have tried to use warnings plugin the following cases.
1) Linux, Jenkins 1.568, Warnings plugin 4.41, compiler option "-Xlint"
2) Linux, Jenkins 1.568, Warnings plugin 4.41, compiler option "-Xlint -J-Duser.language=en"
3) Windows, Jenkins 1.578, Warnings plugin 4.41, compiler option "-Xlint"
4) Windows, Jenkins 1.578, Warnings plugin 4.41, compiler option "-Xlint -J-Duser.language=en"

Notes: My environment is japanese(user.language=ja), and javac output message is internationalized, especially 'warning' is '警告'.

The results and row log file name attached are as follows:
1) Good, 24611_case1.log
2) Good, 24611_case2.log
3) Bad, 24611_case3.log
4) Good, 24611_case4.log

For OS encoding, Linux default encoding is UTF-8. Windows default encoding is not UTF-8 but windows-31j (CP932).

Seems that the parser already supports the Japanese words for warning, did these names change?

In case 3, did you specify the correct encoding in the configuration screen?

Code changed in jenkins
User: Ulli Hafner
Path:
src/test/java/hudson/plugins/warnings/parser/AntJavacParserTest.java
http://jenkins-ci.org/commit/warnings-plugin/387e9c5a2dd6c2dc07155be9c19131a204c35381
Log:
JENKINS-24611 Added testcase.

Code changed in jenkins
User: Ulli Hafner
Path:
src/test/java/hudson/plugins/warnings/parser/AntJavacParserTest.java
http://jenkins-ci.org/commit/warnings-plugin/51df96d0253d4541d3131bbe1f4927d16b4da3fb
Log:
JENKINS-24611 Added testcase.

Would it be possible if you could update the test case (and parser) on your own? Seems not easy to work in a foreign locale for me. I added the test case in AntJavacParserTest#testIssue24611.

I have tried the test case. But this is the first time to build and run jenkins plugin.

• cloned git repository
  git clone https://github.com/jenkinsci/warnings-plugin.git

• moved current directory in the cloned directory(i.e. jenkins-plugin)
  cd warnings-plugin

• run mvn test
  mvn test

I attached the output of mvn test as "mvn-test.log"

• There is no file in target\surefire-reports.

The following are my environment
>mvn -version
Apache Maven 3.0.4 (r1232337; 2012-01-17 17:44:56+0900)
Maven home: C:\java\apache-maven-3.0.4\bin\..
Java version: 1.8.0_20, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk1.8.0_20\jre
Default locale: ja_JP, platform encoding: MS932
OS name: "windows 7", version: "6.1", arch: "amd64", family: "dos"

For case 3, I have set some encodings at project "Configure" > "Post-build Actions" > "Scan for compiler warnings" > "Advanced" > "Default Encoding".
encodings tried are as follows:

• blank (default)
• "windows-31j"
• "UTF-8"
• "UTF-16"
• "euc-jp-linux"

Results are same, 0 warnings.

note) "windows-31j" and "MS932" are same. IANA registration name is "Windows-31J".

Expected to succeed in my environment（Windows OS, platform encoding Windows-31J):

• blank
• "windows-31j"

note)blank is explained to mean platform default encoding.

I tried to run jenkins with debugger attachable command-line option, then attached NetBeans debugger to the jenkins and analyzed the situation.

• jenkins commandline
  > java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=8086,suspend=n -jar jenkins-1.579.war

• put break point at AntJavacParser.isLineInteresting.
  In this point, argument "line" is invalid in any encoding like this.
  "\ufffd\ufffd\ufffd[\ufffdU\ufffd[anonymous\ufffd\ufffd\ufffd\ufffd\ufffds"
  Non ascii character is misconverted.

So, I looked for where reading console output and what encoding specified, from
stack trace from AntJavacParser.isLineInteresting.

The place is at ParserRegistory.createReader

protected Reader createReader(final InputStream inputStream)

The specified encoding is a instance field "defaultCharset".
This field is set in constructor.

public ParserRegistry(final List<? extends AbstractWarningsParser> parsers, final String defaultEncoding,
final String includePattern, final String excludePattern) {
defaultCharset = EncodingValidator.defaultCharset(defaultEncoding);

This ParserRegistry constructor is called from WarningsPublisher.parseConsoleLog method.

Collection<FileAnnotation> warnings = new　ParserRegistry(
ParserRegistry.getParsers(parserName),
CONSOLE_LOG_ENCODING, getIncludePattern(), getExcludePattern()).parse(build.getLogFile());

Above code, encoding is specified by WarningsPublisher class final field CONSOLE_LOG_ENCODING.

private static final String CONSOLE_LOG_ENCODING = "UTF-8";

This looks like causing miss detections on non UTF-8 encoding platform.

Possible fix is

Collection<FileAnnotation> warnings = new ParserRegistry(
ParserRegistry.getParsers(parserName),
getDefaultEncoding(), getIncludePattern(),
getExcludePattern()).parse(build.getLogFile());

I changed the 2nd argument from CONSOLE_LOG_ENCODING to getDefaultEncoding().
Then the default encoding specified in configuration is applied, and
warnings can detected on Windows OS windows-31j(MS932) encoding.

OK, I see. I thought that the console encoding is always UTF8. I.e., if I change that to defaultEncoding then everything works as expected?

For example, in traditional Unix OS is not using "UTF-8" but "EUC-JP" or other encoding as Japanese environment. Recently, especially for Linux, uses "UTF-8" encoding as Japanese environment.

I have tried the warnings plugin with defaultEncoding modified for encodings of EUC-JP, UTF-8, and Windows-31J. These three cases are Ok for counting ant javac warning messages both english and localized(Japanese).

Without any fix, parsing can only handle ASCII characters on the platform other than UTF-8 because many encodings contain ASCII code. And advanced default encoding configuration is ignored.

For example, in traditional Unix OS is not using "UTF-8" but "EUC-JP" or other encoding as Japanese environment. Recently, especially for Linux, uses "UTF-8" encoding as Japanese environment.

OS X is UTF-8 for the last few years as well (not claiming it's "traditional Unix", just an extra data point).

For some perspective, based on anonymous usage stats, use of Unixes other than Linux and OS X is really low. Of the ~105600 nodes with known OS, only 2250, or ~2.5%, are not Windows, Linux, or OS X; and of those, more than half are Solaris (SunOS).

I'm not talking about the linux or unix console. My question is if the console log file created by Jenkins uses the default locale and not UTF-8? But since you tested it the suggested fix will be ok indeed!

Code changed in jenkins
User: Ulli Hafner
Path:
.idea/libraries/Maven__org_apache_commons_commons_lang3_3_1.xml
analysis-core
tasks
http://jenkins-ci.org/commit/analysis-suite-plugin/db3e86644600ecedef1315cc00291d2003f154b6
Log:
[FIXED JENKINS-24611] Use default encoding when reading the console log.

Code changed in jenkins
User: Ulli Hafner
Path:
src/main/java/hudson/plugins/warnings/WarningsPublisher.java
warnings.iml
http://jenkins-ci.org/commit/warnings-plugin/6f457dce7e88581d733a1bf93da0f1091831d852
Log:
[FIXED JENKINS-24611] Use default encoding when reading the console log.

If you still want to add a good Japanese test case feel free to add a pull request...

> I'm not talking about the linux or unix console. My question is if the console log file created by Jenkins uses the default locale and not UTF-8?

Yes, jenkins generates a console log as platform default encoding, not UTF-8.
I analyzed some jenkins console log file which is /var/lib/jenkins/jobs/<job name>/builds/<year-month-date_hour_min_sec>/log on Linux or C:\Users\<user name>\.jenkins\jobs\<job name>\builds\<year-month-date_hour_min_sec>/log on Windows.

Analyzied cases are as follows:
1) Linux UTF-8 encoding platform => log file is UTF-8 encoding.
This log is attached as name 24611_case1.log and 24611_case2.log
2) Linux EUC-JP encoding platform => log file is euc-jp encoding.
3) Windows Windows-31J encoding platform => log file is windows-31j encoding.
This log is attached as name 24611_case3.log and 24611_case4.log

I tried but failed to create unit test for WarningsPublisher.
It is so difficult for me to prepare the instances of build and project, and to control them for testing.
Mock approach may be suitable but I don't try yet.

For the unit test you just need to replace the file issue24611.txt with your log file and adapt the unit test in https://github.com/jenkinsci/warnings-plugin/blob/51df96d0253d4541d3131bbe1f4927d16b4da3fb/src/test/java/hudson/plugins/warnings/parser/AntJavacParserTest.java#L33.

You need to change the locale and the number of warnings in the log. (Optionally you can check for the warning attributes, too.)