Jakub Pawlinski This is a known with the Unicode spec and the Java platform implementation of it, not Pipeline. In UTF-8 the BOM is neither needed nor suggested - since the BOM is essentially meaningless in UTF-8, Java transparently passes the BOM through.

First I'd make sure to add the "encloding: 'UTF-8'" argument to your readFile step to ensure it reads as UTF-8. Then we do postprocessing to correct for nonstandard input.

Some suggested solutions are available on StackOverflow.

Personally, I'd do something like this to sanitize your input:

/** These are UTF-8 BOM characters */
private static String removeUTF8BOM(String s) {
    return s.replace("\uEFBBBF", "");
}

(might need to be \u FEFF, try it both ways).

There's also code snippets out there that do a more efficient approach, which only considers the leading bytes of the String.

This is due to a known problem with Java's implementation of the UTF-8 spec. Suggested an easy workaround in Pipeline code to solve the issue.

Ok, but if its Java issue, why I could not replicate in locally using Groovy Version: 2.6.0-alpha-1 JVM: 1.8.0_111 Vendor: Oracle Corporation OS: Windows 10

I guess Sam used vague wording.  It's the files that harbour the UTF-8-encoded BOM mark at the beginning, which is useless because UTF-8's bytewise storage does not depend on the architecture's byte order.

$ python -c 'u = b"\xEF\xBB\xBF".decode("utf-8"); print "%04X" % (ord(u[0]),)'
FEFF

Microsoft creates files with these useless but confusing 3 bytes at the beginning of its UTF-8-encoded files.  Now every program that reads such files needs to trim the Unicode BOM character at the beginning of the contents after decoding to Unicode.

public static CharSequence deBOM(CharSequence s) {
    if (s == null) {
        return null
    } else if (s.length() == 0) {
        return s
    } else if (s[0] == '\uFEFF') {
        return s.drop(1)
    } else {
        return s
    }
}

https://stackoverflow.com/questions/5406172/utf-8-without-bom

Perhaps, newer XMLSlurper performs this santitation.

This appears a post-modern BOM that is supposed to tell the encoding of the file before decoding it.

          +------------------+----------+
          | Leading sequence | Encoding |
          +------------------+----------+
          | FF FE 00 00      | UTF-32LE |
          | 00 00 FE FF      | UTF-32BE |
          | FF FE            | UTF-16LE |
          | FE FF            | UTF-16BE |
          | EF BB BF         | UTF-8    |
          +------------------+----------+

http://www.rfc-editor.org/rfc/rfc4329.txt

So readFile needs a mode (or a special value for the encoding parameter) to sense the post-modern BOM and decode the rest of the contents accordingly.

same issue with readCSV, possibly all other ways of reading files via jenkins. The issue with readCSV is more severe as I cannot step in between reading the content of the file and the content being processed to Commons CSV structure. Only way to do this is to readFile and parse it manually which makes readCSV (and other functionalities like that) redundant.

I still don't understand why you claim its not jenkins but java issue while its not replicable even in newer groovy version.

Affected functionalities:

• readCSV
• readJSON
• readManifest
• readMavenPom
• readProperties
• readYaml