We've got a really weird bug in Jenkins that we can reproduce locally, but we have no idea what actually causes it. We also have a workaround in our pipeline scripts, but it's quite hacky and we thought we'd report the actual issue here in hopes that it could be fixed.

We first noticed this bug in our Jenkins controller around 4 years ago, and then it "disappeared" (due to modifications in our pipeline script which accidentally led to the workaround that I will describe shortly), and then it re-appeared again recently when further modifications were made. As such, we're pretty confident that this bug still exists in the latest Jenkins version, but we've reproduced it with 2.476.

The bug is as follows: if a job's XML configuration file contains an inline Groovy pipeline script that has an XML-escaped character right around the 8k character mark, it will result in mangled Groovy output. Without pasting our entire ~20k char config XML file, here's the offending section (the "character number" line is not part of the script but has been added here for clarity:

  return notifications.findAll { notification ->
    // character number 8112 ------------------^
    // Provide default values, since we can't parse the schema
    String onFailure = notification['on_failure'] ?: 'always'
    String onSuccess = notification['on_success'] ?: 'change'

The resulting Groovy script that we see in the actual job after asking Jenkins to process this config.xml file is:

  return notifications.findAll { notification ->
    // Provide default values, since we can't parse the schema
    String onFailure = notification['on_failure']
    String onFailure = notification['on_failure' ?: 'always' 

This is of course a syntax error and causes the pipeline to abort. Our "workaround" to this issue was changing the number of characters in the file and also removing the XML-encoded section to use the implicit it variable in Groovy as such:

  return notifications.findAll {
    // Provide default values, since we can't parse the schema
    String onFailure = it['on_failure'] ?: 'always'
    String onSuccess = it['on_success'] ?: 'change' 

We can also reproduce this bug by adding or deleting single characters in the first parts of the XML file outside of the pipeline script (for example, the job description). This changes the character that the XML-escape sequence starting with & starts on, and influences whether or not the bug occurs.

We have spent some hours trying to chase after this bug and don't know what exactly is causing it. We think that the bug is in hudson.XmlFile#unmarshal(java.lang.Object, boolean), and we think that it's due to the combination of the BufferedInputStream (which defines DEFAULT_BUFFER_SIZE to 8192) when used with XStream, but we aren't familiar enough with the Jenkins internal architecture to say for certain.