Copied from https://github.com/jenkinsci/jenkins/issues/27168 with special thanks to https://github.com/byn-beno-zurman

Reproduction steps

• Create a JUnit 5 test that fails ~50% of the time:

  import org.junit.jupiter.api.Test;
  import static org.junit.jupiter.api.Assertions.assertTrue;
  
  class RandomlyFailingTest {
      @Test
      void randomlyFails() {
          double value = Math.random();
          System.out.println("chosen value: " + value);
          assertTrue(value > 0.5);
      }
  }
  

• Enable in-build retries in Maven Surefire:

  <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
          <rerunFailingTestsCount>10</rerunFailingTestsCount>
      </configuration>
  </plugin>
  

With 10 retries the test ends flaky (fail → pass) in ~50% of builds, passes clean in ~50%, and fails all attempts in ~0.05% — so a handful of builds produces both affected and unaffected records.

• Create a freestyle job:
  • Build step: mvn test (or any Maven/shell step running the module)
  • Post-build action: Publish JUnit test result report, pattern **/target/surefire-reports/**/*.xml
    • Additional test report features: Publish JUnit flaky stats (JUnitFlakyTestDataPublisher)
  • (Optional, matches our setup: also add the top-level Publish JUnit flaky test reports / JUnitFlakyResultArchiver — not required to reproduce.)
• Run ~10 builds. Note which builds retried the test — visible in the console log:

  [ERROR]   Run 1: RandomlyFailingTest.randomlyFails:12 expected: <true> but was: <false>
  [INFO]   Run 2: PASS
  

or via <build>/testReport/.../randomlyFails/api/json → non-empty flakyFailures (retried, then passed) or rerunFailures (failed all attempts). Builds where value > 0.5 on the first attempt have neither.

At this point all builds display correctly in the UI — the Run objects are still in controller memory. On disk, however, each build with a retried test has a multi-MB build.xml, versus a few hundred KB without the flaky publisher.

• Force a lazy reload of the build records. Either:
  • edit the job config and save (a no-op change suffices; a Job DSL seed-job re-run also triggers it), or
  • restart Jenkins.
• Query build metadata:

  GET <job>/api/json?tree=builds[number,result,timestamp,duration]
  

A Jenkins core maintainer independently built and confirmed a standalone repro using these same steps — see the attached gh-core-27009.zip on jenkinsci/jenkins#27009, a self-contained Docker/Configuration-as-Code setup that deterministically reproduces the corruption after a restart.

Expected Results

Every build in which the test was retried at least once (flaky pass, or failed all retry attempts) should reload from build.xml with its correct result, timestamp, and duration — the same values shown immediately after the build ran and the same values recorded in the on-disk build.xml, which is well-formed and semantically complete (verified with a SAX parser; the true result is also visible at the end of the build's console log, e.g. Finished: UNSTABLE).

Actual Results

Every build in which the test was retried at least once now returns:

{"number": N, "result": "ABORTED", "timestamp": 0, "duration": 0}

and renders in the UI as ABORTED, "Started 56 yr ago", "Took 0 ms". Builds where the test passed on the first attempt (no retry) reload correctly. In an 11-build test run, the split was exact: 3 builds with no retry loaded fine, 8 builds that retried the test were broken.

This is a read-side deserialization failure, not write-time corruption. JUnitFlakyTestDataPublisher.contributeTestData() re-parses all surefire XML reports from disk independently of the main JUnit result parsing (see the FlakyTestResult(TestResult) constructor, which takes only file paths from the JUnit result and re-parses them itself), and serializes this into TestResultAction's <testData> element in build.xml. For any build with a retried test, the resulting build.xml ends up serializing every suite twice, in two different shapes: once as the plugin's independently re-parsed copy, and once as a plain hudson.tasks.junit.SuiteResult/CaseResult graph reachable via object references from the rerun/flaky entries (FlakyCaseResult → <parent>). Same underlying file, two entries, different computed durations.

On the next XStream load, RobustReflectionConverter fails on this graph. One observed manifestation is an explicit ConversionException from XStream2$BlacklistedTypesConverter refusing to unmarshal java.util.Map$Entry inside testCaseFlakyInfoMap (a JEP-200 class-filtering rejection, since java.util.Map$Entry has no codebase location and isn't in the default whitelist):

com.thoughtworks.xstream.converters.ConversionException: Refusing to unmarshal entry for security reasons; see https://www.jenkins.io/redirect/class-filter/
---- Debugging information ----
message             : Refusing to unmarshal entry for security reasons; see https://www.jenkins.io/redirect/class-filter/
class               : java.util.Map$Entry
required-type       : java.util.Map$Entry
converter-type      : hudson.util.XStream2$BlacklistedTypesConverter
path                : /build/actions/hudson.tasks.junit.TestResultAction/testData/com.google.jenkins.flakyTestHandler.plugin.JUnitFlakyTestData/testCaseFlakyInfoMap/entry[9749]

Either way, the load silently drops critical Run fields and the record becomes the <broken data JENKINS-45892> placeholder. Loading the file manually via Run.XSTREAM2.fromXML(buildXml) reproduces the failure deterministically, confirming the on-disk data is intact but the object graph is unreconstructable — so an in-memory patch/repair script cannot help once a build has already been reloaded once, and the correct values cannot be recovered.

Affected builds never appear in the Old Data Monitor, so there is no built-in way to detect which historical builds are affected.

Build/test duration and suite size are not the trigger. A sub-second flaky test in a 2-6 minute build reproduces the corruption on every build where it retries, while much larger builds (~10 hours, ~9,500 tests) are unaffected as long as no test in them was retried. The only variable that matters is whether a retried test's data was embedded in the build. Builds appear to display correctly for a while after they run only because the in-memory Run object hasn't yet been evicted/reloaded — the first lazy reload (config save, seed job rerun, or restart) is what exposes the bug.

Anything else?

Originally filed against Jenkins core as jenkinsci/jenkins#27009. Root cause was narrowed down there with help from core maintainers, who confirmed the issue is specific to this plugin's testData serialization rather than core, and asked that it be reported here.

Verified workaround (not a fix): removing JUnitFlakyTestDataPublisher while keeping JUnitFlakyResultArchiver resolves it completely. In our Job DSL:

testDataPublishers {
    publishFlakyTestsReport()
}

— removing just this block (keeping jUnitFlakyResultArchiver()) dropped build.xml size from 12 MB to 8.2 KB on our smaller suite, and confirmed a forced cache purge + reload loads the record correctly. The job-level Flaky History table (historyAggregate, fed by JUnitFlakyResultArchiver's separate junitFlakyStatsResult.xml) keeps working; only the per-case rerun detail in individual build test reports is lost. Confirmed at production scale: after removing this from three branches' freestyle Job DSL, overnight ~10-hour, ~9,500-test builds have produced 9.8-12.7 KB build.xml files that survive reload without issue, and no new broken builds have occurred since.

Given how common rerunFailingTestsCount-style retries are in Maven Surefire configs, this likely affects many installations using JUnitFlakyTestDataPublisher, with no visibility into the damage until someone goes looking at a specific build's history.

Are you interested in contributing a fix?

I don't think I'm capable to do this...