tibor17 Does SUREFIRE-1559 imply a breaking change in Maven Surefire?

> This should be fixed in Maven Surefire. If you use this XML and visualize the reports, pls adapt to the fix.

What is ETA of the fix? IIUC there is only an issue report without confirmed ETA or action plan (e.g. major version number bump, etc.). 

Whatever the fix is needed on the Jenkins side, we will need to support both previous and new formats

That's the question because for me it breaks the form of report in something (only 4 elements I mentioned above) but the rest remains intact. If you do not mind in the stack trace system-out/err, then you would not see any difference.
One way or another it was badly designed XSD/XML which could not be parsed and could not be valid. The fact that Jenkins plugins are parsing this XML without any problem means that they do not mind in the elements of (rerunError, rerunFailure, flakyFailure, flakyError). If they were used in Jenkins plugins and if the XML simple value was observed then it means that your users sometimes observed weird text in stack trace:
<system-out>my console log...</system-out>

These 4 elements are relatively new. Two were missing in XSD but present in XML.
As I said bard work on our side.

oleg_nenashev
It seems you agree since you have not reply. I am going to push it to master in ASF Maven. Any objections?

> tibor17The fact that Jenkins plugins are parsing this XML without any problem means that they do not mind in the elements of (rerunError, rerunFailure, flakyFailure, flakyError)

Not true at all, plugins like xunit use the XSD to validate the XML (and with 2.22.1 is broken, files are skipped). It's not clear for me what is the gain of the stacktrace element and make rerun elements different from failure and error.

EDIT: you missed to list the component that supports flaky elements... xunit and flaky-test-handler and for this reason you did get a response in time because no one of major interested people was informed. I'm here now because the new version had broken our organisation pipelines and I got this link from SUREFIRE-1559.

I have tried to involve as many plugin as possible in Jenkins and Travis. I saw XUnit related to .Net and not Java.
Sorry for that. Mistakes happen the same as developers make mistakes with the XML design regarding elements /flaky and rerun.
This is the chance to improve the s/w. Btw we plan to release several release candidates for version 3.0.0 and we will break backwards compatibility in API (not in XML) this time and it is because we have no choice after 15 years. There is time and milestone where we have to say that we have to break something but hopefully not everything. We are not going to create a new plugin but after 15 years it is reasonable.
Nikolas, the XML and old XSD was broken.
Similar changes happened in Surefire but it was not this schema. It was also with standard output stream which was broken by user's libraries and the error was uncovered in logs however the users do not like it but they had to accept that we also do not want to be responsible for "their" mistakes in their project or used external libraries.

The main issues is than you had introduced these changes (not back compatible with version 2.22.0) in version 2.22.1 not in a major release. The people will not expect that.

version 2.22.0

    <rerunFailure message="expected:&lt;2&gt; but was:&lt;3&gt;" type="junit.framework.AssertionFailedError">junit.framework.AssertionFailedError: expected:&lt;2&gt; but was:&lt;3&gt;
	at junit.framework.Assert.fail(Assert.java:57)
	at junit.framework.Assert.failNotEquals(Assert.java:329)
	at junit.framework.Assert.assertEquals(Assert.java:78)
	at junit.framework.Assert.assertEquals(Assert.java:234)
	at junit.framework.Assert.assertEquals(Assert.java:241)
	at junit.framework.TestCase.assertEquals(TestCase.java:409)
        ...
</rerunFailure>

version 2.22.1

    <rerunFailure type="java.lang.AssertionError">
      <stackTrace>java.lang.AssertionError
	at com.acme.faulttolerance.internal.test.FooTest.foo(FooTest.java:10)
</stackTrace>
    </rerunFailure>

the difference are:

• 1 new required (not optional) element
• 1 row stacktrace against full stracktrace
• no XML text allowed

Question for you tibor17 At this point why do not add an attribute to mark the differences between flaky failures and failure also in the XML (not only in console) ? With the XML you have to parse whole file and make logic on top to calculate real failures.

nfalco
Hi Nicolas,

Not sure what you mean by
At this point why do not add an attribute to mark the differences between flaky failures and failure also in the XML

With the XML you have to parse whole file and make logic on top to calculate real failures.
As always because this is XML and the stacktrace is not big like kilobytes.

Not sure why you validate XML but anyway. Parsing the XML and displaying the statistics is what has to be done by the plugins. In this case we have to find a way how you detect version of the XSD. We have not had any version in the XSD which would be easier to have it in past. What alternatives are you facing in order to detect which XSD is actual?
We can fix this as following.
I will deploy the old XSD to the old schema location URL=https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd and I will deploy a new schema location for the current status 2.22.1 with another name of XSD file and new schema location which would have version 2.22.1.
Do you agree or you see another solution?

If the change present in 3.0 version, we would have to wait quite long time and there would be a risk that the rerun feature as well as flaky failures are heavily used. One way or another these Jenkins plugins do not care about Maven plugin versions because they only read the XML report as the outcome of the Maven plugins and Jenkins plugins do not use to check the version of surefire in POM. Not that easy for them. The fact that nobody claimed that the stacktrace contained std-out in one string of XML Value means that these features are not much used or the users do not print logs in tested logic on the console, but the frequency of use of these features is more probable.

nfalco
If there are no objections I will introduce XML version in the XSD schema and we will proceed with plugin version 3.0.0-M1.
We will have multiple versions of this XML in web deployment and the old versions of plugin would be validated against the old schema.
I will open a pull request where you can have a look if you like and this way you will have the result under your control.
Is this fair?

For me it's ok

nfalco
We are near to Surefire 3.0.0-M2 release. I have made a fix for this issue. There is a new XSD version 3.0 and documentation explaining the decisions, see index.apt.vm. I have pushed a branch SUREFIRE-1590 on GitHub. Please review it. Thx.

It seems ok

nfalco
We are in release Vote for the version 3.0.0-M2
Please join us in the Vote:
https://lists.apache.org/thread.html/d44c7a8198e4f4d214211e40909249bcca67bdabd7d476731f2e1665@%3Cdev.maven.apache.org%3E

nfalco
The version 3.0.0-M2 is available in Maven Central.
Can you please adapt your plugin xUnit with the path of XSD?
It has a new location
https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report-3.0.xsd
and old location for legacy versions
https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd
I hope this would work for you. Sorry.

The XSD embedded in this plugin is not a copy of surefure XSD, it have to support old and new version and ant junit format version. So it is not an issue if the URl is changed but YES I have to check if latest version 3.0.0-M2 cause some issue to the actual plugin. Within this weekend I will perform the needed verification. Thank you.

tibor17 maybe it's the case also to open an issue to the flaky-test-handler-plugin that is the plugin that decorate JUnit reports with flaky tests in the XML.

If you want I could open for you.

nfalco
It's up to you. If you find an issue, I am all the ear.
But I do not know if your plugin can recognize the XSD version. I hope it can.