Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64471

warnings-ng tool for nunit

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We have tried to record results of nunit2 execution via warnings-ng plugin. Unfortunately we haven't found a ready to use warnings-ng tool for nunit (we are still using nunit2 and not the nunit3).
      So we have used Microsoft tool (mssql.exe) with the xsl-convertion taken from https://github.com/jenkinsci/nunit-plugin/tree/master/src/main/resources/hudson/plugins/nunit (and added the encoding "utf-8" parameter to generate junit-files, which can be consumed by xunit-plugin).
      The resulting files could be successfully integrated into the jenkins build reports using xunit-plugin:

      xunit ( tools: [JUnit(pattern: '.out\\*.Test.junit-result.xml', stopProcessingIfError: true)] )

      But we cannot see any issues in the warnings-ng plugin (using reportIssues as follows):
      recordIssues(
                          name: 'Unit Tests',
                          enabledForFailure: true,
                          aggregatingResults: false,
                          tools: [junitParser(id: 'unitTests', name: 'UnitTests', pattern: '.out\\*.Test.junit-result.xml', reportEncoding: 'UTF-8')]
                        )

        Attachments

          Activity

          Hide
          smorita Shin-ichi Morita added a comment -

          It seems to have a couple of issues:

          • NUnit is not supported by warnings-ng-plugin.
          • Reusing nunit-plugin's xsl seems not working.

          Regarding the former, please consult with wargings-ng-plugin.

          According to https://github.com/jenkinsci/warnings-ng-plugin/blob/master/doc/Documentation.md#creating-support-for-a-custom-tool, you can create a custom parser, although I'm not familiar with it.

          Regarding the latter, could you tell whether your nunit2 report is successfully processed by nunit-plugin or not?

           

          Regards,

          Show
          smorita Shin-ichi Morita added a comment - It seems to have a couple of issues: NUnit is not supported by warnings-ng-plugin. Reusing nunit-plugin's xsl seems not working. Regarding the former, please consult with wargings-ng-plugin. According to https://github.com/jenkinsci/warnings-ng-plugin/blob/master/doc/Documentation.md#creating-support-for-a-custom-tool,  you can create a custom parser, although I'm not familiar with it. Regarding the latter, could you tell whether your nunit2 report is successfully processed by nunit-plugin or not?   Regards,
          Hide
          drulli Ulli Hafner added a comment -

          The warnings plugin is not intended to show unit test results, so I removed the component from this issue.

          Show
          drulli Ulli Hafner added a comment - The warnings plugin is not intended to show unit test results, so I removed the component from this issue.
          Hide
          mrm927 Rashid added a comment -

          @Ulli Hafner, what is then the intent to provide junitParser in the warnings-ng plugin, if it doesn't consider failed and skipped tests?

          Our expectation was, that we could get some statistics about the failed / skipped tests via warnings-ng plugin.

          The nunit-plugin can of cause import the nunit-reports.
          Even xunit-plugin is able to import the manually converted "junit-reports", made of nunit-report as described in the description of the issue.
          Our issue was importing the results into warnings-ng

          Show
          mrm927 Rashid added a comment - @ Ulli Hafner , what is then the intent to provide junitParser in the warnings-ng plugin, if it doesn't consider failed and skipped tests? Our expectation was, that we could get some statistics about the failed / skipped tests via warnings-ng plugin. The nunit-plugin can of cause import the nunit-reports. Even xunit-plugin is able to import the manually converted "junit-reports", made of nunit-report as described in the description of the issue. Our issue was importing the results into warnings-ng
          Hide
          drulli Ulli Hafner added a comment -

          The intend was to get some features of the warnings plugin for free that were not available in the JUnit plugin (basically the JS chart and delta computation).
          (And another internal reason: I use the API to obtain the number of passed, skipped and failed tests in my autograding module.)

          What are you missing in the JUnit plugin?

          Show
          drulli Ulli Hafner added a comment - The intend was to get some features of the warnings plugin for free that were not available in the JUnit plugin (basically the JS chart and delta computation). (And another internal reason: I use the API to obtain the number of passed, skipped and failed tests in my autograding module.) What are you missing in the JUnit plugin?
          Hide
          tkachenko Konstantin Tkachenko added a comment -

          I've attached a sample build report from our jenkins. The build has 2 failed tests (imported from junit-reports via xunit plugin), but the warnings-ng plugin hasn't found any warnings using junitParser (for the same reports).

          That was exactly the reason why we tried to use warnings-ng plugin for our nunit-tests - we were trying to get chart and some statistics about skipped and failed tests.

          Show
          tkachenko Konstantin Tkachenko added a comment - I've attached a sample build report from our jenkins. The build has 2 failed tests (imported from junit-reports via xunit plugin), but the warnings-ng plugin hasn't found any warnings using junitParser (for the same reports). That was exactly the reason why we tried to use warnings-ng plugin for our nunit-tests - we were trying to get chart and some statistics about skipped and failed tests.
          Hide
          drulli Ulli Hafner added a comment -

          If your file is not in JUnit format it is clear that it does not work with the warnings plugin, I am using an external library to parse those files. Those things that are done in the xunit are not part of the warnings plugin parser.

          Show
          drulli Ulli Hafner added a comment - If your file is not in JUnit format it is clear that it does not work with the warnings plugin, I am using an external library to parse those files. Those things that are done in the xunit are not part of the warnings plugin parser.
          Hide
          multani Jonathan Ballet added a comment -

          I think we have a similar issue, using a file produced by JUnit itself: although it contains test cases and skipped tests, the warnings-ng plugin doesn't show anything.

          Our Jenkinsfile configuration:

          recordIssues(
            tools: [
              junitParser(pattern: '*/build/test-results/**/*.xml'),
          
              taskScanner(includePattern: '**/*.java', highTags: 'FIXME', normalTags: 'TODO', ignoreCase: true),
            ],
            aggregatingResults: true,
          )
          

          The JUnit XML file (cleaned up, but it's produced by JUnit):

          <testsuite name="com.example.xxx.contracts.ContractTests" tests="3" skipped="1" failures="0" errors="0" timestamp="2021-03-25T16:36:38" hostname="burrito" time="0.136">
          <properties/>
          <testcase name="shouldNotDoSomething" classname="com.example.xxx.contracts.ContractTests" time="0.064"/>
          <testcase name="expectedSomething" classname="com.example.xxx.contracts.ContractTests" time="0.071"/>
          <testcase name="getSomething" classname="com.example.xxx.contracts.ContractTests" time="0.0">
          <skipped/>
          </testcase>
          <system-out>
          2021-03-25 16:36:29.251 ERROR 1913 --- [ Test worker] o.h.metamodel.internal.MetadataContext : HHH015007: Illegal argument on static metamodel field injection : ...
          </system-out>
          <system-err></system-err>
          </testsuite>
          

          And the output of the JUnit warnings-ng plugin when it tries to parse the XML file:

          [Pipeline] recordIssues
          [JUnit] Searching for all files in '/workspace/app123' that match the pattern '*/build/test-results/**/*.xml'
          [JUnit] -> found 1202 files
          [JUnit] Successfully parsed file /workspace/app123/xxx/build/test-results/integrationTest/TEST-com.example.xxx.ContractVerifierTest.xml
          [JUnit] -> found 0 issues (skipped 0 duplicates)
          [JUnit] Successfully parsed file /workspace/app123/xxx/build/test-results/integrationTest/TEST-com.example.xxx.configuration.HibernateConfigTests.xml
          [JUnit] -> found 0 issues (skipped 0 duplicates)
          [JUnit] Successfully parsed file /workspace/app123/xxx/build/test-results/integrationTest/TEST-com.example.xxx.configuration.CachingConfigurationTests.xml
          [JUnit] -> found 0 issues (skipped 0 duplicates)
          [JUnit] Successfully parsed file /workspace/app123/xxx/build/test-results/integrationTest/TEST-com.example.xxx.contracts.ContractTests.xml
          [JUnit] -> found 0 issues (skipped 0 duplicates)
          [...]
          [JUnit] Skipping post processing
          [JUnit] No filter has been set, publishing all 0 issues
          [JUnit] Repository miner is not configured, skipping repository mining
          [Open Tasks Scanner] Searching for files in workspace '/workspace/app123' that match the include pattern '**/*.java' and exclude pattern ''
          [...]
          

          The [JUnit] part goes on for all the files with the same result: found 0 issues (skipped 0 duplicates).

          I (mistakenly) replaced the call to the

          junit testResults: '*/build/test-results/**/*.xml', allowEmptyResults: true

          with the call to recordIssues + junitParser, to realize only now that I was completely wrong about what to expect from the new call, and it's not clear what statistics it actually brings.

          Show
          multani Jonathan Ballet added a comment - I think we have a similar issue, using a file produced by JUnit itself: although it contains test cases and skipped tests, the warnings-ng plugin doesn't show anything. Our Jenkinsfile configuration: recordIssues( tools: [ junitParser(pattern: '*/build/test-results /**/ *.xml' ), taskScanner(includePattern: '**/*.java' , highTags: 'FIXME' , normalTags: 'TODO' , ignoreCase: true ), ], aggregatingResults: true , ) The JUnit XML file (cleaned up, but it's produced by JUnit): <testsuite name= "com.example.xxx.contracts.ContractTests" tests= "3" skipped= "1" failures= "0" errors= "0" timestamp= "2021-03-25T16:36:38" hostname= "burrito" time= "0.136" > <properties/> <testcase name= "shouldNotDoSomething" classname= "com.example.xxx.contracts.ContractTests" time= "0.064" /> <testcase name= "expectedSomething" classname= "com.example.xxx.contracts.ContractTests" time= "0.071" /> <testcase name= "getSomething" classname= "com.example.xxx.contracts.ContractTests" time= "0.0" > <skipped/> </testcase> <system-out> 2021-03-25 16:36:29.251 ERROR 1913 --- [ Test worker] o.h.metamodel.internal.MetadataContext : HHH015007: Illegal argument on static metamodel field injection : ... </system-out> <system-err> </system-err> </testsuite> And the output of the JUnit warnings-ng plugin when it tries to parse the XML file: [Pipeline] recordIssues [JUnit] Searching for all files in '/workspace/app123' that match the pattern '*/build/test-results /**/ *.xml' [JUnit] -> found 1202 files [JUnit] Successfully parsed file /workspace/app123/xxx/build/test-results/integrationTest/TEST-com.example.xxx.ContractVerifierTest.xml [JUnit] -> found 0 issues (skipped 0 duplicates) [JUnit] Successfully parsed file /workspace/app123/xxx/build/test-results/integrationTest/TEST-com.example.xxx.configuration.HibernateConfigTests.xml [JUnit] -> found 0 issues (skipped 0 duplicates) [JUnit] Successfully parsed file /workspace/app123/xxx/build/test-results/integrationTest/TEST-com.example.xxx.configuration.CachingConfigurationTests.xml [JUnit] -> found 0 issues (skipped 0 duplicates) [JUnit] Successfully parsed file /workspace/app123/xxx/build/test-results/integrationTest/TEST-com.example.xxx.contracts.ContractTests.xml [JUnit] -> found 0 issues (skipped 0 duplicates) [...] [JUnit] Skipping post processing [JUnit] No filter has been set, publishing all 0 issues [JUnit] Repository miner is not configured, skipping repository mining [Open Tasks Scanner] Searching for files in workspace '/workspace/app123' that match the include pattern '**/*.java' and exclude pattern '' [...] The [JUnit] part goes on for all the files with the same result: found 0 issues (skipped 0 duplicates) . I (mistakenly) replaced the call to the junit testResults: '*/build/test-results /**/ *.xml' , allowEmptyResults: true with the call to recordIssues + junitParser , to realize only now that I was completely wrong about what to expect from the new call, and it's not clear what statistics it actually brings.
          Hide
          drulli Ulli Hafner added a comment -

          It does not bring anything useful for your use case. The warnings plugin just collects the number of failed tests. Passed and skipped tests are just ignored.

          Show
          drulli Ulli Hafner added a comment - It does not bring anything useful for your use case. The warnings plugin just collects the number of failed tests. Passed and skipped tests are just ignored.

            People

            Assignee:
            smorita Shin-ichi Morita
            Reporter:
            mrm927 Rashid
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated: