Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45761

Developer wants to see cucumber test results in Blue Ocean

      When running the Cucumber Test Results plugin in a pipeline, the test results are bundled into a section called "Cucumber Test Results" whereas the rest of the JUnit test results are placed into the "Test Result" section.

      This means that when the Blue Ocean view looks at test results, it doesn't include the Cucumber tests, just the xUnit ones. Given that BlueOcean doesn't fail individual stages in this version, in order to see if a test fails I have to look through the console logs or at the top level reports. If the JUnit tests aren't archived, the Cucumber ones are not published to the BlueOcean set.

      The plugin should publish reports to the standard test result section. Note, if I add xUnit tests they are published correctly

          [JENKINS-45761] Developer wants to see cucumber test results in Blue Ocean

          James Nord added a comment - - edited

          The plugin should publish reports to the standard test result section. Note, if I add xUnit tests they are published correctly

          If this was done you would loose all the extra functionality you would have such as the details of what step(s) fails in the scenario in the standard classic view.

          What I think you are (or should be asking for is that)

          1) Blue Ocean should look for all test result actions and render the test output (unless there is a BO extension handling the more specific type of tests).
          2) that there should be a Blue Ocean rendering for Cucumber reports (when the drop down is called)

          jdumay / michaelneale are there appropriate extension points for this?

          James Nord added a comment - - edited The plugin should publish reports to the standard test result section. Note, if I add xUnit tests they are published correctly If this was done you would loose all the extra functionality you would have such as the details of what step(s) fails in the scenario in the standard classic view. What I think you are (or should be asking for is that) 1) Blue Ocean should look for all test result actions and render the test output (unless there is a BO extension handling the more specific type of tests). 2) that there should be a Blue Ocean rendering for Cucumber reports (when the drop down is called) jdumay / michaelneale are there appropriate extension points for this?

          Michael Neale added a comment -

          Do people look at cucumber reports via the Jenkins (classic) way or more like a generated HTML report (years ago I remember cucumber had some beautiful ways to show things). If the latter, there was a plan for HTML publishing. 

          If former, I would have to ask kzantow as he worked on the test tab feature (I believe there is an extension point, but it may be too high level to be useful). I am not sure if the test tab itself has an extension point that could cope with cucumber reports though. 

           

          Definitely the cucumber plugin could be enhanced to add to blue ocean somewhere, given enough work/time. 

           

          Michael Neale added a comment - Do people look at cucumber reports via the Jenkins (classic) way or more like a generated HTML report (years ago I remember cucumber had some beautiful ways to show things). If the latter, there was a plan for HTML publishing.  If former, I would have to ask kzantow as he worked on the test tab feature (I believe there is an extension point, but it may be too high level to be useful). I am not sure if the test tab itself has an extension point that could cope with cucumber reports though.    Definitely the cucumber plugin could be enhanced to add to blue ocean somewhere, given enough work/time.   

          James Nord added a comment -

          This cucumber plugin uses a classic way of showing reports, but they are not your normal junit reports.
          The plugin does not generate html at parse time (there is a plugin that does that however it misses features like trends and failing since)
          See the second screen shot at https://wiki.jenkins.io/display/JENKINS/Cucumber+Test+Result+Plugin

          James Nord added a comment - This cucumber plugin uses a classic way of showing reports, but they are not your normal junit reports. The plugin does not generate html at parse time (there is a plugin that does that however it misses features like trends and failing since) See the second screen shot at https://wiki.jenkins.io/display/JENKINS/Cucumber+Test+Result+Plugin

          James Dumay added a comment - - edited

          teilo there is a Blue Ocean extension point - BlueTestResultFactory and BlueTestResult - that is used for providing data to the test results.

          We had tried using test result actions directly however we ran into issues where:

          • There are too many different ways to call a TestResultAction. Standard methods that should be implemented return null or empty collections. Many define similarly named methods on their implementations and call these directly from Jelly (the abstraction is a lie).
          • The order that methods are called in the Jelly file determines if there is data to be found when calling these methods at all (lots of lazy loading in implementations for what I assume is performance reasons).
          • Some plugins use TestResultAction in the test package and some use it from the junit package or a mixture of both (the JUnit plugin does this). It is a massive rats nest of polymorphism which makes it hard to determine what objects we are getting from untyped collections.
          • TestResultAction does not support any sort of pagination. We had originally serialised all actions and sent them to the client for rendering but we got reports of 40mb test result pages. Abstraction here was necessary to support proper serialisation and pagination, which Jenkins core types (such as actions) are not appropriate for on the web.

          Most plugins can use the extension point in Blue Ocean 1.1 with very minimal dependencies (blueocean-rest and blueocean-commons only from memory). If we want to support other test plugins, simply extend the classes I mentioned and release the plugin. Then everything works swell and avoids all the pitfalls of using TestResultAction.

          James Dumay added a comment - - edited teilo there is a Blue Ocean extension point - BlueTestResultFactory and BlueTestResult - that is used for providing data to the test results. We had tried using test result actions directly however we ran into issues where: There are too many different ways to call a TestResultAction. Standard methods that should be implemented return null or empty collections. Many define similarly named methods on their implementations and call these directly from Jelly (the abstraction is a lie). The order that methods are called in the Jelly file determines if there is data to be found when calling these methods at all (lots of lazy loading in implementations for what I assume is performance reasons). Some plugins use TestResultAction in the test package and some use it from the junit package or a mixture of both (the JUnit plugin does this). It is a massive rats nest of polymorphism which makes it hard to determine what objects we are getting from untyped collections. TestResultAction does not support any sort of pagination. We had originally serialised all actions and sent them to the client for rendering but we got reports of 40mb test result pages. Abstraction here was necessary to support proper serialisation and pagination, which Jenkins core types (such as actions) are not appropriate for on the web. Most plugins can use the extension point in Blue Ocean 1.1 with very minimal dependencies (blueocean-rest and blueocean-commons only from memory). If we want to support other test plugins, simply extend the classes I mentioned and release the plugin. Then everything works swell and avoids all the pitfalls of using TestResultAction.

          James Dumay added a comment -

          I remember one plugin I tried half the information was missing from it because they had used different methods to call from their jelly. I would say that its better to show no reports, than reports that look broken.

          James Dumay added a comment - I remember one plugin I tried half the information was missing from it because they had used different methods to call from their jelly. I would say that its better to show no reports, than reports that look broken.

          James Nord added a comment -

          jamesdumay the abstraction should be better since JENKINS-19898.

          Plugins should not mix the junit and the tasks.test code - junit should use the junit and other non junit extensions should use the tasks.tests, mixing them would be a bug in the plugin (I think they are now separate enough to abstract) - did you file bugs for this?

          Looking at the code BlueTestResult is not rich enough for cucumber reporting, (it really is a lowest common denominator) - unless you want a sucky experience - so I would also need to extend the front end - what js extension point exists for that?

          James Nord added a comment - jamesdumay the abstraction should be better since JENKINS-19898 . Plugins should not mix the junit and the tasks.test code - junit should use the junit and other non junit extensions should use the tasks.tests, mixing them would be a bug in the plugin (I think they are now separate enough to abstract) - did you file bugs for this? Looking at the code BlueTestResult is not rich enough for cucumber reporting, (it really is a lowest common denominator) - unless you want a sucky experience - so I would also need to extend the front end - what js extension point exists for that?

          James Dumay added a comment -

          teilo I did not file bugs for those.

          There are no extension points on the frontend for that yet. What data makes the cucumber tests richer than JUnit?

          James Dumay added a comment - teilo I did not file bugs for those. There are no extension points on the frontend for that yet. What data makes the cucumber tests richer than JUnit?

          James Nord added a comment -

          jamesdumay  a "test" in Junit is equivalent to a "Scenario" in Cucumber.

          However unlike a junit test which can have multiple "assertThat(...)" statements and it only records failures,  the scenario will record all the "steps" in the scenario.  the individual steps can be skipped/passed/failed/unimplemented  and a step that fails "there should be 3 books in the library" make no sense unless viewed with the entire scenario as the same step can be called multiple times and there are also the "Given" that setup the tests that should be recorded (bscially it is Behavioural testing rather than Unit Testing and as such the Scenario has much more detail that is important to understanding the failure).

           

          I suggest you look at the screenshots for the plugin to get a visual overview, or check the https://cucumber.io/ web site.

          James Nord added a comment - jamesdumay  a "test" in Junit is equivalent to a "Scenario" in Cucumber. However unlike a junit test which can have multiple "assertThat(...)" statements and it only records failures,  the scenario will record all the "steps" in the scenario.  the individual steps can be skipped/passed/failed/unimplemented  and a step that fails "there should be 3 books in the library" make no sense unless viewed with the entire scenario as the same step can be called multiple times and there are also the "Given" that setup the tests that should be recorded (bscially it is Behavioural testing rather than Unit Testing and as such the Scenario has much more detail that is important to understanding the failure).   I suggest you look at the screenshots for the plugin to get a visual overview, or check the https://cucumber.io/  web site.

            Unassigned Unassigned
            steveworkman Steve Workman
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: