Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43892

Developer would like to see all successful test results

    • Icon: New Feature New Feature
    • Resolution: Fixed
    • Icon: Minor Minor
    • blueocean-plugin
    • None
    • Jenkins ver. 2.46.2 (Docker-based)
    • Blue Ocean 1.5 - beta 2

      In many cases people want to see the full set of tests, including all passing tests. 

      Scope

      • Add a new paginatable section for "Passed" tests. Section should be below all other test sections.
      • Functional and unit tests to cover passed tests section.

      Original request
      In the classic version of Jenkins we grant access to our customers the test output. It's supposed to be some kind of proof that we implemented their feature requests. Furthermore, this is a possibility for them to verify that requested behaviour works as expected/specified in the requirements. In the classic verison one can always access published test results and also enter the output being produced in the tests. In blue ocean I only get a message that all tests succeeded when entering the "Test" tab (see attached screenshot).

      It would give value if one could access the tests and also enter the test output in any case. If it would be an Extension point one could provide their own implementation.

          [JENKINS-43892] Developer would like to see all successful test results

          James Dumay added a comment -

          janr thanks for filing this feature request.

          Unfortunately there is not an easy fix without a redesign. Blue Ocean has to cope with Pipelines range from having a few tests to hundreds of thousands of tests which means cannot display all successful tests and have it work for all users.

          We could also look at alternative ways to get you this data. For example, if your client is looking for proof of tests we could allow you to export the tests to a Excel or CSV file rather than displaying every test on this screen.

          What do you think?

          James Dumay added a comment - janr thanks for filing this feature request. Unfortunately there is not an easy fix without a redesign. Blue Ocean has to cope with Pipelines range from having a few tests to hundreds of thousands of tests which means cannot display all successful tests and have it work for all users. We could also look at alternative ways to get you this data. For example, if your client is looking for proof of tests we could allow you to export the tests to a Excel or CSV file rather than displaying every test on this screen. What do you think?

          Jan added a comment -

          Hi James, thanks for your feedback. I'm afraid that providing especially the log output as CSV is not appropriate since tests might produce plenty of Information. What do you think?

          Jan added a comment - Hi James, thanks for your feedback. I'm afraid that providing especially the log output as CSV is not appropriate since tests might produce plenty of Information. What do you think?

          James Dumay added a comment -

          janr is the log output for successful tests relevant to your use case?

          James Dumay added a comment - janr is the log output for successful tests relevant to your use case?

          Jan added a comment -

          Hi James. Yes, it is very important for me and my clients because most often it reflects some kind of protocol to which the customer has access. I saw that you changed the title of this issue, unfortunately this isn't reflected in the title anymore.

          Jan added a comment - Hi James. Yes, it is very important for me and my clients because most often it reflects some kind of protocol to which the customer has access. I saw that you changed the title of this issue, unfortunately this isn't reflected in the title anymore.

          on this request.  While I understand that the BlueOcean UI is quite different from the classic UI,

          it is important to display the results of tests and a finer granularity than: "TESTS PASS" or "TESTS FAIL".

          I use the JUnit plugin quite heavily to publish the results of JUnit style tests.

          If there was a way to plug in the output of the JUnit plugin (and other similar plugins which publish test results) into the Blue Ocean UI, that would be really nice.

          Even if there was a link from the Blue Ocean UI

          to the classic UI's JUnit plugin test results, that would be a step in the right direction.

          Craig Rodrigues added a comment - on this request.  While I understand that the BlueOcean UI is quite different from the classic UI, it is important to display the results of tests and a finer granularity than: "TESTS PASS" or "TESTS FAIL". I use the JUnit plugin quite heavily to publish the results of JUnit style tests. If there was a way to plug in the output of the JUnit plugin (and other similar plugins which publish test results) into the Blue Ocean UI, that would be really nice. Even if there was a link from the Blue Ocean UI to the classic UI's JUnit plugin test results, that would be a step in the right direction.

          Brett Higgins added a comment -

          Don't you already have this problem for failing tests as well? What if a build generates a Junit test result with hundreds of thousands of failing tests? How will Blue Ocean handle it?

          If the current answer is that the page becomes sluggish or otherwise fails to deliver the list of results, I could see the case for not doing this for successful test results, since the scenario I described is much less common than having a huge number of passing tests. However, if the current answer is more along the lines of "we cap the number of results returned to a fixed amount," it seems doable to apply the same policy to successful test results. If the cap is fairly high, maybe most projects would be able to see their successful test result details, and the x% at the top with hundreds of thousands of tests would at least not have their browser hang (or whatever other negative experience).

          Brett Higgins added a comment - Don't you already have this problem for failing tests as well? What if a build generates a Junit test result with hundreds of thousands of failing tests? How will Blue Ocean handle it? If the current answer is that the page becomes sluggish or otherwise fails to deliver the list of results, I could see the case for not doing this for successful test results, since the scenario I described is much less common than having a huge number of passing tests. However, if the current answer is more along the lines of "we cap the number of results returned to a fixed amount," it seems doable to apply the same policy to successful test results. If the cap is fairly high, maybe most projects would be able to see their successful test result details, and the x% at the top with hundreds of thousands of tests would at least not have their browser hang (or whatever other negative experience).

          Brett Higgins added a comment -

          I'm particularly interested in seeing timing information for passing tests, so as to aid me in targeted optimizations.

          Brett Higgins added a comment - I'm particularly interested in seeing timing information for passing tests, so as to aid me in targeted optimizations.

          Costin Caraivan added a comment - - edited

          I'd like to add a +1 for this. Storing this somewhere is very, very useful, especially for large test suites. Being able to notice how the slow down of a test suite actually happens, in detail, is quite important.

          Costin Caraivan added a comment - - edited I'd like to add a +1 for this. Storing this somewhere is very, very useful, especially for large test suites. Being able to notice how the slow down of a test suite actually happens, in detail, is quite important.

          James Dumay added a comment -

          ccaraivan we plan on adding a "Trends" tab to the Pipeline so that you can see duration trends for all sorts of things. Perhaps this is where we could show if your tests are taking longer to run etc?

          James Dumay added a comment - ccaraivan we plan on adding a "Trends" tab to the Pipeline so that you can see duration trends for all sorts of things. Perhaps this is where we could show if your tests are taking longer to run etc?

          James Dumay added a comment -

          brettdh what timing information do you specifically need?

          James Dumay added a comment - brettdh what timing information do you specifically need?

          jamesdumay That could help, but they would need to have a break down/drill down. For example for Java we would need to be able to see timings/trends at package level and test level. If Jenkins core can't provide them at least there should be hooks for individual plugins to add stuff there, for example for Junit categories and such.

          Costin Caraivan added a comment - jamesdumay That could help, but they would need to have a break down/drill down. For example for Java we would need to be able to see timings/trends at package level and test level. If Jenkins core can't provide them at least there should be hooks for individual plugins to add stuff there, for example for Junit categories and such.

          Brett Higgins added a comment -

          jamesdumay The same timing information that's available in test results outside Blue Ocean: the execution time of each test, as reported in the JUnit XML file. Blue Ocean currently reports this for failing tests, but not for passing tests (since it reports nothing for passing tests).

          Brett Higgins added a comment - jamesdumay The same timing information that's available in test results outside Blue Ocean: the execution time of each test, as reported in the JUnit XML file. Blue Ocean currently reports this for failing tests, but not for passing tests (since it reports nothing for passing tests).

          James Dumay added a comment -

          Successful test display is shipping in the next release.

          James Dumay added a comment - Successful test display is shipping in the next release.

          jamesdumay Cool! Somewhat related to this - to not slow down page loading, does this only load a partial list at first, with the possibility to load the full list, if wanted? We have thousands of tests.

          Costin Caraivan added a comment - jamesdumay Cool! Somewhat related to this - to not slow down page loading, does this only load a partial list at first, with the possibility to load the full list, if wanted? We have thousands of tests.

          This looks cool - In order to see this - is this coming from a junit generated file?

          Paul Dragoonis added a comment - This looks cool - In order to see this - is this coming from a junit generated file?

          Craig Rodrigues added a comment - - edited

          michaelneale jamesdumay undera

          I am a heavy user of the Performance Plugin https://wiki.jenkins.io/display/JENKINS/Performance+Plugin

           

          Right now, the Performance Plugin creates a new link for each job such as:

          [https://jenkinshost/job/jobName/247/performance/

          T|https://jenkinshost/job/jobName/performance/]here are screenshots of what the Performance Plugin produces here: https://wiki.jenkins.io/display/JENKINS/Performance+Plugin?focusedCommentId=135470852#comment-135470852

           

          At this link there are graphs and tables listed which display the performance results.

          Is it possible to view the test results of the Performance Plugin in the BlueOcean UI?

           

          This would be extremely useful.

          Craig Rodrigues added a comment - - edited michaelneale jamesdumay undera I am a heavy user of the Performance Plugin https://wiki.jenkins.io/display/JENKINS/Performance+Plugin   Right now, the Performance Plugin creates a new link for each job such as: [https://jenkinshost/job/jobName/247/performance/ T|https://jenkinshost/job/jobName/performance/]here are screenshots of what the Performance Plugin produces here:  https://wiki.jenkins.io/display/JENKINS/Performance+Plugin?focusedCommentId=135470852#comment-135470852   At this link there are graphs and tables listed which display the performance results. Is it possible to view the test results of the Performance Plugin in the BlueOcean UI?   This would be extremely useful.

          Michael Neale added a comment -

          dragoonis yes - any junit/xunit type xml report file that is archived by the test archiver (usually a step like: junit 'path/to/stuff') will work with this. 

          hey rodrigc interesting. That is worthy of a fresh ticket. Perhaps you could team up with imeredith who was looking at ways to add data driven extension points so that other plugins could add links to the performance report pages. Would it be enough that the results screen (somewhere...) has links to /performance to start with? (ie no new screens, just links?)

           

           

          Michael Neale added a comment - dragoonis yes - any junit/xunit type xml report file that is archived by the test archiver (usually a step like: junit 'path/to/stuff') will work with this.  hey rodrigc interesting. That is worthy of a fresh ticket. Perhaps you could team up with imeredith who was looking at ways to add data driven extension points so that other plugins could add links to the performance report pages. Would it be enough that the results screen (somewhere...) has links to /performance to start with? (ie no new screens, just links?)    

          Craig Rodrigues added a comment - michaelneale I filed JENKINS-50589

            imeredith Ivan Meredith
            janr Jan
            Votes:
            4 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: