Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-71697

Test Result Aggregator is always showing Total Tests as 1

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker

      • Test Result Aggregator is always showing Total Tests as 1 even when test count is > 1
      • We have done both the global and pipeline configuration needed for integrating Test Result Aggregator

          [JENKINS-71697] Test Result Aggregator is always showing Total Tests as 1

          Hi Piyush, can you please share a part from the JSON response that you'll find if you open in a browser the URL of the job and then add 

          /lastBuild/api/json?depth=1

          You should see a part of the JSON like the following : 

           

          Socrates Sidereas added a comment - Hi Piyush, can you please share a part from the JSON response that you'll find if you open in a browser the URL of the job and then add  /lastBuild/api/json?depth=1 You should see a part of the JSON like the following :   

          Piyush added a comment -

          Hi Socrates, Thanks for replying. Please find the required json response screenshot below:

          Piyush added a comment - Hi Socrates, Thanks for replying. Please find the required json response screenshot below:

          Piyush added a comment - - edited

          Hi Socrates,
          we have observed one more thing:
          When we provide values for parameters recipientsList and subject in testResultsAggregator step, the testResultsAggregator step fails with below error

           

          Generate Email Report
          Error Occurred: Couldn't connect to host, port: localhost, 25; timeout 60000Sent message successfully to : 
          Error Occurred : com.sun.mail.util.MailConnectException: Couldn't connect to host, port: localhost, 25; timeout 60000;
            nested exception is:
          	java.net.ConnectException: Connection refused com.sun.mail.util.MailConnectException: Couldn't connect to host, port: localhost, 25; timeout 60000;
            nested exception is:
          	java.net.ConnectException: Connection refused
          	at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2210)
          	at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:722)
          	at jakarta.mail.Service.connect(Service.java:342)
          	at jakarta.mail.Service.connect(Service.java:222)
          	at jakarta.mail.Service.connect(Service.java:171)
          	at jakarta.mail.Transport.send0(Transport.java:230)
          	at jakarta.mail.Transport.send(Transport.java:100)
          	at com.jenkins.testresultsaggregator.reporter.MailNotification.sendMessage(MailNotification.java:211)
          	at com.jenkins.testresultsaggregator.reporter.MailNotification.send(MailNotification.java:139)
          	at com.jenkins.testresultsaggregator.reporter.Reporter.publishResuts(Reporter.java:88)
          	at com.jenkins.testresultsaggregator.TestResultsAggregator.perform(TestResultsAggregator.java:204)
          	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:101)
          	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:71)
          	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          	at java.base/java.lang.Thread.run(Thread.java:833)
          Caused by: java.net.ConnectException: Connection refused
          	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
          	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
          	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
          	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597)
          	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
          	at java.base/java.net.Socket.connect(Socket.java:633)
          	at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:333)
          	at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:214)
          	at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2160)
          	... 18 more 

          But when I trigger the next build after removing parameters  recipientsList and subject in testResultsAggregator step, we get correct value for Total Tests and Success/Fail Tests, but these values are of the last build not the current one.
          For example, if we have executed 5 tests in the last build, out of which 3 passed and 2 failed.
          And in the current build we have run 10 tests, out of which 7 passed and  3 failed.
          On "Job Results Aggregated" page, we see the summary as follows:

          Total Tests : 5(+5)
          Failed Tests : 2(+2)
          Aborted Tests : 0
          Success Tests : 3(+3) 

          But, the expected summary is as follows:

          Total Tests : 10(+10)
          Failed Tests : 3(+3)
          Aborted Tests : 0
          Success Tests : 7(+7)  

           

          Piyush added a comment - - edited Hi Socrates, we have observed one more thing: When we provide values for parameters recipientsList and subject in testResultsAggregator step, the testResultsAggregator step fails with below error   Generate Email Report Error Occurred: Couldn't connect to host, port: localhost, 25; timeout 60000Sent message successfully to : Error Occurred : com.sun.mail.util.MailConnectException: Couldn't connect to host, port: localhost, 25; timeout 60000; nested exception is: java.net.ConnectException: Connection refused com.sun.mail.util.MailConnectException: Couldn't connect to host, port: localhost, 25; timeout 60000; nested exception is: java.net.ConnectException: Connection refused at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2210) at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:722) at jakarta.mail.Service.connect(Service.java:342) at jakarta.mail.Service.connect(Service.java:222) at jakarta.mail.Service.connect(Service.java:171) at jakarta.mail.Transport.send0(Transport.java:230) at jakarta.mail.Transport.send(Transport.java:100) at com.jenkins.testresultsaggregator.reporter.MailNotification.sendMessage(MailNotification.java:211) at com.jenkins.testresultsaggregator.reporter.MailNotification.send(MailNotification.java:139) at com.jenkins.testresultsaggregator.reporter.Reporter.publishResuts(Reporter.java:88) at com.jenkins.testresultsaggregator.TestResultsAggregator.perform(TestResultsAggregator.java:204) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:101) at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:71) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang. Thread .run( Thread .java:833) Caused by: java.net.ConnectException: Connection refused at java.base/sun.nio.ch.Net.pollConnect(Native Method) at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672) at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542) at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) at java.base/java.net.Socket.connect(Socket.java:633) at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:333) at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:214) at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2160) ... 18 more But when I trigger the next build after removing parameters   recipientsList and subject in testResultsAggregator step, we get correct value for Total Tests and Success/Fail Tests, but these values are of the last build not the current one. For example, if we have executed 5 tests in the last build, out of which 3 passed and 2 failed. And in the current build we have run 10 tests, out of which 7 passed and  3 failed. On "Job Results Aggregated" page, we see the summary as follows: Total Tests : 5(+5) Failed Tests : 2(+2) Aborted Tests : 0 Success Tests : 3(+3) But, the expected summary is as follows: Total Tests : 10(+10) Failed Tests : 3(+3) Aborted Tests : 0 Success Tests : 7(+7)  

          Socrates Sidereas added a comment - - edited

          This one is strange : 

          Why there are tests into a class named com,jenkins.testresultsaggregator. etc ?

          Probably has to do with the fact that the execution of the tests and the call to aggregator are running from the same groovy file and sharing the same workspace. 

          This is not the purpose of this plugin , meaning that since you are having only one job running tests why do you need the aggregator plugin ? 

           

          Socrates Sidereas added a comment - - edited This one is strange :  Why there are tests into a class named com,jenkins.testresultsaggregator. etc ? Probably has to do with the fact that the execution of the tests and the call to aggregator are running from the same groovy file and sharing the same workspace.  This is not the purpose of this plugin , meaning that since you are having only one job running tests why do you need the aggregator plugin ?   

          About the 
          Connection refused com.sun.mail.util.MailConnectException: Couldn't connect to host, port: localhost, 25; timeout 60000
          You should set on Global Jenkins configuration an SMTP mail server (This is something not written in Aggregator plugin page and I should write something)

          Socrates Sidereas added a comment - About the  Connection refused com.sun.mail.util.MailConnectException: Couldn't connect to host, port: localhost, 25; timeout 60000 You should set on Global Jenkins configuration an SMTP mail server (This is something not written in Aggregator plugin page and I should write something)

          As concerns the fact that you are seeing the previous test results , this has to do with this 

           

          please see https://github.com/jenkinsci/test-results-aggregator-plugin/blob/master/README_Pipeline.adoc 

          ignoreRunningJobs Ignore from report jobs with status RUNNING. Options true/false. If false then jobs are reported with status RUNNING. If true then jobs are reported with the previous status and results. Moreover an asterisk at the side of status declares that the job is still running.

          Socrates Sidereas added a comment - As concerns the fact that you are seeing the previous test results , this has to do with this    please see https://github.com/jenkinsci/test-results-aggregator-plugin/blob/master/README_Pipeline.adoc   ignoreRunningJobs Ignore from report jobs with status RUNNING. Options true/false. If false then jobs are reported with status RUNNING. If true then jobs are reported with the previous status and results. Moreover an asterisk at the side of status declares that the job is still running.

          I have pushed a fix for the first issue, even if this is not the way to use this plugin 

          For the 2nd issue, I have updated the documentation and throw a more appropriate error for this case.

          The 3rd issue is designed like that. For running jobs if "ignoreRunning" is true get the previous results, 

          and due to the fact that both junits and aggregator are running in the same job, then aggregator will always find itself as running.

          Socrates Sidereas added a comment - I have pushed a fix for the first issue, even if this is not the way to use this plugin  For the 2nd issue, I have updated the documentation and throw a more appropriate error for this case. The 3rd issue is designed like that. For running jobs if "ignoreRunning" is true get the previous results,  and due to the fact that both junits and aggregator are running in the same job, then aggregator will always find itself as running.

          Socrates Sidereas added a comment - - edited

          Release v1.2.15 will handle and ignore from final results the class com.jenkins.testresultsaggregator

          For the rest of the requests please read carefully the comments above.

          Socrates Sidereas added a comment - - edited Release v1.2.15 will handle and ignore from final results the class com.jenkins.testresultsaggregator For the rest of the requests please read carefully the comments above.

            socrates Socrates Sidereas
            pkcodes Piyush
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: