Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37739

Support multiple output.xml's to support parallel test executions in pipeline

    XMLWordPrintable

Details

    • Improvement
    • Status: Closed (View Workflow)
    • Trivial
    • Resolution: Won't Do
    • robot-plugin
    • None

    Description

      I am building running my robot framework test case parallel in Jenkins pipeline. However the Robot Test Summary generated post build is not correct, it displaying the count/summary of last finished in parallelism, however it should show the count of individual job. It was overwriting the report also earlier but that I saved as different name & it worked fine, but still the Robot Test Summary is problem(please not my both suite below have different number of test cases).

      suite1:{
      try

      { mvn -s /settings.xml install robotframework:run -Dexcludes=inprogress -Dsuites=test1 -DLogFileName=suite1_log.html -DoutputFileName=suite1_output.xml -DreportFileName=suite1_report.html' currentBuild.result = 'SUCCESS' }

      catch(any)

      { currentBuild.result = 'FAILURE' throw any }

      finally

      { step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite1_log.html', outputFileName: 'suite1_output.xml', reportFileName: 'suite1_report.html']) }

      },
      suite2:{
      try

      { mvn -s /settings.xml robotframework:run -Dexcludes=inprogress -Dsuites=test2 -DLogFileName=suite2_log.html -DoutputFileName=suite2_output.xml -DreportFileName=suite2_report.html' currentBuild.result = 'SUCCESS' }

      catch(any)

      { currentBuild.result = 'FAILURE' //sendMail() throw any }

      finally

      { step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports/', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite2_log.html', outputFileName: 'suite2_output.xml', reportFileName: 'suite2_report.html']) }

      Attachments

        Activity

          singhshyam shyam singh created issue -
          singhshyam shyam singh made changes -
          Field Original Value New Value
          Attachment Robot Test Summary.png [ 33749 ]
          singhshyam shyam singh made changes -
          Attachment Robot Test Summary.png [ 33750 ]
          singhshyam shyam singh made changes -
          Description I am building running my robot framework test case parallel in Jenkins pipeline. However the Robot Test Summary generated post build is not correct, it displaying the count/summary of last finished in parallelism, however it should show the count of individual job. It was merging the report also earlier but that I saved as different name & it worked fine, but still the Robot Test Summary is problem(please not my both suite below have different number of test cases).

          suite1:{
                              try {
          mvn -s /settings.xml install robotframework:run -Dexcludes=inprogress -Dsuites=test1 -DLogFileName=suite1_log.html -DoutputFileName=suite1_output.xml -DreportFileName=suite1_report.html'
                              
                                   currentBuild.result = 'SUCCESS'
                              }catch(any){
                                  currentBuild.result = 'FAILURE'
                                  throw any
                              }finally{
                                  step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite1_log.html', outputFileName: 'suite1_output.xml', reportFileName: 'suite1_report.html'])
                              }
          },
          suite2:{
                              try {
          mvn -s /settings.xml robotframework:run -Dexcludes=inprogress -Dsuites=test2 -DLogFileName=suite2_log.html -DoutputFileName=suite2_output.xml -DreportFileName=suite2_report.html'
                              
                                   currentBuild.result = 'SUCCESS'
                              }catch(any){
                                  currentBuild.result = 'FAILURE'
                                  //sendMail()
                                  throw any
                              }finally{
                                  step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports/', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite2_log.html', outputFileName: 'suite2_output.xml', reportFileName: 'suite2_report.html'])
                              }
          I am building running my robot framework test case parallel in Jenkins pipeline. However the Robot Test Summary generated post build is not correct, it displaying the count/summary of last finished in parallelism, however it should show the count of individual job. It was merging the report also earlier but that I saved as different name & it worked fine, but still the Robot Test Summary is problem(please not my both suite below have different number of test cases).

          suite1:{
                              try {
          mvn -s /settings.xml install robotframework:run -Dexcludes=inprogress -Dsuites=test1 -DLogFileName=suite1_log.html -DoutputFileName=suite1_output.xml -DreportFileName=suite1_report.html'
                              
                                   
          currentBuild.result = 'SUCCESS'
                              }catch(any){
                                  currentBuild.result = 'FAILURE'
                                  throw any
                              }finally{
                                  step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite1_log.html', outputFileName: 'suite1_output.xml', reportFileName: 'suite1_report.html'])
                              }
          },
          suite2:{
                              try {
          mvn -s /settings.xml robotframework:run -Dexcludes=inprogress -Dsuites=test2 -DLogFileName=suite2_log.html -DoutputFileName=suite2_output.xml -DreportFileName=suite2_report.html'
                              
                                   currentBuild.result = 'SUCCESS'
                              }catch(any){
                                  currentBuild.result = 'FAILURE'
                                  //sendMail()
                                  throw any
                              }finally{
                                  step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports/', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite2_log.html', outputFileName: 'suite2_output.xml', reportFileName: 'suite2_report.html'])
                              }
          singhshyam shyam singh made changes -
          Description I am building running my robot framework test case parallel in Jenkins pipeline. However the Robot Test Summary generated post build is not correct, it displaying the count/summary of last finished in parallelism, however it should show the count of individual job. It was merging the report also earlier but that I saved as different name & it worked fine, but still the Robot Test Summary is problem(please not my both suite below have different number of test cases).

          suite1:{
                              try {
          mvn -s /settings.xml install robotframework:run -Dexcludes=inprogress -Dsuites=test1 -DLogFileName=suite1_log.html -DoutputFileName=suite1_output.xml -DreportFileName=suite1_report.html'
                              
                                   
          currentBuild.result = 'SUCCESS'
                              }catch(any){
                                  currentBuild.result = 'FAILURE'
                                  throw any
                              }finally{
                                  step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite1_log.html', outputFileName: 'suite1_output.xml', reportFileName: 'suite1_report.html'])
                              }
          },
          suite2:{
                              try {
          mvn -s /settings.xml robotframework:run -Dexcludes=inprogress -Dsuites=test2 -DLogFileName=suite2_log.html -DoutputFileName=suite2_output.xml -DreportFileName=suite2_report.html'
                              
                                   currentBuild.result = 'SUCCESS'
                              }catch(any){
                                  currentBuild.result = 'FAILURE'
                                  //sendMail()
                                  throw any
                              }finally{
                                  step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports/', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite2_log.html', outputFileName: 'suite2_output.xml', reportFileName: 'suite2_report.html'])
                              }
          I am building running my robot framework test case parallel in Jenkins pipeline. However the Robot Test Summary generated post build is not correct, it displaying the count/summary of last finished in parallelism, however it should show the count of individual job. It was overwriting the report also earlier but that I saved as different name & it worked fine, but still the Robot Test Summary is problem(please not my both suite below have different number of test cases).

          suite1:{
                              try {
          mvn -s /settings.xml install robotframework:run -Dexcludes=inprogress -Dsuites=test1 -DLogFileName=suite1_log.html -DoutputFileName=suite1_output.xml -DreportFileName=suite1_report.html'
                              
                                   
          currentBuild.result = 'SUCCESS'
                              }catch(any){
                                  currentBuild.result = 'FAILURE'
                                  throw any
                              }finally{
                                  step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite1_log.html', outputFileName: 'suite1_output.xml', reportFileName: 'suite1_report.html'])
                              }
          },
          suite2:{
                              try {
          mvn -s /settings.xml robotframework:run -Dexcludes=inprogress -Dsuites=test2 -DLogFileName=suite2_log.html -DoutputFileName=suite2_output.xml -DreportFileName=suite2_report.html'
                              
                                   currentBuild.result = 'SUCCESS'
                              }catch(any){
                                  currentBuild.result = 'FAILURE'
                                  //sendMail()
                                  throw any
                              }finally{
                                  step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports/', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite2_log.html', outputFileName: 'suite2_output.xml', reportFileName: 'suite2_report.html'])
                              }
          iwelch Ian Welch made changes -
          Attachment MultipleRobotResults-Summary.png [ 34362 ]
          iwelch Ian Welch added a comment -

          Forewarning: I'm not a Java Developer so I may say stupid things
          I'm working on a pull request to resolve this issue. Thus far I have been able to add an optional parameter that defines a path in the Job directory to store the results. I'm also able to properly display the results of each test result file (using the result directory in the summary window. The next, and final, piece of the change is to properly link out to the Report visualization under "Browse Results".
          I've pored over the code for the last few days and I think I have identified the this data comes from `RobotResult.java` but I can't for the life of me figure out how the data is populated into this class. Can anyone point me in the proper direction to figure out how this data is populated?
          My best guess is that the data is stored in

          private transient List<RobotResultStatistics> overallStats;
          

          and populated from

          public void setStatsByCategory(List<RobotResultStatistics> statsByCategory) {
          		this.overallStats = statsByCategory;
          	}
          

          iwelch Ian Welch added a comment - Forewarning: I'm not a Java Developer so I may say stupid things I'm working on a pull request to resolve this issue. Thus far I have been able to add an optional parameter that defines a path in the Job directory to store the results. I'm also able to properly display the results of each test result file (using the result directory in the summary window. The next, and final, piece of the change is to properly link out to the Report visualization under "Browse Results". I've pored over the code for the last few days and I think I have identified the this data comes from `RobotResult.java` but I can't for the life of me figure out how the data is populated into this class. Can anyone point me in the proper direction to figure out how this data is populated? My best guess is that the data is stored in private transient List<RobotResultStatistics> overallStats; and populated from public void setStatsByCategory(List<RobotResultStatistics> statsByCategory) { this .overallStats = statsByCategory; }
          pradeeppt Pradeep PT added a comment -

          Is this issue fixed ?

          pradeeppt Pradeep PT added a comment - Is this issue fixed ?
          dirkrichter Dirk Richter added a comment -

          I do not see any open or merged pull requests. Thus i think issue is still present.

          In my opinion having multiple test summaries is ugly. Did you considered merging them up to only 1 output.xml e.g. using rebot (see here)

          When your tests run locally on the same host, try using pabot which has less overhead and merges results, too.

          I decreased priority, because there are 2 possible workarounds.

          dirkrichter Dirk Richter added a comment - I do not see any open or merged pull requests. Thus i think issue is still present. In my opinion having multiple test summaries is ugly. Did you considered merging them up to only 1 output.xml e.g. using rebot (see here ) When your tests run locally on the same host, try using pabot which has less overhead and merges results, too. I decreased priority, because there are 2 possible workarounds.
          dirkrichter Dirk Richter made changes -
          Priority Critical [ 2 ] Trivial [ 5 ]
          singhshyam shyam singh added a comment -

          dirkrichter, Yes we are doing the same way now, merging the reports  & constructing a merged output.xml & same feed to plugin, but with the number of parallel plan & Test cases growing, merging of report is an overhead, it's increasing the pipeline time. Also Sometime it's great if we can plot the data on module/domain basis separately instead of merging all. Here main concern is the time while merging of report. we have around 10 parallel plan running & merging itself take ~5 min. Good to have this feature so we can restrict merging to module or domain level only.

          singhshyam shyam singh added a comment - dirkrichter , Yes we are doing the same way now, merging the reports  & constructing a merged output.xml & same feed to plugin, but with the number of parallel plan & Test cases growing, merging of report is an overhead, it's increasing the pipeline time. Also Sometime it's great if we can plot the data on module/domain basis separately instead of merging all. Here main concern is the time while merging of report. we have around 10 parallel plan running & merging itself take ~5 min. Good to have this feature so we can restrict merging to module or domain level only.
          adir_atias Adir Atias made changes -
          Assignee jpiironen [ jpiironen ] Adir Atias [ adir_atias ]
          adir_atias Adir Atias made changes -
          Assignee Adir Atias [ adir_atias ] jpiironen [ jpiironen ]
          megafon Jan Duris added a comment -

          Hi guys. Any progress with this issue? I used it same way (merged results by rebot), but I would prefer to display separate Robot Test Summaries for separate tests which runs in parallel.

          megafon Jan Duris added a comment - Hi guys. Any progress with this issue? I used it same way (merged results by rebot), but I would prefer to display separate Robot Test Summaries for separate tests which runs in parallel.
          jpiironen jpiironen added a comment -

          Hi,

          We've resumed development of the plugin and currently doing a release for more urgent bugs. This seems like a useful improvement for the next iteration.

          jpiironen jpiironen added a comment - Hi, We've resumed development of the plugin and currently doing a release for more urgent bugs. This seems like a useful improvement for the next iteration.
          jpiironen jpiironen made changes -
          Issue Type Bug [ 1 ] Improvement [ 4 ]
          beeofdoom Matt Calderaz added a comment -

          Just encountered this issue today in overhauling CI for our RF tests; it's not critical, but a fix would be greatly appreciated.

          beeofdoom Matt Calderaz added a comment - Just encountered this issue today in overhauling CI for our RF tests; it's not critical, but a fix would be greatly appreciated.
          graines Glen Raines added a comment -

          Our team here would also prefer to display separate Robot Test Summaries, so a fix would be really appreciated.

          graines Glen Raines added a comment - Our team here would also prefer to display separate Robot Test Summaries, so a fix would be really appreciated.

          I'm facing the same issue on my project. I confirm that a fix would be really appreciated 

          Thanks in advance

          mpenven Mériadec Penven added a comment - I'm facing the same issue on my project. I confirm that a fix would be really appreciated  Thanks in advance
          hifi Juho Saarinen made changes -
          Assignee jpiironen [ jpiironen ] Juho Saarinen [ hifi ]
          hifi Juho Saarinen made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          singhshyam shyam singh made changes -
          Attachment RobotFramework_testTable.PNG [ 46025 ]
          singhshyam shyam singh added a comment -

          We have achieved the same in-house by modifying the same plugin, instead of merging the report, we are showing the merge count and all the report in tabular form,  user can access individual report. Indeed this is very fast, whereas we used to spend lots of time in merging the report . Please refer the attached screen shot.

          singhshyam shyam singh added a comment - We have achieved the same in-house by modifying the same plugin, instead of merging the report, we are showing the merge count and all the report in tabular form,  user can access individual report. Indeed this is very fast, whereas we used to spend lots of time in merging the report . Please refer the attached screen shot.

          singhshyam Can you please provide details of how you were able to generate the table?

          sipani909 Abhijit Sipani added a comment - singhshyam Can you please provide details of how you were able to generate the table?

          Hi, something similar was addressed in the warning-ng plugin https://issues.jenkins-ci.org/browse/JENKINS-54027?attachmentViewMode=list.
          Maybe if the id and name parameters would be added, it could solve this issue.

          astaruch Andrej Staruch added a comment - Hi, something similar was addressed in the warning-ng plugin https://issues.jenkins-ci.org/browse/JENKINS-54027?attachmentViewMode=list . Maybe if the id and name parameters would be added, it could solve this issue.
          astaruch Andrej Staruch made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          tattoo Tatu Kairi added a comment - - edited

          If you are running tests in parallel, there's few best practices to keep in mind:

          • if the paraller runs are writing in to same Jenkins workspace, the Robot outputs' file names will clash. Easy fix is to utilise --timestampoutputs or write different test executions into separate folders with --outputdir/-d.
          • after runs had been made, one can combine separate output.xml's into one with rebot
            • if you are combining outputs, it's good idea for the sake of performance to use -l NONE and -r NONE. this way in the end, there's only one output.xml, log.html and report.html from the entire pipeline, that can then be handled by robot Jenkins plugin as normal.
          tattoo Tatu Kairi added a comment - - edited If you are running tests in parallel, there's few best practices to keep in mind: if the paraller runs are writing in to same Jenkins workspace, the Robot outputs' file names will clash. Easy fix is to utilise --timestampoutputs or write different test executions into separate folders with --outputdir/-d . after runs had been made, one can combine separate output.xml 's into one with rebot if you are combining outputs, it's good idea for the sake of performance to use -l NONE and -r NONE . this way in the end, there's only one output.xml , log.html and report.html from the entire pipeline, that can then be handled by robot Jenkins plugin as normal.
          tattoo Tatu Kairi added a comment -

          singhshyam astaruch

          Hi,
          Do I understand the request correctly: you would wish an option in plugin that if there are multiple output.xml's available, the plugin would show the Robot Test Summary for each?

          (Also see my comment above this one)

          tattoo Tatu Kairi added a comment - singhshyam astaruch Hi, Do I understand the request correctly: you would wish an option in plugin that if there are multiple output.xml's available, the plugin would show the Robot Test Summary for each? (Also see my comment above this one)
          singhshyam shyam singh added a comment - - edited

          tattoo, Yes if you have multiple output.xml plugin should show summary for each. Regarding merging please refer my comment above(https://issues.jenkins.io/browse/JENKINS-37739?focusedCommentId=312799&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-312799).  We have address this issue by modifying the plugin & it's displaying as we desired(please refer my comment above and sample screen shot of the same, same is running fine for long & we didn't see issue so far https://issues.jenkins.io/browse/JENKINS-37739?focusedCommentId=360583&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-360583 ). We are happy to create a pull request and deliver the same if other folks need it and agreed on same.

          singhshyam shyam singh added a comment - - edited tattoo , Yes if you have multiple output.xml plugin should show summary for each. Regarding merging please refer my comment above( https://issues.jenkins.io/browse/JENKINS-37739?focusedCommentId=312799&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-312799 ).  We have address this issue by modifying the plugin & it's displaying as we desired(please refer my comment above and sample screen shot of the same, same is running fine for long & we didn't see issue so far https://issues.jenkins.io/browse/JENKINS-37739?focusedCommentId=360583&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-360583  ). We are happy to create a pull request and deliver the same if other folks need it and agreed on same.
          tattoo Tatu Kairi made changes -
          Summary Plugin not displaying Robot Test Summary correctly if robot framework task running in parallel in pipeline Support multiple output.xml's to support parallel test executions in pipeline
          tattoo Tatu Kairi added a comment -

          singhshyam,

          Just wanted to clarify what is the exact issue discussed here. I have edited the issue title accordingly for it to be more clear.

          As said in my previous comment, parallel test executions with plugin can be done already by utilising Robot's built-in tool rebot and designing the jobs well. That's at least our own personal experience using the plugin in a wide variety of customer cases and never having a need for the feature(s) this issue discusses. After talking with other maintainers, our current view is that this is already-supported, generic way that does not increase the complexity of the plugin itself. This route has also been taken by the parallel executor tool pabot. We do not therefore plan to include this feature in the plugin, unless something changes. We're keeping this issue open for such future discoveries.

          It surprises me that combining 10 output.xml's would take five minutes to complete. Can you share what is roughly the file size of these output.xml's? There might be a case to be made to make Robot Framework core devs improve the performance of rebot if it's demonstrable that performance is that bad. "The overhead" of combining different results is being payed somewhere in any case, be it in RF itself or in the plugin.

          Also, be aware that Robot Framework 4.0 is due to be released this month and the new release introduces changes the structure of output.xml. We have had to make changes in the plugin for that and I'm guessing you might need to do so too.

          tattoo Tatu Kairi added a comment - singhshyam , Just wanted to clarify what is the exact issue discussed here. I have edited the issue title accordingly for it to be more clear. As said in my previous comment , parallel test executions with plugin can be done already by utilising Robot's built-in tool rebot and designing the jobs well. That's at least our own personal experience using the plugin in a wide variety of customer cases and never having a need for the feature(s) this issue discusses. After talking with other maintainers, our current view is that this is already-supported, generic way that does not increase the complexity of the plugin itself. This route has also been taken by the parallel executor tool pabot . We do not therefore plan to include this feature in the plugin, unless something changes. We're keeping this issue open for such future discoveries. It surprises me that combining 10 output.xml 's would take five minutes to complete. Can you share what is roughly the file size of these output.xml 's? There might be a case to be made to make Robot Framework core devs improve the performance of rebot  if it's demonstrable that performance is that bad. "The overhead" of combining different results is being payed somewhere in any case, be it in RF itself or in the plugin. Also, be aware that Robot Framework 4.0 is due to be released this month and the new release introduces changes the structure of output.xml . We have had to make changes in the plugin for that and I'm guessing you might need to do so too.
          singhshyam shyam singh added a comment - - edited

          tattoo, thanks for your reply. 

          I captured the issue in description,  also I wouldn't call this as an issue, instead some expected behavior is missing, so it's an improvement good to have . Let me explain again

          1. Is running in parallel is an issue? --> No, it's not an issue, we can very well do it Jenkins parallel concept(please note we are using maven-robotframework plugin)  and it's running fine.
          2. Is merging is an issue? --> No, as I mention in earlier comment, merging can very well done and final output.xml can be passed to robot plugin & it plugin worked well.
          3. Why not going for merging?–> Indeed we were doing that, but with number of parallel run increases and kind of test we have size of output.xml also increases(each output.xml vary from 30MB to 60MB, depend upon kind of test, logging type and other factor) & we observed the merging time was on higher side, it was ~5 min when I file an improvement, it increases more further, as we increases in term of parallel job. So definitely it's not the right choice to merge the report(it may work well with light reports). One more disadvantage it has, the final html log report become too bulky that it will hang browser, also navigation/expand/collapse etc. is the biggest challenge. I really want to share dev feedback on this, & frankly none of them likes it.
          4. What was the issue with plugin?–> Plugin work well with what it stated, no doubt in that, but if we pass multiple output.xml, it's not considering and this is expected behavior.
          5. What advantage if we can publish in parallel–> We will not have extra overhead for merging. People who has bulky report can easily publish the report. Direct insight on reports, like which TAG/Plan fail so I can directly go to that report. Reports are lighter, & easy to navigate/collapse/expand. Also some user on this ticket has interest on such view 
          6. Do we still face the issue:- No, we don't, we modify the plugin & it worked well, no overhead of merging the report. Without merging we are publishing the consolidated count, as well as we are publishing the report in tabular format, each has a link to it's own report. It's very fast. In some Jobs we are publishing 50+ output.xml in parallel and we don't see any issue.

           

          I fully agree with you that with robot  Robot Framework 4.0 , output.xml has some changes, it may impact, but we have a plan to adopt for it, once we go on 4.0. I kept this open as some other might have interest and good to have such nice feature.

          Attaching a snips how it looks like in our production jenkins(dummy report), Ignore the time part, since it;s dummy output.xml so showing 0, it's showing actual time in production

           

          singhshyam shyam singh added a comment - - edited tattoo , thanks for your reply.  I captured the issue in description,  also I wouldn't call this as an issue, instead some expected behavior is missing, so it's an improvement good to have . Let me explain again Is running in parallel is an issue? --> No, it's not an issue, we can very well do it Jenkins parallel concept(please note we are using maven-robotframework plugin)  and it's running fine. Is merging is an issue? --> No, as I mention in earlier comment, merging can very well done and final output.xml can be passed to robot plugin & it plugin worked well. Why not going for merging?–> Indeed we were doing that, but with number of parallel run increases and kind of test we have size of output.xml also increases(each output.xml vary from 30MB to 60MB , depend upon kind of test, logging type and other factor) & we observed the merging time was on higher side, it was ~5 min when I file an improvement, it increases more further, as we increases in term of parallel job. So definitely it's not the right choice to merge the report(it may work well with light reports). One more disadvantage it has, the final html log report become too bulky that it will hang browser, also navigation/expand/collapse etc. is the biggest challenge. I really want to share dev feedback on this, & frankly none of them likes it. What was the issue with plugin?–> Plugin work well with what it stated, no doubt in that, but if we pass multiple output.xml, it's not considering and this is expected behavior. What advantage if we can publish in parallel–> We will not have extra overhead for merging. People who has bulky report can easily publish the report. Direct insight on reports, like which TAG/Plan fail so I can directly go to that report. Reports are lighter, & easy to navigate/collapse/expand. Also some user on this ticket has interest on such view  Do we still face the issue:- No, we don't, we modify the plugin & it worked well, no overhead of merging the report. Without merging we are publishing the consolidated count, as well as we are publishing the report in tabular format, each has a link to it's own report. It's very fast. In some Jobs we are publishing 50+ output.xml in parallel and we don't see any issue.   I fully agree with you that with robot   Robot Framework 4.0  , output.xml has some changes, it may impact, but we have a plan to adopt for it, once we go on 4.0. I kept this open as some other might have interest and good to have such nice feature. Attaching a snips how it looks like in our production jenkins(dummy report), Ignore the time part, since it;s dummy output.xml so showing 0, it's showing actual time in production  
          singhshyam shyam singh made changes -
          Attachment image-2021-03-02-16-20-43-422.png [ 54088 ]
          dirkrichter Dirk Richter added a comment -

          concerning point 6.: if you already have a working implementation, why you don't create a pull request here: https://github.com/jenkinsci/robot-plugin/pulls ? I can see there a pull request for RobotFramework 4.0 support, too...

          dirkrichter Dirk Richter added a comment - concerning point 6.: if you already have a working implementation, why you don't create a pull request here: https://github.com/jenkinsci/robot-plugin/pulls ? I can see there a pull request for RobotFramework 4.0 support, too...
          singhshyam shyam singh added a comment -

          dirkrichter, Yes we can create pull request if it's agreed , as it will change the report view in Jenkins also.

          singhshyam shyam singh added a comment - dirkrichter , Yes we can create pull request if it's agreed , as it will change the report view in Jenkins also.
          prabhav Prabhav added a comment -

          Hi, is there any update on releasing this feature.

          We have a scenario where this is required.

          prabhav Prabhav added a comment - Hi, is there any update on releasing this feature. We have a scenario where this is required.
          tattoo Tatu Kairi added a comment -

          prabhav,

          As said in my previous comment, parallel test executions with plugin can be done already by utilising Robot's built-in tool rebot and designing the jobs well. That's at least our own personal experience using the plugin in a wide variety of customer cases and never having a need for the feature(s) this issue discusses. After talking with other maintainers, our current view is that this is already-supported, generic way that does not increase the complexity of the plugin itself. This route has also been taken by the parallel executor tool pabot. We do not therefore plan to include this feature in the plugin, unless something changes. We're keeping this issue open for such future discoveries.

          tattoo Tatu Kairi added a comment - prabhav , As said in my previous comment , parallel test executions with plugin can be done already by utilising Robot's built-in tool rebot and designing the jobs well. That's at least our own personal experience using the plugin in a wide variety of customer cases and never having a need for the feature(s) this issue discusses. After talking with other maintainers, our current view is that this is already-supported, generic way that does not increase the complexity of the plugin itself. This route has also been taken by the parallel executor tool pabot . We do not therefore plan to include this feature in the plugin, unless something changes. We're keeping this issue open for such future discoveries.
          aleksisimell Aleksi Simell added a comment -

          We decided not to add this functionality to the plugin as discussed in the comments.

          aleksisimell Aleksi Simell added a comment - We decided not to add this functionality to the plugin as discussed in the comments.
          aleksisimell Aleksi Simell made changes -
          Resolution Won't Do [ 10001 ]
          Status Open [ 1 ] Closed [ 6 ]

          People

            hifi Juho Saarinen
            singhshyam shyam singh
            Votes:
            10 Vote for this issue
            Watchers:
            20 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: