Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37739

Support multiple output.xml's to support parallel test executions in pipeline

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Trivial
    • Resolution: Unresolved
    • Component/s: robot-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      I am building running my robot framework test case parallel in Jenkins pipeline. However the Robot Test Summary generated post build is not correct, it displaying the count/summary of last finished in parallelism, however it should show the count of individual job. It was overwriting the report also earlier but that I saved as different name & it worked fine, but still the Robot Test Summary is problem(please not my both suite below have different number of test cases).

      suite1:{
      try

      { mvn -s /settings.xml install robotframework:run -Dexcludes=inprogress -Dsuites=test1 -DLogFileName=suite1_log.html -DoutputFileName=suite1_output.xml -DreportFileName=suite1_report.html' currentBuild.result = 'SUCCESS' }

      catch(any)

      { currentBuild.result = 'FAILURE' throw any }

      finally

      { step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite1_log.html', outputFileName: 'suite1_output.xml', reportFileName: 'suite1_report.html']) }

      },
      suite2:{
      try

      { mvn -s /settings.xml robotframework:run -Dexcludes=inprogress -Dsuites=test2 -DLogFileName=suite2_log.html -DoutputFileName=suite2_output.xml -DreportFileName=suite2_report.html' currentBuild.result = 'SUCCESS' }

      catch(any)

      { currentBuild.result = 'FAILURE' //sendMail() throw any }

      finally

      { step([$class: 'RobotPublisher', outputPath:'/source/xxxxxx/target/robotframework-reports/', passThreshold: 0, unstableThreshold: 0, otherFiles: "", logFileName: 'suite2_log.html', outputFileName: 'suite2_output.xml', reportFileName: 'suite2_report.html']) }

        Attachments

        1. image-2021-03-02-16-20-43-422.png
          228 kB
          shyam singh
        2. MultipleRobotResults-Summary.png
          83 kB
          Ian Welch
        3. RobotFramework_testTable.PNG
          55 kB
          shyam singh
        4. Robot Test Summary.png
          20 kB
          shyam singh

          Activity

          Hide
          singhshyam shyam singh added a comment - - edited

          Tatu Kairi, thanks for your reply. 

          I captured the issue in description,  also I wouldn't call this as an issue, instead some expected behavior is missing, so it's an improvement good to have . Let me explain again

          1. Is running in parallel is an issue? --> No, it's not an issue, we can very well do it Jenkins parallel concept(please note we are using maven-robotframework plugin)  and it's running fine.
          2. Is merging is an issue? --> No, as I mention in earlier comment, merging can very well done and final output.xml can be passed to robot plugin & it plugin worked well.
          3. Why not going for merging?–> Indeed we were doing that, but with number of parallel run increases and kind of test we have size of output.xml also increases(each output.xml vary from 30MB to 60MB, depend upon kind of test, logging type and other factor) & we observed the merging time was on higher side, it was ~5 min when I file an improvement, it increases more further, as we increases in term of parallel job. So definitely it's not the right choice to merge the report(it may work well with light reports). One more disadvantage it has, the final html log report become too bulky that it will hang browser, also navigation/expand/collapse etc. is the biggest challenge. I really want to share dev feedback on this, & frankly none of them likes it.
          4. What was the issue with plugin?–> Plugin work well with what it stated, no doubt in that, but if we pass multiple output.xml, it's not considering and this is expected behavior.
          5. What advantage if we can publish in parallel–> We will not have extra overhead for merging. People who has bulky report can easily publish the report. Direct insight on reports, like which TAG/Plan fail so I can directly go to that report. Reports are lighter, & easy to navigate/collapse/expand. Also some user on this ticket has interest on such view 
          6. Do we still face the issue:- No, we don't, we modify the plugin & it worked well, no overhead of merging the report. Without merging we are publishing the consolidated count, as well as we are publishing the report in tabular format, each has a link to it's own report. It's very fast. In some Jobs we are publishing 50+ output.xml in parallel and we don't see any issue.

           

          I fully agree with you that with robot  Robot Framework 4.0 , output.xml has some changes, it may impact, but we have a plan to adopt for it, once we go on 4.0. I kept this open as some other might have interest and good to have such nice feature.

          Attaching a snips how it looks like in our production jenkins(dummy report), Ignore the time part, since it;s dummy output.xml so showing 0, it's showing actual time in production

           

          Show
          singhshyam shyam singh added a comment - - edited Tatu Kairi , thanks for your reply.  I captured the issue in description,  also I wouldn't call this as an issue, instead some expected behavior is missing, so it's an improvement good to have . Let me explain again Is running in parallel is an issue? --> No, it's not an issue, we can very well do it Jenkins parallel concept(please note we are using maven-robotframework plugin)  and it's running fine. Is merging is an issue? --> No, as I mention in earlier comment, merging can very well done and final output.xml can be passed to robot plugin & it plugin worked well. Why not going for merging?–> Indeed we were doing that, but with number of parallel run increases and kind of test we have size of output.xml also increases(each output.xml vary from 30MB to 60MB , depend upon kind of test, logging type and other factor) & we observed the merging time was on higher side, it was ~5 min when I file an improvement, it increases more further, as we increases in term of parallel job. So definitely it's not the right choice to merge the report(it may work well with light reports). One more disadvantage it has, the final html log report become too bulky that it will hang browser, also navigation/expand/collapse etc. is the biggest challenge. I really want to share dev feedback on this, & frankly none of them likes it. What was the issue with plugin?–> Plugin work well with what it stated, no doubt in that, but if we pass multiple output.xml, it's not considering and this is expected behavior. What advantage if we can publish in parallel–> We will not have extra overhead for merging. People who has bulky report can easily publish the report. Direct insight on reports, like which TAG/Plan fail so I can directly go to that report. Reports are lighter, & easy to navigate/collapse/expand. Also some user on this ticket has interest on such view  Do we still face the issue:- No, we don't, we modify the plugin & it worked well, no overhead of merging the report. Without merging we are publishing the consolidated count, as well as we are publishing the report in tabular format, each has a link to it's own report. It's very fast. In some Jobs we are publishing 50+ output.xml in parallel and we don't see any issue.   I fully agree with you that with robot   Robot Framework 4.0  , output.xml has some changes, it may impact, but we have a plan to adopt for it, once we go on 4.0. I kept this open as some other might have interest and good to have such nice feature. Attaching a snips how it looks like in our production jenkins(dummy report), Ignore the time part, since it;s dummy output.xml so showing 0, it's showing actual time in production  
          Hide
          dirkrichter Dirk Richter added a comment -

          concerning point 6.: if you already have a working implementation, why you don't create a pull request here: https://github.com/jenkinsci/robot-plugin/pulls ? I can see there a pull request for RobotFramework 4.0 support, too...

          Show
          dirkrichter Dirk Richter added a comment - concerning point 6.: if you already have a working implementation, why you don't create a pull request here: https://github.com/jenkinsci/robot-plugin/pulls ? I can see there a pull request for RobotFramework 4.0 support, too...
          Hide
          singhshyam shyam singh added a comment -

          Dirk Richter, Yes we can create pull request if it's agreed , as it will change the report view in Jenkins also.

          Show
          singhshyam shyam singh added a comment - Dirk Richter , Yes we can create pull request if it's agreed , as it will change the report view in Jenkins also.
          Hide
          prabhav Prabhav added a comment -

          Hi, is there any update on releasing this feature.

          We have a scenario where this is required.

          Show
          prabhav Prabhav added a comment - Hi, is there any update on releasing this feature. We have a scenario where this is required.
          Hide
          tattoo Tatu Kairi added a comment -

          Prabhav,

          As said in my previous comment, parallel test executions with plugin can be done already by utilising Robot's built-in tool rebot and designing the jobs well. That's at least our own personal experience using the plugin in a wide variety of customer cases and never having a need for the feature(s) this issue discusses. After talking with other maintainers, our current view is that this is already-supported, generic way that does not increase the complexity of the plugin itself. This route has also been taken by the parallel executor tool pabot. We do not therefore plan to include this feature in the plugin, unless something changes. We're keeping this issue open for such future discoveries.

          Show
          tattoo Tatu Kairi added a comment - Prabhav , As said in my previous comment , parallel test executions with plugin can be done already by utilising Robot's built-in tool rebot and designing the jobs well. That's at least our own personal experience using the plugin in a wide variety of customer cases and never having a need for the feature(s) this issue discusses. After talking with other maintainers, our current view is that this is already-supported, generic way that does not increase the complexity of the plugin itself. This route has also been taken by the parallel executor tool pabot . We do not therefore plan to include this feature in the plugin, unless something changes. We're keeping this issue open for such future discoveries.

            People

            Assignee:
            hifi Juho Saarinen
            Reporter:
            singhshyam shyam singh
            Votes:
            10 Vote for this issue
            Watchers:
            20 Start watching this issue

              Dates

              Created:
              Updated: