Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56750

Micro Focus Application Automation Tools - Build log incorrect for multiple test instances of the same test.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Minor Minor
    • hp-application-automation-tools-plugin

      I'm having a reporting issue with the Jenkins plug-in for QC. I opened a bug in the Jira for their plug-in but no one has looked at it. Can you assist and discuss with Micro Focus if needed? I downgraded to 5.6.2 from 5.7 because of another issue "length cannot be zero". I've attached the build log that has the error. I have a test case in a test set 9 times. The point is that I need to run the test on 9 PCs. The error in the build log shows the 9th PC repeatedly and it should show 1,2,3,4..9. See attached files.

      To clarify, the Jenkins job is running three Test Sets. Each Test Set has one test to run on 9 PCs. The build log should show 27 test instances consisting of 3 (1-9) tests. The report should have 3 test instances of (1), (2), (3), .... (9). Instead, there are 27 instances of (9).

        1. RunTestOnHostName.pptx
          195 kB
        2. image-2019-09-17-10-11-01-078.png
          image-2019-09-17-10-11-01-078.png
          11 kB
        3. BuildStep.png
          BuildStep.png
          14 kB
        4. build.log
          31 kB
        5. RunResults.png
          RunResults.png
          33 kB
        6. Tests about to run.png
          Tests about to run.png
          20 kB

          [JENKINS-56750] Micro Focus Application Automation Tools - Build log incorrect for multiple test instances of the same test.

          Carles Capdevila added a comment -

          I believe the Log File Filter Plugin has nothing to do with this? If this is only related to the Quality Center Plugin please use the "qc" component as mentioned in the plugin page

          Carles Capdevila added a comment - I believe the Log File Filter Plugin has nothing to do with this? If this is only related to the Quality Center Plugin please use the "qc" component as mentioned in the plugin page

          Jim Sears added a comment -

          I've updated the plugin to qc-plugin per Carles suggestion.

          Jim Sears added a comment - I've updated the plugin to qc-plugin per Carles suggestion.

          Carles Capdevila added a comment -

          jsoref / evanchen - Roman Seguy is put as the plugin maintainer but it looks to me like he's not anymore. As I've seen you're assigned to other qc tickets maybe you know who should be assigned to this one? I'm the maintainer of the Log File Filter Plugin and don't really know anything about the qc plugin. Thanks!

          Carles Capdevila added a comment - jsoref / evanchen - Roman Seguy is put as the plugin maintainer but it looks to me like he's not anymore. As I've seen you're assigned to other qc tickets maybe you know who should be assigned to this one? I'm the maintainer of the Log File Filter Plugin and don't really know anything about the qc plugin. Thanks!

          Josh Soref added a comment -

          I'm a drive by contributor across the general project. (I'm mostly touching plugins for a specific UI refresh, and just happen to do other cleanup as I go.)

          Sorry, I can't help. It's probably possible to mark the plug-in as "up for adoption".

          Josh Soref added a comment - I'm a drive by contributor across the general project. (I'm mostly touching plugins for a specific UI refresh, and just happen to do other cleanup as I go.) Sorry, I can't help. It's probably possible to mark the plug-in as "up for adoption".

          Jim Sears added a comment -

          Hi gront, admdoc

          As creator and last to modify the plug-in for Quality Center, I have reached out to for assistance. 

          I was asked to enter the plugin for this bug and this is the closest I could find. The website says the plugin does handle performance testing too so maybe this is the correct plug-in. 

          micro-focus-performance-center-integration-plugin

          Can either of you assist with the auto-assign of this bug or how this issue should be handled?

          Jim

           

          Jim Sears added a comment - Hi  gront , admdoc As creator and last to modify the plug-in for Quality Center, I have reached out to for assistance.  I was asked to enter the plugin for this bug and this is the closest I could find. The website says the plugin does handle performance testing too so maybe this is the correct plug-in.  micro-focus-performance-center-integration-plugin Can either of you assist with the auto-assign of this bug or how this issue should be handled? Jim  

          Joshua Yates added a comment -

          jimsears  I corrected the components to hp-application-automation-tools-plugin instead of performance center

           

          Thanks,

           

          Josh

          Joshua Yates added a comment - jimsears   I corrected the components to  hp-application-automation-tools-plugin  instead of performance center   Thanks,   Josh

          Daniel Gront added a comment -

          Hey jimsears, I can't understand where it the issue.

          Is is it in Micro Focus Application Automation Tools?

          • If so what is the build step that you're are trying to run?

          Or is it a issue with micro-focus-performance-center-integration-plugin or qc-plugin?

          Daniel Gront added a comment - Hey jimsears , I can't understand where it the issue. Is is it in Micro Focus Application Automation Tools? If so what is the build step that you're are trying to run? Or is it a issue with micro-focus-performance-center-integration-plugin or qc-plugin?

          Jim Sears added a comment -

          Hi @Daniel Gront.  I'm new to this Jira Project so please bear with me. When I opened the Jira bug for this issue, the component list did not contain your plug-in "Micro Focus Application Automation Tools". The list only contained the very old version for qc-plugin. Joshua from Orasi, our support vendor for Micro Focus products added the correct plugin to the component field. I didn't know you could use free form text to enter a value.

           

          The issue is not related to micro-focus-performance-center-integration-plugin or qc-plugin. I think there is some cleanup needed if your calling the new plugin version "Micro Focus Application Automation Tools" by the component name "qc-plugin".

           

          The answer to your questions are as follows. The build step is for "Micro Focus Application Automation Tools". I've attached the build log to this bug. I've also attached an image of the build step. The build log shows the error with plug-in. Please let me know how else I can help.

          Jim

          Jim Sears added a comment - Hi @Daniel Gront.  I'm new to this Jira Project so please bear with me. When I opened the Jira bug for this issue, the component list did not contain your plug-in "Micro Focus Application Automation Tools". The list only contained the very old version for qc-plugin. Joshua from Orasi, our support vendor for Micro Focus products added the correct plugin to the component field. I didn't know you could use free form text to enter a value.   The issue is not related to micro-focus-performance-center-integration-plugin or qc-plugin. I think there is some cleanup needed if your calling the new plugin version "Micro Focus Application Automation Tools" by the component name "qc-plugin".   The answer to your questions are as follows. The build step is for "Micro Focus Application Automation Tools". I've attached the build log to this bug. I've also attached an image of the build step. The build log shows the error with plug-in. Please let me know how else I can help. Jim

          Daniel Gront added a comment - - edited

          Hey jimsears, No worries it's okay

          The step you're executing is part of the Micro Focus Application Automation Tools (formerly HP/HPE Application Automation Tools, which is why the component in Jira called hp-application-..).

          By "The Quality Center plug-in is 5.6.2. and 5.7" You mean Micro Focus Application Tools plugin? Because our versions are pretty similar (see here).

          If not please add the correct version of the plugin.

          I'll move this defect to the developer of this step from the ALM side.

          If there is anything else let me know, or just ping me if there are things that are not moving.

          Daniel Gront added a comment - - edited Hey jimsears , No worries it's okay The step you're executing is part of the Micro Focus Application Automation Tools (formerly HP/HPE Application Automation Tools, which is why the component in Jira called hp-application-..). By "The Quality Center plug-in is 5.6.2. and 5.7" You mean Micro Focus Application Tools plugin? Because our versions are pretty similar (see here ). If not please add the correct version of the plugin. I'll move this defect to the developer of this step from the ALM side. If there is anything else let me know, or just ping me if there are things that are not moving.

          Roy Lu added a comment -

          Hi gront,

          Yes this is a issue of a build step belongs to our plugin. We may need Bogdan's help on this.

          Roy Lu added a comment - Hi gront , Yes this is a issue of a build step belongs to our plugin. We may need Bogdan's help on this.

          Jim Sears added a comment - - edited

          Hi gront, it has been over 2 months since the last comment on this issue.

          May I have an update? Did you get Bogdan's help as you mentioned?

          I see the issue is assigned to aslaakso as well but I don't see any updates.

           

          Jim Sears added a comment - - edited Hi gront , it has been over 2 months since the last comment on this issue. May I have an update? Did you get Bogdan's help as you mentioned? I see the issue is assigned to aslaakso as well but I don't see any updates.  

          Anda Sorina Laakso added a comment -

          Hi Jim,

          Let me know if i understud correctly the case since i just received this ticket. You have 3 sets of tests, and in each set there is a test case which is run on 9 different machines. The problem is that the test summary shows only the test from the last set 27th times. Is that right?

           

           

          Anda Sorina Laakso added a comment - Hi Jim, Let me know if i understud correctly the case since i just received this ticket. You have 3 sets of tests, and in each set there is a test case which is run on 9 different machines. The problem is that the test summary shows only the test from the last set 27th times. Is that right?    

          Jim Sears added a comment - - edited

          Hi Anda,

          I'll give you the steps to recreate and then show why it is a bug.

          1) Create HostList in QC for 3 PCs.

          2) Create a Test Set in QC and add the same test 3 times. (This will create three instances of the same test.)

          3) Update the "Planned HostName" for each test. Assign a different PC to each test instance.

          4) Create a job in Jenkins using the QC Plugin and execute the Test Set.

          The results in the console output only shows the Test execution for one "Planned HostName" and it repeats it 3 times.

          The results should be one test instance for each Planned HostName.

          Here are the wrong results in console output: (See the [3].

          ================================================
          Run status: Job failed, total tests: 3, succeeded: 3, failures: 0, errors: 0
          Passed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files*[3]*DeleteTempFiles
          Failed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files*[3]*DeleteTempFiles
          Passed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files*[3]*DeleteTempFiles

          ================================================

           

          This is what it should look like:

          ================================================
          Run status: Job failed, total tests: 3, succeeded: 3, failures: 0, errors: 0
          Passed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files*[1]*DeleteTempFiles
          Failed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files*[2]*DeleteTempFiles
          Passed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files*[3]*DeleteTempFiles

          ================================================

           

          The test is specific to the PC and if the results only show [3] then I can't tell which test passed or failed.

           

           

           

          Jim Sears added a comment - - edited Hi Anda, I'll give you the steps to recreate and then show why it is a bug. 1) Create HostList in QC for 3 PCs. 2) Create a Test Set in QC and add the same test 3 times. (This will create three instances of the same test.) 3) Update the "Planned HostName" for each test. Assign a different PC to each test instance. 4) Create a job in Jenkins using the QC Plugin and execute the Test Set. The results in the console output only shows the Test execution for one "Planned HostName" and it repeats it 3 times. The results should be one test instance for each Planned HostName. Here are the wrong results in console output: (See the  [3] . ================================================ Run status: Job failed, total tests: 3, succeeded: 3, failures: 0, errors: 0 Passed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files* [3] *DeleteTempFiles Failed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files* [3] *DeleteTempFiles Passed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files* [3] *DeleteTempFiles ================================================   This is what it should look like: ================================================ Run status: Job failed, total tests: 3, succeeded: 3, failures: 0, errors: 0 Passed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files* [1] *DeleteTempFiles Failed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files* [2] *DeleteTempFiles Passed : Root\Automation CoE\LabMaintenance\LabMaintenance-Files* [3] *DeleteTempFiles ================================================   The test is specific to the PC and if the results only show  [3] then I can't tell which test passed or failed.      

          Anda Sorina Laakso added a comment -

          Hi Jim,

          Can you tell me where I could upload the fix for this issue? Maybe an ftp server.

          Thank you.

           

          Anda Sorina Laakso added a comment - Hi Jim, Can you tell me where I could upload the fix for this issue? Maybe an ftp server. Thank you.  

          Jim Sears added a comment - - edited

          aslaakso

          I can share a drop box location but I need your email address to do so.

          Do you have an alternate location in mind?

          Jim

          Jim Sears added a comment - - edited aslaakso I can share a drop box location but I need your email address to do so. Do you have an alternate location in mind? Jim

          Anda Sorina Laakso added a comment -

          Hi Jim,

          I uploaded the file to the location indicated by you.

          If there are still problems let me know.

          Anda Sorina Laakso added a comment - Hi Jim, I uploaded the file to the location indicated by you. If there are still problems let me know.

          Jim Sears added a comment - - edited

          aslaakso

          The older HP Plugin has an issue and if you built on top of that then I may not be able to use it. We are using the 5.62 because of that error so when I try your patch, I get this error.

          We are using HP QC 12.55.548 Patch 3.

           
          Started by user
          Jim Sears
          Building on master in workspace d:\Jenkins\.jenkins\workspace\MF QC Test Set Execution\Automation CoE\TAF LAB PC Maintenance-Files
          ERROR: Build step failed with exception
          java.lang.NullPointerException
          at com.microfocus.application.automation.tools.model.RunFromAlmModel.CreateProperties(RunFromAlmModel.java:161)
          at com.microfocus.application.automation.tools.model.RunFromAlmModel.getProperties(RunFromAlmModel.java:151)
          at com.microfocus.application.automation.tools.run.RunFromAlmBuilder.perform(RunFromAlmBuilder.java:211)
          at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
          at hudson.model.Build$BuildExecution.build(Build.java:206)
          at hudson.model.Build$BuildExecution.doRun(Build.java:163)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
          at hudson.model.Run.execute(Run.java:1816)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:97)
          at hudson.model.Executor.run(Executor.java:429)
          Build step 'Execute Micro Focus functional tests from Micro Focus ALM' marked build as failure
          [WS-CLEANUP] Deleting project workspace...
          [WS-CLEANUP] Deferred wipeout is used...
          [WS-CLEANUP] done
          Finished: FAILURE

          Jim Sears added a comment - - edited aslaakso The older HP Plugin has an issue and if you built on top of that then I may not be able to use it. We are using the 5.62 because of that error so when I try your patch, I get this error. We are using HP QC 12.55.548 Patch 3.   Started by user Jim Sears Building on master in workspace d:\Jenkins\.jenkins\workspace\MF QC Test Set Execution\Automation CoE\TAF LAB PC Maintenance-Files ERROR: Build step failed with exception java.lang.NullPointerException at com.microfocus.application.automation.tools.model.RunFromAlmModel.CreateProperties(RunFromAlmModel.java:161) at com.microfocus.application.automation.tools.model.RunFromAlmModel.getProperties(RunFromAlmModel.java:151) at com.microfocus.application.automation.tools.run.RunFromAlmBuilder.perform(RunFromAlmBuilder.java:211) at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1816) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Build step 'Execute Micro Focus functional tests from Micro Focus ALM' marked build as failure [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Deferred wipeout is used... [WS-CLEANUP] done Finished: FAILURE

          Jim Sears added a comment - - edited

          aslaakso

          For reference, I'm reinstalling 5.8 version to see if the error is different. I may need a second ticket for 5.8 :-0.

           

          5.8 has an issue and doesn't start the test run. Please see the log below.

           

          [RouteOne Medium] $ "d:\Jenkins\.jenkins\workspace\Agile Enablement Factories\Sales And Product\1-2 Dealer Sales\RouteOne Medium\HpToolsLauncher.exe" -paramfile props13092019181324010.txt
          "Started..."
          Timeout is set to: 7200
          Run mode is set to: RUN_PLANNED_HOST
          ============================================================================
          Starting test set execution
          Test set name: Demo, Test set id: 1423
          Number of tests in set: 5
          Test 1: [1]R1_AKL_CAL_BUS_LSE will run on host: <missing in log but is assigned in QC>
          Test 2: [1]R1_AKL_CAL_BUS_RET will run on host:
          Test 3: [1]R1_AKL_CAL_IND_LSE will run on host:
          Test 4: [1]R1_AKL_CAL_IND_RET will run on host:
          Test 5: [1]R1_AKL_CA_BUS_ERR will run on host:
          Problem while running TestSet: Exception from HRESULT: 0x80040459
          "Scheduler started at:13/09/2019 18:13:28
          -------------------------------------------------------------------------------------------------------
          Test set: Demo, finished at 13/09/2019 18:13:29
          ================================================
          Run status: Job succeeded, total tests: 5, succeeded: 0, failures: 0, errors: 0, warnings: 0
          Unknown:
          Unknown:
          Unknown:
          Unknown:
          Unknown:
          ================================================

           

          Jim Sears added a comment - - edited aslaakso For reference, I'm reinstalling 5.8 version to see if the error is different. I may need a second ticket for 5.8 :-0.   5.8 has an issue and doesn't start the test run. Please see the log below.   [RouteOne Medium] $ "d:\Jenkins\.jenkins\workspace\Agile Enablement Factories\Sales And Product\1-2 Dealer Sales\RouteOne Medium\HpToolsLauncher.exe" -paramfile props13092019181324010.txt "Started..." Timeout is set to: 7200 Run mode is set to: RUN_PLANNED_HOST ============================================================================ Starting test set execution Test set name: Demo, Test set id: 1423 Number of tests in set: 5 Test 1: [1] R1_AKL_CAL_BUS_LSE will run on host: <missing in log but is assigned in QC> Test 2: [1] R1_AKL_CAL_BUS_RET will run on host: Test 3: [1] R1_AKL_CAL_IND_LSE will run on host: Test 4: [1] R1_AKL_CAL_IND_RET will run on host: Test 5: [1] R1_AKL_CA_BUS_ERR will run on host: Problem while running TestSet: Exception from HRESULT: 0x80040459 "Scheduler started at:13/09/2019 18:13:28 ------------------------------------------------------------------------------------------------------- Test set: Demo, finished at 13/09/2019 18:13:29 ================================================ Run status: Job succeeded, total tests: 5, succeeded: 0, failures: 0, errors: 0, warnings: 0 Unknown: Unknown: Unknown: Unknown: Unknown: ================================================  

          Daniel Gront added a comment - - edited

          Hey jimsears, the other issue seems more severe. Please open another ticket so we can asses this properly.

          narcisamgalan FYI.

          Daniel Gront added a comment - - edited Hey jimsears , the other issue seems more severe. Please open another ticket so we can asses this properly. narcisamgalan FYI.

          Jim Sears added a comment -

          I opened the second ticket and tried the new beta patch and I feel like the new patch is working without errors but I am still experiencing the HostName problem. I am unable to run a test on a specific HostName. I've update the other ticket with the details and I think one of these is now a duplicate of the other.

          Let me know how you would like to proceed or if both tickets should remain open.
          Jim

          Jim Sears added a comment - I opened the second ticket and tried the new beta patch and I feel like the new patch is working without errors but I am still experiencing the HostName problem. I am unable to run a test on a specific HostName. I've update the other ticket with the details and I think one of these is now a duplicate of the other. Let me know how you would like to proceed or if both tickets should remain open. Jim

          Anda Sorina Laakso added a comment -

          Hi Jim,

          Since the patch solved the initial issue for this ticket, you should close this ticket and update the other ticket with all the details about the other issue.

          Thanks.

          Anda Sorina Laakso added a comment - Hi Jim, Since the patch solved the initial issue for this ticket, you should close this ticket and update the other ticket with all the details about the other issue. Thanks.

          Anda Sorina Laakso added a comment -

          Can you also provide me the properties file? Thanks.

          Anda Sorina Laakso added a comment - Can you also provide me the properties file? Thanks.

          Anda Sorina Laakso added a comment -

          As you can see on the attach image for me the host name is not missing, Did you remember to add itin the "advanced" section?

          Anda Sorina Laakso added a comment - As you can see on the attach image for me the host name is not missing, Did you remember to add itin the "advanced" section?

          Anda Sorina Laakso added a comment -

          As you can see in the attached image for the me the host name is present when selecting run on planed host option. Did you specify the host name in the "Advanced " section of the job configuration?

          Thanks.

          Anda Sorina Laakso added a comment - As you can see in the attached image for the me the host name is present when selecting run on planed host option. Did you specify the host name in the "Advanced " section of the job configuration? Thanks.

          Jim Sears added a comment -

          Hi aslaakso

          Before I close this ticket, may I ask two things from you?

          1) Repeat your exercise but add the same test "GUITest100" so that you have three instances, [1], [2], [3]. And set the "Another Host" to be Host1, Host2, Host3.

          When you look at the results, the test report should say what test ran on which PC. Your example is reversed because you use the same host.

          2) You asked if I populated the HostName but if I did, that would be inefficient. One Jenkins job runs 10 tests, each test on a different PC. If I set the HostName in the Jenkins job then I would have to set it to one PC. That means I would need 10 jobs, 1 for each PC to run one test.

          Please see attached PPT.

           

          Jim Sears added a comment - Hi aslaakso Before I close this ticket, may I ask two things from you? 1) Repeat your exercise but add the same test "GUITest100" so that you have three instances,  [1] ,  [2] ,  [3] . And set the "Another Host" to be Host1, Host2, Host3. When you look at the results, the test report should say what test ran on which PC. Your example is reversed because you use the same host. 2) You asked if I populated the HostName but if I did, that would be inefficient. One Jenkins job runs 10 tests, each test on a different PC. If I set the HostName in the Jenkins job then I would have to set it to one PC. That means I would need 10 jobs, 1 for each PC to run one test. Please see attached PPT.  

          Hilda added a comment - - edited

          Hello,

          I have tried to reproduce this issue, but it is working as expected for me with the latest plugin version.

          Are you facing the issue with the latest version of our plugin?

          I am going to Close this case, but feel free to reopen it anytime, and we can discuss further steps.

          Thank you,

          Hilda Both

          Hilda added a comment - - edited Hello, I have tried to reproduce this issue, but it is working as expected for me with the latest plugin version. Are you facing the issue with the latest version of our plugin? I am going to Close this case, but feel free to reopen it anytime, and we can discuss further steps. Thank you, Hilda Both

            aslaakso Anda Sorina Laakso
            jimsears Jim Sears
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: