-
Improvement
-
Resolution: Done
-
Major
Request for change for the plugin: Micro Focus Application Automation Tools
Currently the configuration for this plugin has the parameter “Test Folder” and it is defined as follows:
The path of the test folder that will contain the uploaded test. The path doesn't include the Root test folder (Subject). For example, sampletestfolder\subfolder means, the tests will be uploaded to test folder named 'subfolder', which is under the test folder named 'sampletestfolder', and 'sampletestfolder' is under the root test folder 'Subject'.
The configuration of this parameter does NOT allow to upload tests cases in different subfolders.
Let me explain it with an example:
Our current project structure is someone like that:
Root/
- Automated tests/
- Module 1/
- Test A
- Test B
- Module 2/
- Test C
- Module 3/
- Test D
- Module 1/
Let’s imagine that we have configure Jenkins like that:
When we made an execution, the plugin upload the tests like follows:
Root/
- Automated tests/
- Test A
- Test B
- Test C
- Test D
- Module 1/
- Test A
- Test B
- Module 2/
- Test C
- Module 3/
- Test D
In this case, we want to avoid the duplication of these test cases.
We think that this is a very common situation where typically test cases are defined in different folders to have a better organizations, so the plugin should be able to localize them in the hierarchy.
See attached document
[JENKINS-60175] To be able to define test cases in hierarchy folder
Sergio Cuenca
added a comment - Hi Roy!!
Let’s imagine we have this structure in the test plan (this structure has been created manually, defining test folders and test cases):
| · Automated tests/ o Module 1/ § Module 1_Test A § Module 1_Test B o Module 2/ § Module 2_Test C o Module 3/ § Module 3_Test D |
Each test case has been correctly defined inside ALM manually, and all of them have their steps and all the required atributes.
Then, I use this XML file to load a test execution done with an external tool:
| <?xml version='1.0' encoding='UTF-8'?> <result> <suites> <suite> <cases> <case> <duration>9.0020</duration> <className>Module 1</className> <testName>Test A</testName> <skipped>false</skipped> <failedSince>0</failedSince> </case> <case> <duration>36.0010</duration> <className>Module 1</className> <testName>Test B</testName> <skipped>false</skipped> <failedSince>0</failedSince> </case> </cases> </suite> </suites> <duration>0.576</duration> </result> |
When tests executions are loaded in ALM, the test plan folder have the following structure:
| · Automated tests/ o Module 1_Test A o Module 1_Test B o Module 1/ § Module 1_Test A § Module 1_Test B o Module 2/ § Module 2_Test C o Module 3/ § Module 3_Test D |
Module 1_Test A and Module 1_Test B are duplicated within the Test Plan structure and this is a problem for us.
We want that the plugin recognizes that they already exist inside folder Automated tests / Module 1.
Sergio Cuenca
added a comment - Hi Roy!!
Let’s imagine we have this structure in the test plan (this structure has been created manually, defining test folders and test cases):
· Automated tests/
o Module 1/
§ Module 1_Test A
§ Module 1_Test B
o Module 2/
§ Module 2_Test C
o Module 3/
§ Module 3_Test D
Each test case has been correctly defined inside ALM manually, and all of them have their steps and all the required atributes.
Then, I use this XML file to load a test execution done with an external tool:
<?xml version='1.0' encoding='UTF-8'?>
<result>
<suites>
<suite>
<cases>
<case>
<duration>9.0020</duration>
<className>Module 1</className>
<testName>Test A</testName>
<skipped>false</skipped>
<failedSince>0</failedSince>
</case>
<case>
<duration>36.0010</duration>
<className>Module 1</className>
<testName>Test B</testName>
<skipped>false</skipped>
<failedSince>0</failedSince>
</case>
</cases>
</suite>
</suites>
<duration>0.576</duration>
</result>
When tests executions are loaded in ALM, the test plan folder have the following structure:
· Automated tests/
o Module 1_Test A
o Module 1_Test B
o Module 1/
§ Module 1_Test A
§ Module 1_Test B
o Module 2/
§ Module 2_Test C
o Module 3/
§ Module 3_Test D
Module 1_Test A and Module 1_Test B are duplicated within the Test Plan structure and this is a problem for us.
We want that the plugin recognizes that they already exist inside folder Automated tests / Module 1.
Hi scuenca,
I understand. You want the plugin not to create new tests if there are already tests with the same name. Just new runs be created. Am I correct?
Then, what about the test sets? Are there test sets already exists?
Besides, what if other customer still want new testsets and tests being uploaded to the specified folder even there’re same name in other folders?
In many years after Jenkins plugin being published, it's the first time we got such request. Please understand we have to consider most common usage.
Sergio Cuenca
added a comment - Hi Roy,
You are right. Test sets don't exist, so they can be created.
I don't understand why someone could want to repeat the same test definition twice (or more times) in the test plan.
How we use ALM (and we though that this was the expected use of the tool) is the following way:
- We have a catalogue of tets cases defined inside the test plan. These tests are organized inside folder (which are the modules of our software).
- When we need to execute one or more tests, we créate a new test set and we add a test instance insde the test set. Then we execute it.
- We always use the same tests, we don't repeat test cases. Sometimes we define new tests cases to cover new functionalities, but we don't créate the same tests in each execution.
- Now, we are automating some of the tests that we have defined. We are using an external tool to automate the tests cases. When we make an execution of a test case, we donant to repeat it in the test plan, because it already exists.
So, why to repeat a test case definition in every execution?
Sergio Cuenca
added a comment - Hi Roy,
You are right. Test sets don't exist, so they can be created.
I don't understand why someone could want to repeat the same test definition twice (or more times) in the test plan.
How we use ALM (and we though that this was the expected use of the tool) is the following way:
We have a catalogue of tets cases defined inside the test plan. These tests are organized inside folder (which are the modules of our software).
When we need to execute one or more tests, we créate a new test set and we add a test instance insde the test set. Then we execute it.
We always use the same tests, we don't repeat test cases. Sometimes we define new tests cases to cover new functionalities, but we don't créate the same tests in each execution.
Now, we are automating some of the tests that we have defined. We are using an external tool to automate the tests cases. When we make an execution of a test case, we donant to repeat it in the test plan, because it already exists.
So, why to repeat a test case definition in every execution? Hi scuenca,
Usually user creates test cases in ALM 'Test Plan' module to execute ALM tests such as 'VAPI', 'MANUAL', 'BUSINESS PROCESS' etc. These are ALM tests not external test. What we upload to ALM server through Jenkins plugin are external tests such like Junit, Nunit etc. For these kinds of tests, we don't create test plan before upload. They're created during upload. Besides, what test type do you chose when you create the test case? There's no 'EXTERNAL-TEST' option. This type is assigned to test case during upload too.
Do you mean you're automating some of the tests that used to be executed manually in ALM? Which are actually ALM tests?
Sergio Cuenca
added a comment - Do you mean you're automating some of the tests that used to be executed manually in ALM? Yes, that's the situation.
Which are actually ALM tests?
We have different types of tests like system tests, integration tests, among others, that usually are executed manually (all of them were designed in ALM are they all exists in ALM test plan). But now, we are starting using an automation tool and we automating some of them.
We are starting doing executions and we want to upload these executions in ALM. Here comes my request.
Sergio Cuenca
added a comment - Do you mean you're automating some of the tests that used to be executed manually in ALM? Yes, that's the situation.
Which are actually ALM tests?
We have different types of tests like system tests, integration tests, among others, that usually are executed manually (all of them were designed in ALM are they all exists in ALM test plan). But now, we are starting using an automation tool and we automating some of them.
We are starting doing executions and we want to upload these executions in ALM. Here comes my request. Hi scuenca,
I got your point. But I think you're a little milk the bull. How about a better solution?
You know, there're two jobs in the plugin called 'Execute functional tests from ALM' and 'Execute tests using ALM Lab Management' either of them can execute ALM test. You can configure them in your Jenkins build and ALM will execute the tests and got the result. So simple, this is how other customers use this plugin to do automation and this is why no other customer has such requirement like yours.
If you're insist to have such feature in the plugin, I can do it. But as it is over-specific, it may take time to make a new job to do this. Because your request is actually not upload external test result. It's actually upload the result back to test which already exist in ALM. They're a little different and we couldn't change the function of the current job. But even if we achieve this, the whole thing sounds like going around in circle, doesn't it? It's not elegant.
The plugin already provided a better way to automation. So could you please think about my suggestion?
Hi scuenca,
1. Are you saying the tests were uploaded to Root/Automated tests/ and either to Root/Automated/Module 1, Root/Automated/Module 2, Root/Automated/Module 3?
2. Currently only one path you can define in Jenkins plugin. For example here's Root/Automated tests/. If you want to define multiple patchs, which folder would you like the different tests being uploaded to?