Hi Vignesh!

Glad to hear that you are using the plug-in and the execution status is working. Let's focus on the attachments now.

Could you give more information on your project?

Try to explain here what you are using in your project. Say, if it's a TestNG project with Selenium, how are you creating/destroying WebDriver. You're using tap4j, right? How are you using TestNG/JUnit with TAP? Or are you using something different, like Python, Ruby? The more information you can provide on your project, the merrier.

I will try to set up a similar environment here.

Cheers, B

Hi Bruno,

I am using JUnit project with Selenium. JUnit reports are generated once the scripts are run. I am not using TAP. I am just trying to attach the results in the JUnit results into each of the test case in Testlink.

Below is the scenario:

Have JUnit automated test scripts
Have a job in Jenkins to run those scripts using ANT
Once the job finishes testlink XMP-RPC scans through the test results which has more than one Test class name in it. But the attachments which gets passed to Testlink is wrongly attached meaning 1st testcase in the test execution has 6 test results, then the next one has 5 and so on.

Have one key custom field which matches to the test class name in the test results XML.

In the image last_tc.png you can see I have 6 test cases in it.

I hope this helps.

Hello Vignesh, thanks for the quick response.

I'm not sure if I got it right. You are using the plug-in only to execute your JUnit+Selenium tests in Jenkins and have the execution status updated in TestLink. Then you are using a custom script to scan your workspace and upload attachments to TestLink. Is that right?

Your set up for the plug-in seems correct. The only part that I didn't quite get it was how you upload the attachments. The plug-in does that, but only when you are using TAP.

Thanks for providing further information
B

I dont have a custom script that scans and uploads the document. Actually the plugin attaches the results to test cases in Testlink automatically, without me doing anything. I dont know whether that is the right behavior. Also wanted to know if I want to attach JUnit test results how will i go about it.

Hi Vignesh!

Thanks for clearing it up, it's much more clear for me now. The plug-in, by default, attaches the Test Result it found. If the plug-in found a TestNG XML, then it will attach this one. The same goes for JUnit and TAP files. However, the behavior that you described here looks like a Bug.

Just few more questions (by the way, thanks for being so patient and answering all my dumb questions .

> Also I have one custom field which maps to the test suite name and test class name.

Does that mean that each test case in TestLink has a custom field, which value is the same as both suite and test class name in JUnit? I never used the plug-in this way. Always used either the suite name or test name or test class name (the new version of the plug-in will be easier to handle different test results and result seeking strategy).

Thanks! Have a great weekend and Carnival (not sure if you celebrate it though

B

Hi Bruno,

For me both the names of the Test suite name and the class name is the same, this i found out by looking at the junit report it showed me the same name.
The problem only occurs while attaching the JUnit reports to the test cases it found match with. I have not experienced any other problems with it. Let me know if this bug can be fixed??

Thanks a ton for the quick response...

Vignesh

Hi Bruno,

If you want an example of the Test suite and test class name being same please do let me know, I will sent it across

Thanks

Vignesh

Hi Vignesh!

>For me both the names of the Test suite name and the class name is the same, this i found out by looking at the junit report it showed me the same name.

I believe this issue is occurring for this reason At moment I'm working on a new release that will let you choose between one/or many test result seeking strategies. I.e., you'll be able to configure "For this Job, you have to seek JUnit results using certain pattern to locate files, and match the key custom field with the JUnit Suite Name".

All the strategies are very similar, except by the last term, surrounded by *'s. At moment, I've created the following strategies (the name tells what I'm using to match against the key custom field value):

• JUnit suite name
• JUnit test name
• JUnit test class name
• TestNG suite name
• TestNG class name
• TestNG package.class#method name (com.sample.TestA#testVoid)
• TAP file name

And for a future release, I plan to include:

• JUnit Xpath expression (this way, you'll can say something like: match the key custom field value against suite[name=LocatorsTests*])
• TAP YAMLish field

However, it would be great if you could attach a sample test project that produces similar output. Comments or suggestions on these new strategies are very welcome too! I'll try to cut a release before March, and if you have some spare time to test an alpha version, maybe you could test the plug-in and say if this new version works for your issue

Cheers, B

Hi Bruno,

Wanted to know do u want the Testlink project or the selenium project. Please let me know about this.

Hi Vignesh,

I was talking about the selenium project

Thank you in advance.

Cheers,
Bruno

Have same problem. Moreover, if test class has, for example, 2 test methods - report get attached twice!

Let's look at my example:

I have 3 test classes:

1. ProductCreateCopyDeleteSmokeWebTests (executes first and has 3 test methods)
2. RulesOverrideSmokeWebTests (executes second and has 1 test method)
3. RulesOverrideRuntimeSmokeWebTests (executes third and has 1 test method)

After jenkins job execution I get following results in TestLink:

1 TC	2 TC	3 TC

P.S. Is it possible to prevent jenkins from attaching reports at all?

Hi Jaroslavas! Thanks for confirming that this issue is happening indeed.

I haven't had time to thoroughly investigate what is happening, but I have a hunch that it has something to do with the result seeking strategy. I think you are using JUnit to execute your files. It generates XML's, that the plug-in looks for after it executed the tests.

The problem may be happening if you have more than one test class, test name or suite name that collides with the key custom field value. That happens because if you provided a pattern to find JUnit XML files, the plug-in will use all JUnit strategies to seek test results. I.e.: say you have a test case in testlink which the key custom field value is 'TestOperationA', and you have a JUnit class with name TestOperationA and a suite with name TestOperationA too. The JUnit class is included in this suite. What will happen, is that the plug-in will find these both items (the class and the suite) and for each one will attach its corresponding XML file.

> 2 test methods - report get attached twice!

Depends on the way the XML is generated, the key custom field value, name of test class, test suite. At moment, the plug-in does not work with test methods.

Now the good news.

1. Test methods will be supported in version 3.1
2. For three weeks I've been refactoring the plug-in code (to simplify the code) and add an Extension point (Jenkins API) for Result Seeking Strategy.
3. Now you will be able to choose which strategies you want to use, so you can choose and say: In this job, the plug-in should use only test methods. It will reduce the execution time, as well as will avoid wrong attachments / execution status updates.
4. A result strategy now has options, like the pattern for report files, the key custom field name (it's per strategy now), and whether it will upload attachments or not

> Is it possible to prevent jenkins from attaching reports at all?

It's already coded in the code trunk, but I still have to test the attachment under several circumstances to ensure that I didn't break anything with the refactoring.

Would you like to test an 3.1 alpha version? I can finish testing the upload of attachments in one or two days.

All the best!
Bruno

Hi, Bruno!

Thanks for the quick response.

It would be nice to test 3.1 TestLink plugin version. How can I get it for testing?

I've attached it to issue 11894, you can get it there if you want, but I would like to do some tests on the attachments feature before asking you to start with your tests, as I haven't had a chance to test it after the refactoring

https://issues.jenkins-ci.org/browse/JENKINS-11894

Thank You!

When will 3.1 version official release?

My first intention was to release 3.1 on 1st March, but due to the refactoring, I think it may be a good idea delay the release one or two weeks, so I can fix the remaining issues and give some time for some users to test the alpha version. If everything goes out correctly, we may be able to release 3.1 by around March 7th.

Hi Bruno,

Just wanted to know when will the alpha release happen?

Thanks
Vignesh

Hi there!

@Vignesh: I will attach testlink-3.1-alpha.hpi here tonight or tomorrow Last night I tested in a distributed environment and found some issues regarding object serialization. Still need to work on the result notes and other small things.

@Jaroslavas: Sorry Jaroslavas. I wrote a lot and didn't say anything. It is not possible in 3.0.2, but it is already ready in version 3.1, which is being tested by some users of the plug-in, such as Vignesh. Would you like to try this version before the release too?

Thank y'all!

Bruno,

Yes, it would be nice to try it before release. Actually I've already tested it a bit (got it from JENKINS-11894).

P.S. I've found one BUG while tessing:

1. If in 'Custom Fields' you define more than one field - plugin cannot find any test result (using JUnit). See screenshots:

   Result found	No result found

2. Also, help text (marked with "?" on the right) for some new fields are wrong.

Hi Jaroslavas,

I was aware of 2, but I haven't tested scenario 1 with many custom fields. It looks like a bug. I'll test today and will fill an issue here if it's confirmed it's an issue. Thanks a lot!!! This alpha test by end users save other users a lot of time

Hi Bruno.

I have one question about 3.1 version.

There are two seeking strategies in 3.1:

1. JUnit class name - searches only for <testcase classname="tests.AAAAAAAAAAA"> property in reports *.xml files and if classname match custom field value - result are recorder in TestLink (correct me if I am wrong)
2. JUnit case name - searches only for <testcase name="a1"/> property in reports *.xml files and if name match custom field value - result are recorder in TestLink (correct me if I am wrong)

How can I mix both strategies together? I'll explain what I mean. Imagine there are two classes:

• TestClass1 with methods TestCase1 and TestCase2
• TestClass2 with methods TestCase1 and TestCase2 (as you can notice method names in classes are the same)

So, for proper results in TestLink I need to match both classname and name, and only if both properties are the same as defined in custom fields then result for TC should be recorded.

Hi Jaroslavas,

We could create a new strategy for this, like JUnitClassNameAndCaseNameResultSeeker.

The reason for not changing the two existing ones, is that there are many tools that generate JUnit report, and some of them use only class name, others only case name in the XML (or none sometimes heh).

This strategy would match for <classname>.<testname> (e.g.: TestClass1.TestCase1).

What do you think?

Bruno.

It would be nice to have such strategy too. Thank you!

How this strategy will be presented in Jenkins?

1. Will it be two fields in Jenkins and TestLink. For example:
   • 'Test Class': tests.TestClass
   • 'Test Method': TestMethod
2. or it will be one field. For example:
   • 'Test Class with Method': tests.TestClass.TestMethod

Hi Jaroslavas,

Never mention it, thank you for helping to improve the plug-in with good ideas

More likely to be like option #2. At moment, there is no easy way to implement the option #1.

I'll have a working version ready for tests today or tomorrow.

@Jaroslavas, @Vignesh: I have just attached a new alpha version of the plug-in here, could you give it a try, please? The file name is testlink-3.1-alpha2.hpi

Thank you in advance!

TestLink Plug-in 3.1 alpha #2

Thank You! I'll try it today or/and tomorrow, and I will inform you about results.

What I have already found:

1. it is possible now to define multiple fields in 'Custom Fields'
2. help text problem remains

Till now I haven't found any problem in new version, however, I will try some more cases tomorrow. New plugin version works fine for me.

Thank You a lot for your work!

Hi Bruno,

I tested some scenarios. And it is working as expected. The results are getting attached correctly to the correct test case in Test link. I found just one Help problem. The help text for "Include Pattern" and "Key Custom Field" is the same. Apart from that I didnt find any problems.

Thank you for the work! This will help me a lot.

When is the official release?

Thanks,

Vignesh

Hi Bruno,

I tested some scenarios. And it is working as expected. The results are getting attached correctly to the correct test case in Test link. I found just one Help problem. The help text for "Include Pattern" and "Key Custom Field" is the same. Apart from that I didnt find any problems.

Thank you for the work! This will help me a lot.

When is the official release?

Thanks,

Vignesh

Thanks for the feedback Vignesh!

I'm trying to find some time to review all the HTML help files, log messages and the test summary notes

> Thank you for the work! This will help me a lot.

Thank you too! This alpha testing helps a lot!

>When is the official release?

It was planned for March 1st, but due to some bug fixing in this alpha version, and the HTML help files changed after the refactoring, I will postpone it for next week's Wednesday (but it may happen before that

Cheers, B

Hi Jaroslavas, sorry, I marked the JIRA e-mail with your comment in my inbox, but forgot to comment here.

> help text problem remains

Will review it in the next days. I'm working on another issue related to Maven projects (must be easy to fix). Then I'll start packaging the final release, fixing all the messages, texts, and add notes for i18n.

> Thank You a lot for your work!

Thank you too, for your time testing the alpha version. It helps me a lot, and prevents many bugs of passing to other users. You guys are the quality net of the plug-in

Cheers, B

Hi, Bruno.

Release is good. I haven't found any issues.

One question. What about platform support? Is it any chance to have it in future?

Oh, I have found what it's supported from 3.0 version (JENKINS-9054).. only for TAP. How it should be used?

Is it possible to have it for JUnit?

Now, if I run Test Plan with platform I get an exception

br.eti.kinoshita.testlinkjavaapi.TestLinkAPIException: (reportTCResult) - Parameter platformname OR platformid is required, but has not been provided
	at br.eti.kinoshita.testlinkjavaapi.BaseService.checkResponseError(BaseService.java:119)
	at br.eti.kinoshita.testlinkjavaapi.BaseService.executeXmlRpcCall(BaseService.java:92)
	at br.eti.kinoshita.testlinkjavaapi.TestCaseService.reportTCResult(TestCaseService.java:634)
	at br.eti.kinoshita.testlinkjavaapi.TestLinkAPI.reportTCResult(TestLinkAPI.java:1115)
	at hudson.plugins.testlink.TestLinkSite.updateTestCase(TestLinkSite.java:166)
	at hudson.plugins.testlink.result.AbstractJUnitResultSeeker.handleResult(AbstractJUnitResultSeeker.java:82)
	at hudson.plugins.testlink.result.JUnitCaseNameResultSeeker.seek(JUnitCaseNameResultSeeker.java:98)
	at hudson.plugins.testlink.TestLinkBuilder.perform(TestLinkBuilder.java:158)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:703)
	at hudson.model.Build$RunnerImpl.build(Build.java:178)
	at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
	at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:473)
	at hudson.model.Run.run(Run.java:1408)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:238)

So, maybe it is a trivial problem of one additional field in plugin options configuration on jenkins job?

Hi Jaroslavas!

That's a very interesting feature, that I really would like to see available for all strategies. However, there's one technical limitation from JUnit and TestNG that I never could overcome. They are not extensible. Let me explain why that's important.

When you're using platform, sometimes your test plan has more than one platform. So the platform has to exist for each test case, and then I cannot add one field in plugin options configuration that was my first try some time ago (there was an issue on that).

TAP can be extended with YAMLish, which looks a lot like JSON, but minus '

'. So with TAP, I expect you to include the platform for each case in your test result (like a test case in JUnit XML).

One alternative for JUnit and TestNG, is using tap4j JUnit Runner or TestNG Listener. This way, besides generating JUnit XML and TestNG XML, your tests will output TAP too. But you'll have to use the TAP file name strategy.

Does it make sense? Do you have any other idea to enable platforms on JUnit and TestNG?

Thanks

I don't really understood why it is JUnit or TestNG limitation.. I'll explain my situation.

Let's imagine we have such Test plan/Platforms/Test Case structure

For simplicity, let's use only Test Plan 1 for execution.

As I understand (correct me if I am wrong), now it works like this:

1. plugin gets set of test cases from defined Test Plan (TC11, TC12, TC13, TC14, TC15, TC16; let's call it 'definedTestCasesByTestPlan')
2. executes all tests that are specified as maven argument
3. records results for 'definedTestCasesByTestPlan' in TestLink for defined Test Plan

How it should work with platform property:

1. plugin gets set of test cases from defined Test Plan and Platform (TC11, TC12, TC13; let's call it 'definedTestCasesByTestPlanAndPlatform')
2. executes all tests that are specified as maven argument
3. record result for 'definedTestCasesByTestPlanAndPlatform' in TestLink for defined Test Plan and Platform combination

So, the problem is to get proper set of Test Cases for execution in TestLink (in step 1). However, JUnit/TestNG/TAP takes a part on step 2 and step 3.

Sorry if I am wrong in some cases (please, correct me).. I just want to understand the problem and try to help

Your understanding is perfect, and very clear for me. The actual problem is where to find which platform the test case used.

> How it should work with platform property:

Let's include one more step here.

1. plugin gets set of test cases from defined Test Plan and Platform (TC11, TC12, TC13; let's call it 'definedTestCasesByTestPlanAndPlatform')
2. executes all tests that are specified as maven argument
3. seeks result + platform in JUnit/TestNG XML
4. record result for 'definedTestCasesByTestPlanAndPlatform' in TestLink for defined Test Plan and Platform combination

I couldn't find an easy way to include the platform in JUnit or TestNG XML. That's way only TAP supports platforms at moment.

Thanks Jaroslavas, it's really great to discuss this issue with someone else that is keen to help

Bruno

Hi, Bruno.

Why plugin should seek for platform option in XML?

There's no test plan option in generated JUnit/TestNG XMLs, but no problem for plugin to find TC only from defined Test Plan.

Thank You, Bruno, for productive communication! And sorry for my silly questions. I'm not really programer guy - I'm tester. So, my point of view to problem may be different

Hi Jaroslavas

> Why plugin should seek for platform option in XML?

That's because a plan has many platforms. Let's say you have a test plan with two platforms. Now the plug-in must always include the platform ID or the platform name when invoking TestLink XML-RPC API methods. Otherwise an error is raised by TestLink.

> There's no test plan option in generated JUnit/TestNG XMLs, but no problem for plugin to find TC only from defined Test Plan.

That's a good argument

But here's no way to include the platform in the job configuration screen (imagine a scenario with 20 platforms). The platform is linked to the test case, and not to the test project (like test plan). Below I tried to represent how is the relation of each entity more or less (omitted test case version for simplicity)

test project -> has many -> test plan -> has many -> test case -> has one -> platform
..................................... -> has many -> platform

> Thank You, Bruno, for productive communication! And sorry for my silly questions.

I really enjoy having this type of conversation Jaroslavas. And no need for sorry your questions are very good.

> I'm not really programer guy - I'm tester. So, my point of view to problem may be different

That's why it's so important that you participate in the project. Each time a user like you propose a new feature, find a bug, or just has a cool conversation like this, the plug-in gets better and more useful for testers, developers, devops and so it goes.

This weekend I think I will have the first release candidate for 3.1, with all the issues found during alpha test fixed, as well as the release check list done (https://wiki.jenkins-ci.org/display/JENKINS/Release+check+list). I also want to include all the alpha testers in the plug-in team as contributors (unless they prefer remain anonymous

Cheers
Bruno

The issue with attachments has been fixed with the refactoring in the code and creation of result seeking strategies.

@Jaroslavas: I'm closing this issue to cut a new release. But feel free to add more comments here, or send e-mails too.

@Jaroslavas, @Vignesh: I want to include you guys to the plug-in list of contributors. I'll need the following information:

• id (usually the one that you use to log in here in JIRA)
• name
• e-mail
• GMT time zone (for example, mine is -3)

You're not required to be a contributor, but I really would like to include your names to the project. It's the least I can do after all the hard work with ideas, comments and alpha testing.

Have a nice weekend y'all. Bruno.

Thank You, Bruno, for your work!

My info:

• id: jar
• name: Jaroslavas
• e-mail: j.daskevicius@gmail.com
• GMT time zone: +2

Is RC version available for testing?

Your information has been added to the project Jaroslavas! Thank you for your support too

https://github.com/jenkinsci/testlink-plugin/blob/master/pom.xml#L110

The ID wasn't necessary, sorry.

> Is RC version available for testing?

It is! I attached the latest release candidate here. If no errors are found, we'll release a new version this week.

Thanks again!

release candidate 2

Hey Bruno,

My info:

name: Vignesh Senapathy
e-mail: vignesh.senapathy@gmail.com
GMT time zone: +5

Sorry my email is vigneshsenapathy@gmail.com

Hi Vignesh! Added as well https://github.com/jenkinsci/testlink-plugin/blob/master/pom.xml#L115

Thank you!

Fixed in 3.1

Bruno, hi.

No any issues from my side. Works as expected! Waiting for 3.1 release.

Thank You for your work!

Hi Jaroslavas!

Many thanks for your help.

The plug-in has been released minutes ago. It'll take some time to update the repositories and for it to be available from Jenkins update site. There are different ways to check when the new release is available, usually I just keep an eye in @jenkins_release twitter account.

https://twitter.com/#!/jenkins_release

Bruno