Bruno P. Kinoshita added a comment - 2013-01-13 23:31 Hi again Todd :o) You are not the first one to complain about the TAP plug-in not being fully compliant with Perl's test tools When I wrote the plug-in I was focused in using it for TAP with Java (mainly using Selenium + TestNG outputting TAP with attachments [yamlish] ). Could you send or attach an example, please (take care with private or confidential data in your attachments as it takes a while to delete an attachment in JIRA)? tap4j probably can't handle prove's output... but maybe we could create a new parser in tap plug-in, that would pre-parse prove output and then use tap4j parsers to get the TAP information. What do you think? In order to keep backward compatibility, that would be a new option in the plug-in job configuration section, and not enabled by default (i.e. a new unchecked check box with a title like "Parse prove output"). Thanks for filing this issue. I'm using more Perl at work now, so maybe we can fix this issue and I'll learn more about prove and other Perl utilities. /B

Todd Rinaldo added a comment - 2013-01-14 18:47 Sure this is really simple to produce. I apologize for my bad terminology in advance 1. Make a project called YAML-Syck 2. Track git source: git://github.com/toddr/YAML-Syck.git 3. Your build steps are: 3a. perl Makefile.PL 3b. make 3c. prove -wbvm t/*.t 2>&1 > $WORKSPACE/prove.tap 4. Add the TAP plugin to the Post-build Actions. Test results come from prove.tap This is my console output from running it: Started by user Todd Rinaldo Building in workspace / var /lib/jenkins/workspace/YAML-Syck Checkout:YAML-Syck / / var /lib/jenkins/workspace/YAML-Syck - hudson.remoting.LocalChannel@1dc2b19 Using strategy: Default Fetching changes from 1 remote Git repository Fetching upstream changes from git: //github.com/toddr/YAML-Syck.git Seen branch in repository origin/HEAD Seen branch in repository origin/historical/git-svn-conversion Seen branch in repository origin/master Seen branch in repository origin/tmp Commencing build of Revision 9708eff55268b905a9edd5fa4cb51337f8646430 (origin/historical/git-svn-conversion) Checking out Revision 9708eff55268b905a9edd5fa4cb51337f8646430 (origin/historical/git-svn-conversion) No change to record in branch origin/historical/git-svn-conversion [YAML-Syck] $ /bin/sh -xe /tmp/hudson5181651939588478127.sh + perl Makefile.PL *** WARNING *** YAML::Syck version >=0.60 breaks compatibility with earlier versions of YAML::Syck and YAML.pm (<0.60) when serializing blessed references. See the COMPATIBILITY file for more information. Writing Makefile for YAML::Syck Writing MYMETA.yml and MYMETA.json [YAML-Syck] $ /bin/sh -xe /tmp/hudson4366761815976251983.sh + make cp lib/JSON/Syck.pm blib/lib/JSON/Syck.pm cp lib/YAML/Loader/Syck.pm blib/lib/YAML/Loader/Syck.pm cp lib/YAML/Dumper/Syck.pm blib/lib/YAML/Dumper/Syck.pm cp lib/YAML/Syck.pm blib/lib/YAML/Syck.pm /usr/bin/gcc -c -I. -I/usr/local/cpanel/3rdparty/perl/514/include -L/usr/local/cpanel/3rdparty/perl/514/lib -I/usr/local/cpanel/3rdparty/include -L/usr/local/cpanel/3rdparty/lib -DAPPLLIB_EXP= "/usr/local/cpanel" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Os -DVERSION=\ "1.04\" -DXS_VERSION=\ "1.04\" -fPIC "-I/usr/local/cpanel/3rdparty/perl/514/lib/perl5/5.14.3/i386-linux-64int/CORE" bytecode.c /usr/bin/gcc -c -I. -I/usr/local/cpanel/3rdparty/perl/514/include -L/usr/local/cpanel/3rdparty/perl/514/lib -I/usr/local/cpanel/3rdparty/include -L/usr/local/cpanel/3rdparty/lib -DAPPLLIB_EXP= "/usr/local/cpanel" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Os -DVERSION=\ "1.04\" -DXS_VERSION=\ "1.04\" -fPIC "-I/usr/local/cpanel/3rdparty/perl/514/lib/perl5/5.14.3/i386-linux-64int/CORE" emitter.c /usr/bin/gcc -c -I. -I/usr/local/cpanel/3rdparty/perl/514/include -L/usr/local/cpanel/3rdparty/perl/514/lib -I/usr/local/cpanel/3rdparty/include -L/usr/local/cpanel/3rdparty/lib -DAPPLLIB_EXP= "/usr/local/cpanel" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Os -DVERSION=\ "1.04\" -DXS_VERSION=\ "1.04\" -fPIC "-I/usr/local/cpanel/3rdparty/perl/514/lib/perl5/5.14.3/i386-linux-64int/CORE" gram.c /usr/bin/gcc -c -I. -I/usr/local/cpanel/3rdparty/perl/514/include -L/usr/local/cpanel/3rdparty/perl/514/lib -I/usr/local/cpanel/3rdparty/include -L/usr/local/cpanel/3rdparty/lib -DAPPLLIB_EXP= "/usr/local/cpanel" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Os -DVERSION=\ "1.04\" -DXS_VERSION=\ "1.04\" -fPIC "-I/usr/local/cpanel/3rdparty/perl/514/lib/perl5/5.14.3/i386-linux-64int/CORE" handler.c /usr/bin/gcc -c -I. -I/usr/local/cpanel/3rdparty/perl/514/include -L/usr/local/cpanel/3rdparty/perl/514/lib -I/usr/local/cpanel/3rdparty/include -L/usr/local/cpanel/3rdparty/lib -DAPPLLIB_EXP= "/usr/local/cpanel" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Os -DVERSION=\ "1.04\" -DXS_VERSION=\ "1.04\" -fPIC "-I/usr/local/cpanel/3rdparty/perl/514/lib/perl5/5.14.3/i386-linux-64int/CORE" implicit.c /usr/bin/gcc -c -I. -I/usr/local/cpanel/3rdparty/perl/514/include -L/usr/local/cpanel/3rdparty/perl/514/lib -I/usr/local/cpanel/3rdparty/include -L/usr/local/cpanel/3rdparty/lib -DAPPLLIB_EXP= "/usr/local/cpanel" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Os -DVERSION=\ "1.04\" -DXS_VERSION=\ "1.04\" -fPIC "-I/usr/local/cpanel/3rdparty/perl/514/lib/perl5/5.14.3/i386-linux-64int/CORE" node.c /usr/bin/gcc -c -I. -I/usr/local/cpanel/3rdparty/perl/514/include -L/usr/local/cpanel/3rdparty/perl/514/lib -I/usr/local/cpanel/3rdparty/include -L/usr/local/cpanel/3rdparty/lib -DAPPLLIB_EXP= "/usr/local/cpanel" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Os -DVERSION=\ "1.04\" -DXS_VERSION=\ "1.04\" -fPIC "-I/usr/local/cpanel/3rdparty/perl/514/lib/perl5/5.14.3/i386-linux-64int/CORE" syck_.c /usr/bin/gcc -c -I. -I/usr/local/cpanel/3rdparty/perl/514/include -L/usr/local/cpanel/3rdparty/perl/514/lib -I/usr/local/cpanel/3rdparty/include -L/usr/local/cpanel/3rdparty/lib -DAPPLLIB_EXP= "/usr/local/cpanel" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Os -DVERSION=\ "1.04\" -DXS_VERSION=\ "1.04\" -fPIC "-I/usr/local/cpanel/3rdparty/perl/514/lib/perl5/5.14.3/i386-linux-64int/CORE" syck_st.c /usr/bin/gcc -c -I. -I/usr/local/cpanel/3rdparty/perl/514/include -L/usr/local/cpanel/3rdparty/perl/514/lib -I/usr/local/cpanel/3rdparty/include -L/usr/local/cpanel/3rdparty/lib -DAPPLLIB_EXP= "/usr/local/cpanel" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Os -DVERSION=\ "1.04\" -DXS_VERSION=\ "1.04\" -fPIC "-I/usr/local/cpanel/3rdparty/perl/514/lib/perl5/5.14.3/i386-linux-64int/CORE" token.c /usr/local/cpanel/3rdparty/perl/514/bin/perl "-Iinc" /usr/local/cpanel/3rdparty/perl/514/lib/perl5/cpanel_lib/ExtUtils/xsubpp -typemap /usr/local/cpanel/3rdparty/perl/514/lib/perl5/5.14.3/ExtUtils/typemap Syck.xs > Syck.xsc && mv Syck.xsc Syck.c /usr/bin/gcc -c -I. -I/usr/local/cpanel/3rdparty/perl/514/include -L/usr/local/cpanel/3rdparty/perl/514/lib -I/usr/local/cpanel/3rdparty/include -L/usr/local/cpanel/3rdparty/lib -DAPPLLIB_EXP= "/usr/local/cpanel" -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -Os -DVERSION=\ "1.04\" -DXS_VERSION=\ "1.04\" -fPIC "-I/usr/local/cpanel/3rdparty/perl/514/lib/perl5/5.14.3/i386-linux-64int/CORE" Syck.c Running Mkbootstrap for YAML::Syck () chmod 644 Syck.bs rm -f blib/arch/auto/YAML/Syck/Syck.so /usr/bin/gcc -shared -Os -L/usr/local/cpanel/3rdparty/perl/514/lib -L/usr/local/cpanel/3rdparty/lib -L/usr/lib -L/lib -L/usr/local/lib -fstack-protector bytecode.o emitter.o gram.o handler.o implicit.o node.o syck_.o syck_st.o token.o Syck.o -o blib/arch/auto/YAML/Syck/Syck.so \ \ chmod 755 blib/arch/auto/YAML/Syck/Syck.so cp Syck.bs blib/arch/auto/YAML/Syck/Syck.bs chmod 644 blib/arch/auto/YAML/Syck/Syck.bs [YAML-Syck] $ /bin/sh -xe /tmp/hudson1688078711976494714.sh + prove -wbvm t/11-loadfile.t t/12-dumpfile.t t/1-basic.t t/2-scalars.t t/3-objects.t t/4-perl_tag_scheme.t t/json-basic.t t/json-circular-ref.t t/json-empty.t t/json-indent.t t/json-minus.t t/json-newline.t t/json- null .t t/json-refs.t t/json-singlequote.t t/json-usedoublequote.t t/leak.t t/yaml-blessed-ref.t t/yaml-hooks.t t/yaml-implicit-warnings.t t/yaml-tie.t Build step 'Execute shell' marked build as failure TAP Reports Processing: START Looking for TAP results report in workspace using pattern: prove.tap Saving reports... Processing '/ var /lib/jenkins/jobs/YAML-Syck/builds/2013-01-14_12-44-23/tap-master-files/prove.tap' Parsing TAP test result [/ var /lib/jenkins/jobs/YAML-Syck/builds/2013-01-14_12-44-23/tap-master-files/prove.tap]. org.tap4j.parser.ParserException: Error parsing TAP Stream: Duplicated TAP Plan found. at org.tap4j.parser.Tap13YamlParser.parseFile(Tap13YamlParser.java:602) at org.tap4j.plugin.TapParser.parse(TapParser.java:105) at org.tap4j.plugin.TapPublisher.loadResults(TapPublisher.java:297) at org.tap4j.plugin.TapPublisher.perform(TapPublisher.java:239) at hudson.tasks.BuildStepMonitor$2.perform(BuildStepMonitor.java:27) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:810) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:785) at hudson.model.Build$BuildExecution.post2(Build.java:183) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:732) at hudson.model.Run.execute(Run.java:1568) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:236) Caused by: org.tap4j.parser.ParserException: Duplicated TAP Plan found. at org.tap4j.parser.Tap13YamlParser.checkTAPPlanDuplicity(Tap13YamlParser.java:345) at org.tap4j.parser.Tap13YamlParser.parseLine(Tap13YamlParser.java:257) at org.tap4j.parser.Tap13YamlParser.parseFile(Tap13YamlParser.java:597) ... 12 more Found matching files but did not find any TAP results. [BFA] Scanning build for known causes... [BFA] Done. 0s Finished: FAILURE This is the output from prove: t/11-loadfile.t ............. 1..8 ok 1 - LoadFile(*DATA) works ok 2 - LoadFile(\*DATA) works ok 3 - LoadFile works with file names ok 4 - LoadFile works with IO::File ok 5 - LoadFile works with indirect filehandles ok 6 - LoadFile works with ordinary filehandles ok 7 - LoadFile works with glob refs ok 8 - LoadFile works with in-memory files ok t/12-dumpfile.t ............. 1..6 ok 1 - DumpFile works with filenames ok 2 - DumpFile works with IO::File ok 3 - DumpFile works with indirect file handles ok 4 - DumpFile works with ordinary file handles ok 5 - DumpFile works with glob refs ok 6 - DumpFile works with in-memory files ok t/1-basic.t ................. 1..3 # Running under perl version 5.014003 for linux # Current time local: Mon Jan 14 12:44:32 2013 # Current time GMT: Mon Jan 14 18:44:32 2013 # Using Test.pm version 1.25 ok 1 ok 2 ok 3 ok t/2-scalars.t ............... 1..81 # Running under perl version 5.014003 for linux # Current time local: Mon Jan 14 12:44:32 2013 # Current time GMT: Mon Jan 14 18:44:32 2013 # Using Test.pm version 1.25 ok 1 ok 2 ok 3 ok 4 ok 5 ok 6 ok 7 ok 8 ok 9 ok 10 ok 11 ok 12 ok 13 ok 14 ok 15 ok 16 ok 17 ok 18 ok 19 ok 20 ok 21 ok 22 ok 23 ok 24 ok 25 ok 26 ok 27 ok 28 ok 29 ok 30 ok 31 ok 32 ok 33 ok 34 ok 35 ok 36 ok 37 ok 38 ok 39 ok 40 ok 41 ok 42 ok 43 ok 44 ok 45 ok 46 ok 47 ok 48 ok 49 ok 50 ok 51 ok 52 ok 53 ok 54 ok 55 ok 56 ok 57 ok 58 ok 59 ok 60 ok 61 ok 62 ok 63 ok 64 ok 65 ok 66 ok 67 ok 68 ok 69 ok 70 ok 71 ok 72 ok 73 ok 74 ok 75 ok 76 ok 77 ok 78 ok 79 ok 80 ok 81 ok t/3-objects.t ............... 1..29 # Running under perl version 5.014003 for linux # Current time local: Mon Jan 14 12:44:32 2013 # Current time GMT: Mon Jan 14 18:44:32 2013 # Using Test.pm version 1.25 ok 1 ok 2 ok 3 ok 4 ok 5 ok 6 ok 7 ok 8 ok 9 ok 10 ok 11 ok 12 ok 13 ok 14 ok 15 ok 16 not ok 17 # Test 17 got: "--- !!perl/regexp (?^:123)\n" (t/3-objects.t at line 31) # Expected: "--- !!perl/regexp (?-xism:123)\n" not ok 18 # Test 18 got: "--- !!perl/regexp (?^:(?^:123))\n" (t/3-objects.t at line 32) # Expected: "--- !!perl/regexp (?-xism:123)\n" not ok 19 # Test 19 got: "--- !!perl/regexp:Foo (?^i:123)\n" (t/3-objects.t at line 35) # Expected: "--- !!perl/regexp:Foo (?i-xsm:123)\n" not ok 20 # Test 20 got: "--- !!perl/regexp:Foo (?^:(?^i:123))\n" (t/3-objects.t at line 36) # Expected: "--- !!perl/regexp:Foo (?i-xsm:123)\n" ok 21 ok 22 ok 23 ok 24 ok 25 ok 26 ok 27 ok 28 ok 29 Failed 4/29 subtests t/4-perl_tag_scheme.t ....... 1..18 # Running under perl version 5.014003 for linux # Current time local: Mon Jan 14 12:44:32 2013 # Current time GMT: Mon Jan 14 18:44:32 2013 # Using Test.pm version 1.25 ok 1 ok 2 ok 3 ok 4 ok 5 ok 6 ok 7 ok 8 ok 9 ok 10 ok 11 ok 12 ok 13 ok 14 ok 15 ok 16 ok 17 ok 18 ok t/json-basic.t .............. skipped: JSON compatibility broken since JSON 2.0 t/json-circular-ref.t ....... 1..1 ok 1 - No segfault ok t/json-empty.t .............. 1..1 ok 1 ok t/json-indent.t ............. 1..1 ok 1 ok t/json-minus.t .............. 1..1 ok 1 ok t/json-newline.t ............ 1..1 ok 1 ok t/json- null .t ............... 1..2 ok 1 ok 2 ok t/json-refs.t ............... 1..2 ok 1 ok 2 ok t/json-singlequote.t ........ 1..5 ok 1 ok 2 ok 3 ok 4 ok 5 - no extra space in the URL ok t/json-usedoublequote.t ..... 1..14 ok 1 ok 2 ok 3 ok 4 ok 5 ok 6 ok 7 ok 8 ok 9 ok 10 ok 11 ok 12 ok 13 ok 14 ok t/leak.t .................... 1..11 ok 1 # skip Devel::Leak not installed ok 2 # skip Devel::Leak not installed ok 3 # skip Devel::Leak not installed ok 4 # skip Devel::Leak not installed ok 5 # skip Devel::Leak not installed ok 6 # skip Devel::Leak not installed ok 7 # skip Devel::Leak not installed ok 8 # skip Devel::Leak not installed ok 9 # skip Devel::Leak not installed ok 10 # skip Devel::Leak not installed ok 11 # skip Devel::Leak not installed ok t/yaml-blessed-ref.t ........ 1..1 # Running under perl version 5.014003 for linux # Current time local: Mon Jan 14 12:44:33 2013 # Current time GMT: Mon Jan 14 18:44:33 2013 # Using Test.pm version 1.25 ok 1 ok t/yaml-hooks.t .............. 1..3 # Running under perl version 5.014003 for linux # Current time local: Mon Jan 14 12:44:33 2013 # Current time GMT: Mon Jan 14 18:44:33 2013 # Using Test.pm version 1.25 Odd number of elements in hash assignment at t/yaml-hooks.t line 21. Use of uninitialized value $args{ "object_id" } in hash element at t/yaml-hooks.t line 22. not ok 1 # Test 1 got: "--- !!perl/hash:Special \nobject_id: ~\n" (t/yaml-hooks.t at line 28) # Expected: "--- !!perl/hook:Special '12345' \n" Can't use string ( "Special=HASH(0x942f32c)" ) as a HASH ref while "strict refs" in use at t/yaml-hooks.t line 32. Dubious, test returned 255 (wstat 65280, 0xff00) Failed 3/3 subtests t/yaml-implicit-warnings.t .. ok 1 - no warnings 1..1 ok t/yaml-tie.t ................ 1..1 # Running under perl version 5.014003 for linux # Current time local: Mon Jan 14 12:44:33 2013 # Current time GMT: Mon Jan 14 18:44:33 2013 # Using Test.pm version 1.25 ok 1 ok Test Summary Report ------------------- t/3-objects.t (Wstat: 0 Tests: 29 Failed: 4) Failed tests: 17-20 t/yaml-hooks.t (Wstat: 65280 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 255 Parse errors: Bad plan. You planned 3 tests but ran 1. Files=21, Tests=188, 1 wallclock secs ( 0.09 usr 0.03 sys + 0.68 cusr 0.09 csys = 0.89 CPU) Result: FAIL

Todd Rinaldo added a comment - 2013-01-14 18:54 Right now my work around is NASTY. Instead of prove, I can do: mkdir -p $WORKSPACE/t; for test in t/*.t; do; prove -wlvm $test > $WORKSPACE/$test 2>&1; done Of course I'm still being bitten by JENKINS-16326 ...

Jochen Koegel added a comment - 2013-02-19 18:59 You can can use prove's --archive parameter to put one tap-file per test script into a directory: prove *.t --archive tap/ works nicely with jenkins + tap4j in my setup. Does this solve your problem?

Bruno P. Kinoshita added a comment - 2013-02-26 20:58 - edited @Todd, I tried @Jochen proposal, and it works like a charm for me. Can you check if that works for you, please? The plug-in can parse multiple test sets in a single file only as sub tests at moment. Thank you in advance

Bruno P. Kinoshita added a comment - 2013-03-16 15:21 Updated the plug-in Wiki, adding the section "Running Perl tests with prove".

Bruno P. Kinoshita added a comment - 2013-04-06 03:03 Fixed in 1.10

Todd Rinaldo added a comment - 2013-08-30 23:33 Thanks got it working. In the end, I had to do this to get it working: mkdir tap; prove -wbm t/*.t --archive tap