Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21835

Cucumber Test Result Plugin crashes on malformed output in Scenario Outline results produced by some tools

      What we did:
      Ran minimal cucumber feature test with Jenkins:

      Feature
      Feature: Sample Test
        Scenario Outline: Parsing scenarios with multiple examples
          Given I navigate to the root page
          Then I see the text 'home page'
          Examples:
          | a | b |
          | 1 | 2 |
      

      By a shell build step:

      shell build
      #!/bin/bash
      # Set up RVM (Ruby 2.0)
      . ~/.bashrc
      set -e
      CUCUMBER_OPTS='-f json -o cucumber.json features/sample.feature' rake cucumber
      

      What we got:
      [::1::]

      What we expected:
      No error, parsing of cucumber.json containing results from scenario outlines.

      [::1::]

      Output
      /var/lib/jenkins/.rvm/rubies/ruby-2.0.0-p353/bin/ruby -S bundle exec cucumber -f json -o cucumber.json features/sample.feature
      Using the default profile...
      Feature: Sample Test
      
        Scenario Outline: Parsing scenarios with multiple examples # features/sample.feature:2
          Given I navigate to the root page                        # features/step_definitions/navigation_steps.rb:15
          Then I see the text 'home page'                          # features/step_definitions/study_admin.rb:23
      
          Examples: 
            | a | b |
            | 1 | 2 |
      
      1 scenario (1 passed)
      2 steps (2 passed)
      0m1.080s
      Xvfb stopping
      Archiving artifacts
      [Cucumber Tests] Parsing results.
      [Cucumber Tests] parsing cucumber.json
      ERROR: Failed to archive cucumber reports
      java.io.IOException: Failed to parse Cucumber JSON
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:99)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:44)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl$ParseResultCallable.invoke(DefaultTestResultParserImpl.java:161)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl$ParseResultCallable.invoke(DefaultTestResultParserImpl.java:107)
      	at hudson.FilePath.act(FilePath.java:914)
      	at hudson.FilePath.act(FilePath.java:887)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl.parse(DefaultTestResultParserImpl.java:101)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:116)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberTestResultArchiver.perform(CucumberTestResultArchiver.java:93)
      	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:785)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:757)
      	at hudson.model.Build$BuildExecution.post2(Build.java:183)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:706)
      	at hudson.model.Run.execute(Run.java:1703)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:231)
      Caused by: org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberModelException: Step received before previous step handled!
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.GherkinCallback.step(GherkinCallback.java:160)
      	at gherkin.formatter.model.Step.replay(Step.java:68)
      	at gherkin.JSONParser.step(JSONParser.java:106)
      	at gherkin.JSONParser.parse(JSONParser.java:51)
      	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:94)
      	... 17 more
      Build step 'Publish Cucumber test result report' changed build result to FAILURE
      Finished: FAILURE
      

          [JENKINS-21835] Cucumber Test Result Plugin crashes on malformed output in Scenario Outline results produced by some tools

          James Nord added a comment -

          For Option 3 - see https://github.com/cucumber/cucumber/pull/700 which may fix this issue (untested).

          James Nord added a comment - For Option 3 - see https://github.com/cucumber/cucumber/pull/700 which may fix this issue (untested).

          Is there any news?

          Kanstantsin Shautsou added a comment - Is there any news?

          teilo i extended plugin with additional options. If developer unsure about features - then they should be configurable .
          https://github.com/jenkinsci/cucumber-testresult-plugin/pull/2

          • Changed java stacktrace to pretty build Failure with additional information.
          • Removed System.out.println() - never use it plugins. There is "manage jenkins" -> "system logger".
          • Error message describes what is really wrong and notes this issues. So after merge close this issue.
          • Added aditional checkbox for ignoring not existed step results - this fixed my case, so now i may use it.

          PR only has no tests atm...

          Kanstantsin Shautsou added a comment - teilo i extended plugin with additional options. If developer unsure about features - then they should be configurable . https://github.com/jenkinsci/cucumber-testresult-plugin/pull/2 Changed java stacktrace to pretty build Failure with additional information. Removed System.out.println() - never use it plugins. There is "manage jenkins" -> "system logger". Error message describes what is really wrong and notes this issues. So after merge close this issue. Added aditional checkbox for ignoring not existed step results - this fixed my case, so now i may use it. PR only has no tests atm...

          James Nord added a comment -

          Have you tried with a fixed version of cucumber instead?
          This has apparently been fixed upstream.

          James Nord added a comment - Have you tried with a fixed version of cucumber instead? This has apparently been fixed upstream.

          Of course tried. We are using grunt-cuke-tree that we can't update atm. (Tt's not just bump version in package.json).

          Kanstantsin Shautsou added a comment - Of course tried. We are using grunt-cuke-tree that we can't update atm. (Tt's not just bump version in package.json).

          Code changed in jenkins
          User: Kanstantsin Shautsou
          Path:
          src/main/java/org/jenkinsci/plugins/cucumber/jsontestsupport/CucumberJSONParser.java
          src/main/java/org/jenkinsci/plugins/cucumber/jsontestsupport/GherkinCallback.java
          http://jenkins-ci.org/commit/cucumber-testresult-plugin/14c313da7f51527df7468f39140a27ff44ffc33c
          Log:
          JENKINS-21835 Don't print stacktrace. Print friendly log.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kanstantsin Shautsou Path: src/main/java/org/jenkinsci/plugins/cucumber/jsontestsupport/CucumberJSONParser.java src/main/java/org/jenkinsci/plugins/cucumber/jsontestsupport/GherkinCallback.java http://jenkins-ci.org/commit/cucumber-testresult-plugin/14c313da7f51527df7468f39140a27ff44ffc33c Log: JENKINS-21835 Don't print stacktrace. Print friendly log.

          James Nord added a comment -

          workaround enabled.

          Enabling this may cause tests that have failed to not cause the build to be marked as unstable - but there is no information about the result so there is nothing else that can be done.

          The cause of the issue has fixes upstream - so it is a matter of getting those fixes into all the relevant tools to get a real fix.

          James Nord added a comment - workaround enabled. Enabling this may cause tests that have failed to not cause the build to be marked as unstable - but there is no information about the result so there is nothing else that can be done. The cause of the issue has fixes upstream - so it is a matter of getting those fixes into all the relevant tools to get a real fix.

          Daniel Serodio added a comment - - edited

          Unfortunately https://github.com/cucumber/cucumber/issues/700 is a 404. Does anyone know which issue that was, and/or which Cucumber version (if any) contains the fix?

          Daniel Serodio added a comment - - edited Unfortunately https://github.com/cucumber/cucumber/issues/700 is a 404. Does anyone know which issue that was, and/or which Cucumber version (if any) contains the fix?

          Daniel Serodio added a comment - - edited

          Ok, the cucumber repo was renamed to cucumber-ruby and gherkin was renamed to gherkin2. The links above are now:

          https://github.com/cucumber/cucumber-ruby/issues/700

          https://github.com/cucumber/gherkin2/issues/165

           

          Daniel Serodio added a comment - - edited Ok, the cucumber repo was renamed to cucumber-ruby and gherkin was renamed to gherkin2. The links above are now: https://github.com/cucumber/cucumber-ruby/issues/700 https://github.com/cucumber/gherkin2/issues/165  

          James Nord added a comment -

          dserodio thanks for the detective work. If you find the versions with fixes can you comment here?

          James Nord added a comment - dserodio thanks for the detective work. If you find the versions with fixes can you comment here?

            Unassigned Unassigned
            jwmach1 James Wilson
            Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: