Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-21835

Cucumber Test Result Plugin crashes on malformed output in Scenario Outline results produced by some tools

XMLWordPrintable

      What we did:
      Ran minimal cucumber feature test with Jenkins:

      Feature
      Feature: Sample Test
  Scenario Outline: Parsing scenarios with multiple examples
    Given I navigate to the root page
    Then I see the text 'home page'
    Examples:
    | a | b |
    | 1 | 2 |

      By a shell build step:

      shell build
      #!/bin/bash
# Set up RVM (Ruby 2.0)
. ~/.bashrc
set -e
CUCUMBER_OPTS='-f json -o cucumber.json features/sample.feature' rake cucumber

      What we got:
      [::1::]

      What we expected:
      No error, parsing of cucumber.json containing results from scenario outlines.

      [::1::]

      Output
      /var/lib/jenkins/.rvm/rubies/ruby-2.0.0-p353/bin/ruby -S bundle exec cucumber -f json -o cucumber.json features/sample.feature
Using the default profile...
Feature: Sample Test

  Scenario Outline: Parsing scenarios with multiple examples # features/sample.feature:2
    Given I navigate to the root page                        # features/step_definitions/navigation_steps.rb:15
    Then I see the text 'home page'                          # features/step_definitions/study_admin.rb:23

    Examples: 
      | a | b |
      | 1 | 2 |

1 scenario (1 passed)
2 steps (2 passed)
0m1.080s
Xvfb stopping
Archiving artifacts
[Cucumber Tests] Parsing results.
[Cucumber Tests] parsing cucumber.json
ERROR: Failed to archive cucumber reports
java.io.IOException: Failed to parse Cucumber JSON
	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:99)
	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:44)
	at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl$ParseResultCallable.invoke(DefaultTestResultParserImpl.java:161)
	at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl$ParseResultCallable.invoke(DefaultTestResultParserImpl.java:107)
	at hudson.FilePath.act(FilePath.java:914)
	at hudson.FilePath.act(FilePath.java:887)
	at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl.parse(DefaultTestResultParserImpl.java:101)
	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:116)
	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberTestResultArchiver.perform(CucumberTestResultArchiver.java:93)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:785)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:757)
	at hudson.model.Build$BuildExecution.post2(Build.java:183)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:706)
	at hudson.model.Run.execute(Run.java:1703)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:231)
Caused by: org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberModelException: Step received before previous step handled!
	at org.jenkinsci.plugins.cucumber.jsontestsupport.GherkinCallback.step(GherkinCallback.java:160)
	at gherkin.formatter.model.Step.replay(Step.java:68)
	at gherkin.JSONParser.step(JSONParser.java:106)
	at gherkin.JSONParser.parse(JSONParser.java:51)
	at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:94)
	... 17 more
Build step 'Publish Cucumber test result report' changed build result to FAILURE
Finished: FAILURE

        1. cucumber_with_outline_and_basic_scenario.json
          2 kB
        2. cucumber.json
          1 kB

          [JENKINS-21835] Cucumber Test Result Plugin crashes on malformed output in Scenario Outline results produced by some tools

          James Wilson created issue -

          James Wilson added a comment -

          Added formatting

          James Wilson added a comment - Added formatting
          James Wilson made changes -
          Description Original: What we did:
          Ran minimal cucumber feature test with Jenkins

          Feature: Sample Test
            Scenario Outline: Parsing scenarios with multiple examples
              Given I navigate to the root page
              Then I see the text 'home page'
              Examples:
              | a | b |
              | 1 | 2 |

          By a shell build step:
          #!/bin/bash
          # Set up RVM (Ruby 2.0)
          . ~/.bashrc
          set -e
          CUCUMBER_OPTS='-f json -o cucumber.json features/sample.feature' rake cucumber

          What we got:
          [::1::]

          What we expected:
          No error, parsing of cucumber.json containing results from scenario outlines.

          [::1::]
          /var/lib/jenkins/.rvm/rubies/ruby-2.0.0-p353/bin/ruby -S bundle exec cucumber -f json -o cucumber.json features/sample.feature
          Using the default profile...
          Feature: Sample Test

            Scenario Outline: Parsing scenarios with multiple examples # features/sample.feature:2
              Given I navigate to the root page # features/step_definitions/navigation_steps.rb:15
              Then I see the text 'home page' # features/step_definitions/study_admin.rb:23

              Examples:
                | a | b |
                | 1 | 2 |

          1 scenario (1 passed)
          2 steps (2 passed)
          0m1.080s
          Xvfb stopping
          Archiving artifacts
          [Cucumber Tests] Parsing results.
          [Cucumber Tests] parsing cucumber.json
          ERROR: Failed to archive cucumber reports
          java.io.IOException: Failed to parse Cucumber JSON
          at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:99)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:44)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl$ParseResultCallable.invoke(DefaultTestResultParserImpl.java:161)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl$ParseResultCallable.invoke(DefaultTestResultParserImpl.java:107)
          at hudson.FilePath.act(FilePath.java:914)
          at hudson.FilePath.act(FilePath.java:887)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl.parse(DefaultTestResultParserImpl.java:101)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:116)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberTestResultArchiver.perform(CucumberTestResultArchiver.java:93)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:785)
          at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:757)
          at hudson.model.Build$BuildExecution.post2(Build.java:183)
          at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:706)
          at hudson.model.Run.execute(Run.java:1703)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:231)
          Caused by: org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberModelException: Step received before previous step handled!
          at org.jenkinsci.plugins.cucumber.jsontestsupport.GherkinCallback.step(GherkinCallback.java:160)
          at gherkin.formatter.model.Step.replay(Step.java:68)
          at gherkin.JSONParser.step(JSONParser.java:106)
          at gherkin.JSONParser.parse(JSONParser.java:51)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:94)
          ... 17 more
          Build step 'Publish Cucumber test result report' changed build result to FAILURE
          Finished: FAILURE           		New: What we did:
          Ran minimal cucumber feature test with Jenkins:
          {code:title=Feature}
          Feature: Sample Test
            Scenario Outline: Parsing scenarios with multiple examples
              Given I navigate to the root page
              Then I see the text 'home page'
              Examples:
              | a | b |
              | 1 | 2 |
          {code}

          By a shell build step:
          {code:title=shell build}
          #!/bin/bash
          # Set up RVM (Ruby 2.0)
          . ~/.bashrc
          set -e
          CUCUMBER_OPTS='-f json -o cucumber.json features/sample.feature' rake cucumber
          {code}

          What we got:
          [::1::]

          What we expected:
          No error, parsing of cucumber.json containing results from scenario outlines.

          [::1::]
          {code:title=Output}
          /var/lib/jenkins/.rvm/rubies/ruby-2.0.0-p353/bin/ruby -S bundle exec cucumber -f json -o cucumber.json features/sample.feature
          Using the default profile...
          Feature: Sample Test

            Scenario Outline: Parsing scenarios with multiple examples # features/sample.feature:2
              Given I navigate to the root page # features/step_definitions/navigation_steps.rb:15
              Then I see the text 'home page' # features/step_definitions/study_admin.rb:23

              Examples:
                | a | b |
                | 1 | 2 |

          1 scenario (1 passed)
          2 steps (2 passed)
          0m1.080s
          Xvfb stopping
          Archiving artifacts
          [Cucumber Tests] Parsing results.
          [Cucumber Tests] parsing cucumber.json
          ERROR: Failed to archive cucumber reports
          java.io.IOException: Failed to parse Cucumber JSON
          at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:99)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:44)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl$ParseResultCallable.invoke(DefaultTestResultParserImpl.java:161)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl$ParseResultCallable.invoke(DefaultTestResultParserImpl.java:107)
          at hudson.FilePath.act(FilePath.java:914)
          at hudson.FilePath.act(FilePath.java:887)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.DefaultTestResultParserImpl.parse(DefaultTestResultParserImpl.java:101)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:116)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberTestResultArchiver.perform(CucumberTestResultArchiver.java:93)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:785)
          at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:757)
          at hudson.model.Build$BuildExecution.post2(Build.java:183)
          at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:706)
          at hudson.model.Run.execute(Run.java:1703)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
          at hudson.model.ResourceController.execute(ResourceController.java:88)
          at hudson.model.Executor.run(Executor.java:231)
          Caused by: org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberModelException: Step received before previous step handled!
          at org.jenkinsci.plugins.cucumber.jsontestsupport.GherkinCallback.step(GherkinCallback.java:160)
          at gherkin.formatter.model.Step.replay(Step.java:68)
          at gherkin.JSONParser.step(JSONParser.java:106)
          at gherkin.JSONParser.parse(JSONParser.java:51)
          at org.jenkinsci.plugins.cucumber.jsontestsupport.CucumberJSONParser.parse(CucumberJSONParser.java:94)
          ... 17 more
          Build step 'Publish Cucumber test result report' changed build result to FAILURE
          Finished: FAILURE
          {code}

          Sandro Simas added a comment -

          I'm getting this error too. Does anybody knows how to solve that ?

          Sandro Simas added a comment - I'm getting this error too. Does anybody knows how to solve that ?

          Leidiane Vieira added a comment -

          Guys, someone could solved this issue?
          Thanks in advance...

          Leidiane Vieira added a comment - Guys, someone could solved this issue? Thanks in advance...

          James Wilson added a comment -

          I noticed a problem with the attached cucumber.json. It seems that if a Feature file only has an outline, there are no results saved in the file. So I wrote an example Feature file that would have basic scenario and a failing scenario along with the Scenario Outline.

          Feature: Sample test
  Scenario Outline: Parsing scenarios with multiple examples
    Given I navigate to the home page
    Then I see the text 'Home'
  Examples:
      | a | b |
      | 1 | 2 |
  Scenario: Basic
    Given I navigate to the home page
    Then I see the text 'Home'
  Scenario: Basic failure
    Given I navigate to the home page
    Then I see the text 'Hacienda'

          attaching a new json file: cucumber_with_outline_and_basic_scenario.json

          James Wilson added a comment - I noticed a problem with the attached cucumber.json. It seems that if a Feature file only has an outline, there are no results saved in the file. So I wrote an example Feature file that would have basic scenario and a failing scenario along with the Scenario Outline. Feature: Sample test Scenario Outline: Parsing scenarios with multiple examples Given I navigate to the home page Then I see the text 'Home' Examples: | a | b | | 1 | 2 | Scenario: Basic Given I navigate to the home page Then I see the text 'Home' Scenario: Basic failure Given I navigate to the home page Then I see the text 'Hacienda' attaching a new json file: cucumber_with_outline_and_basic_scenario.json

          James Wilson added a comment -

          json result file from a Feature file that has one Scenario Outline and 2 Scenarios.

          James Wilson added a comment - json result file from a Feature file that has one Scenario Outline and 2 Scenarios.
          James Wilson made changes -
          Attachment New: cucumber_with_outline_and_basic_scenario.json [ 25509 ]

          James Wilson added a comment -

          There is a bug [1] in gherkin at the core of this bug. Apparently Gherkin json output does not contain results for scenario outlines. So this bug will not be able to show the results of scenario outlines. But it can better handle the ScenarioOutline event in the GherkinCallback to not fail with exception if the json is missing results (work around the year old bug [1] in Gherkin)

          [1] https://github.com/cucumber/gherkin/issues/165

          James Wilson added a comment - There is a bug [1] in gherkin at the core of this bug. Apparently Gherkin json output does not contain results for scenario outlines. So this bug will not be able to show the results of scenario outlines. But it can better handle the ScenarioOutline event in the GherkinCallback to not fail with exception if the json is missing results (work around the year old bug [1] in Gherkin) [1] https://github.com/cucumber/gherkin/issues/165

          Leidiane Vieira added a comment -

          Hi James, thanks for the answer. I understood the point, but its sorry that I have to use the "scenario" to get the report i need, cuz my feature just work fine today without this. I saw that the link of the issue was 2 years ago, may by this time they have solved, do you know?

          Thanks for all the information and your time spent.

          Leidiane Vieira added a comment - Hi James, thanks for the answer. I understood the point, but its sorry that I have to use the "scenario" to get the report i need, cuz my feature just work fine today without this. I saw that the link of the issue was 2 years ago, may by this time they have solved, do you know? Thanks for all the information and your time spent.

            Unassigned Unassigned
            jwmach1 James Wilson
            Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: