James Nord added a comment - 2016-02-17 21:32 if your tests assertion is not correct then the build is correctly marked as unstable. if your tests error (that is that they failed to run due to glue failure or are pending/undefined) then this is classed as an error in your tests (rather than being unstable - which is that the tests are fine but the result is not expected) and this is currently behaving correctly. so unless you can supply a json file that shows contrary to this then this is not a defect - but possibly a dupe of JENKINS-30083

John Cunliffe added a comment - 2016-02-18 08:35 it is quite possible, that I am using it wrongly, however I can not yet see how. please find the troublesome console output below: Updating selenium standalone downloading https://selenium-release.storage.googleapis.com/2.51/selenium-server-standalone-2.51.0.jar... Updating chromedriver downloading https://chromedriver.storage.googleapis.com/2.21/chromedriver_linux64.zip... chromedriver_2.21.zip downloaded to /home/portos/workspace/front.berater-build-playground/com.dvag.portos.front.berater/node_modules/protractor/selenium/chromedriver_2.21.zip selenium-server-standalone-2.51.0.jar downloaded to /home/portos/workspace/front.berater-build-playground/com.dvag.portos.front.berater/node_modules/protractor/selenium/selenium-server-standalone-2.51.0.jar [18:45:41] Server started http://localhost:5555 Starting selenium standalone server... [launcher] Running 1 instances of WebDriver Selenium standalone server started at http://10.61.9.140:34994/wd/hub Funktionalität: B01 Mein Vermögensberater Es soll ein Bereich "Mein Vermögensberater" existieren, in dem die Daten des Vermögensberaters (VB) angezeigt werden. @Abnahme Szenariogrundriss: Beispiel anzeigen Angenommen ich befinde mich auf Beispielseite mit der ID "8152200"  Dann wird "Max" im Feld "Vorname" des Beispiels angezeigt AssertionError: expected '' to equal ' Max' Dann wird "Mustermann" im Feld "Name" des Beispiels angezeigt Failing scenarios: features/BeraterKomplett.feature:36 # Scenario: Berater in der Komplettansicht anzeigen 1 scenario (1 failed) 3 steps (1 failed, 1 skipped, 1 passed) 0m02.210s Shutting down selenium standalone server. [launcher] 0 instance(s) of WebDriver still running [launcher] phantomjs #1 failed 1 test(s) [launcher] overall: 1 failed spec(s) [launcher] Process exited with error code 1 [18:45:54] Finished 'protractor.start' after 15 s [18:45:54] Starting 'build.docs' ... [18:45:55] Server stopped (...) [Cucumber Tests] Parsing results. [Cucumber Tests] parsing cucumber-abnahme.json Build step 'Publish Cucumber test result report' changed build result to FAILURE Warning: you have no plugins providing access control for builds, so falling back to legacy behavior of permitting any downstream builds to be triggered Finished: FAILURE This generates the json displayed below: [ { "id" : "B01-Mein-Beispiel" , "name" : "B01 Mein Beispiel" , "description" : "Es soll ein Bereich \" Mein Beispiel\ " existieren." , "line" : 3, "keyword" : "Funktionalität" , "uri" : "/home/test/workspace/playground/features/MeinBeispiel.feature" , "elements" : [ { "name" : "Ich bin als Kunde angemeldet." , "keyword" : "Grundlage" , "description" : "", "type" : "background" , "line" : 6 }, { "name" : "Beispiel anzeigen" , "id" : "B01-Mein-Beispiel;beispiel-anzeigen" , "line" : 36, "keyword" : "Scenario" , "description" : "", "type" : "scenario" , "tags" : [ { "name" : "@Abnahme" , "line" : 8 } ], "steps" : [ { "name" : "ich befinde mich auf Beispielseite mit der ID \" 8152200\"", "line" : 10, "keyword" : "Angenommen " , "result" : { "status" : "passed" , "duration" : 1527468401 }, "match" : {} }, { "name" : "wird \" Max\ " im Feld \" Vorname\ " des Beispiels angezeigt" , "line" : 12, "keyword" : "Dann " , "result" : { "status" : "failed" , "duration" : 219161014, "error_message" : "AssertionError: expected '' to equal ' Max'" }, "match" : {} }, { "name" : "wird \" Mustermann\ " im Feld \" Name\ " des Beispiels angezeigt" , "line" : 13, "keyword" : "Dann " , "result" : { "status" : "skipped" }, "match" : {} }, { "keyword" : "After " , "hidden" : true , "result" : { "status" : "passed" , "duration" : 462568284 }, "match" : {}, "embeddings" : [ { "mime_type" : "image/png" , "data" : "(some correct base64-encoded image" } ] } ] } ] } ] Please note that I have taken out all skipped tests and slightly changed the testing value for data protection reasons. the crucial bits - the reason why it was marked a failure due to assertion error, which I believe you told me to result in a unstable state - remain unchanged. The following code generates the assertion error: import * as chaiAsPromised from "chai-as-promised" ; import * as Chai from "chai" ; import CallbackStepDefinition = cucumber.CallbackStepDefinition; Chai.use(chaiAsPromised); module.exports = function () { let scenario: cucumber.StepDefinitions = this ; scenario.Then(/^wird "([^" ]*) " im Feld " ([^ "]*)" des Beispiels angezeigt$/, function (value: string | CallbackStepDefinition, fieldID: string | CallbackStepDefinition, next: CallbackStepDefinition): void { Chai.expect(element(by.id( "qa-" + <string>fieldID)).getText()).to.eventually.equal(value).and.notify(next); }); }

James Nord added a comment - 2016-02-18 10:20 Right so in this case every single feature in every single scenario failed. As this is highly unusual [1] (you would expect at least one scenario to pass) the plugin treats this as a failure. Of you have st least one passing scenario then faiures would mark the build as unstable. [1] something like the SUT is not alive.

John Cunliffe added a comment - 2016-02-18 10:47 This makes sense - I will double check and assuming that a single passing test will result in an unstable status, my problems will be solved.