Major There are too many @RandomlyFails tests. SemaphoreStep should be used more consistently in place of WatchYourStep and waiting for the execution to suspend. (Already prototyped in WorkflowTest.env.)
Also any build logs should be streamed immediately to stderr, rather than forcing the test to include the current log in every assertion message observed to fail. (In combination with SemaphoreStep, this should also more reliably flush recent output: WorkflowRun flushes logs when new steps are run.)
- is related to
-
JENKINS-26398 Fix createSlave for RestartableJenkinsRule
-
- Resolved
-
-
JENKINS-26399 Utility method in JenkinsRule to wait for build to complete
-
- Resolved
-
- links to
[JENKINS-25975] More reliable test infrastructure
Implementing some of these things in PR 59 just to get things to pass on Windows, which is easier using SemaphoreStep than WatchYourStep because we do not need to deal with local file paths embedded in the script.
One class of random failure I think I have diagnosed: CpsThreadGroup.notifyNewHead notifies listeners asynchronously, so WorkflowRun.copyLogs does not run instantly after a step completes, and so the build log may not immediately show its output. CpsFlowExecution.waitForSuspension seems to solve that.
Code changed in jenkins
User: Jesse Glick
Path:
test/src/main/java/org/jvnet/hudson/test/BuildWatcher.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
http://jenkins-ci.org/commit/jenkins/9ac2257d0a3fa86928267a82d69d55c5a4e423c1
Log:
[FIXED JENKINS-26399] JENKINS-25975 Merged #1609.
Compare: https://github.com/jenkinsci/jenkins/compare/b02e71e3dfcb...9ac2257d0a3f
I am occasionally getting test failures whereby a sh step is evidently run but its output is not included in the log. I suspect a race condition between the GraphListener calling copyLogs and the one calling PrintStream.close when DefaultStepContext makes a LogActionImpl.
Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowTest.java
durable-task-step/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java
http://jenkins-ci.org/commit/workflow-plugin/27e5b23046ae4e913edff0fa05256fdfafff0029
Log:
JENKINS-25975 Seem to have fixed a race condition whereby quick `sh` steps would sometimes not print all of their output.
Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/SingleJobTestBase.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowRunTest.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowTest.java
support/src/test/java/org/jenkinsci/plugins/workflow/test/steps/WatchYourStep.java
http://jenkins-ci.org/commit/workflow-plugin/d3ba64ddd6eaaacad401624d5b17809ebbec9b30
Log:
[FIXED JENKINS-25975] Finally able to delete WatchYourStep; using SemaphoreStep instead.
Code changed in jenkins
User: Jesse Glick
Path:
test/src/main/java/org/jvnet/hudson/test/BuildWatcher.java
test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java
http://jenkins-ci.org/commit/jenkins-test-harness/fc7807ec16c4c1d495f293332bf5890f2a3fa282
Log:
[FIXED JENKINS-26399] JENKINS-25975 Merged #1609.
Originally-Committed-As: 9ac2257d0a3fa86928267a82d69d55c5a4e423c1
Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/47ff7954de7713b74ed094a37863046cdedbee28
Log:
JENKINS-25975 Seem to have fixed a race condition whereby quick `sh` steps would sometimes not print all of their output.
Originally-Committed-As: 27e5b23046ae4e913edff0fa05256fdfafff0029
Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/SingleJobTestBase.java
aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowTest.java
http://jenkins-ci.org/commit/workflow-cps-plugin/78b05a55574421d0d9383eab50f926d3954c7dd6
Log:
[FIXED JENKINS-25975] Finally able to delete WatchYourStep; using SemaphoreStep instead.
Originally-Committed-As: d3ba64ddd6eaaacad401624d5b17809ebbec9b30
Code changed in jenkins
User: Jesse Glick
Path:
aggregator/src/test/java/org/jenkinsci/plugins/workflow/WorkflowRunTest.java
http://jenkins-ci.org/commit/workflow-job-plugin/360c2d2337098108e8c3ecee329e8cc0fd4512f3
Log:
[FIXED JENKINS-25975] Finally able to delete WatchYourStep; using SemaphoreStep instead.
Originally-Committed-As: d3ba64ddd6eaaacad401624d5b17809ebbec9b30
Note to self: if deleting WatchYourStep, can also close PR 10.