Daniel expects problems need to be addressed during the beta period if possible.

Root cause of this: the UI was changed, and broke many of the ATH tests. [Guess] Selenium uses page object to describe the page, and many of the tests rely on ID that have changed.

[Suggest Tom or Keith fix up the tests]

Wouldn't it make sense to run the ATH after each UI change to ensure that nothing has been broken? I would also encourage each developer to add new ATH test cases for each of the new UI features that have been introduced...

drulli Definitely something we need to improve. ATH is not visible enough. We could make passing ATH in an ATH snapshot/PR a condition for UI changes. Ideally ATH could be run for all pull requests. WDYT?

For now, I consider this a 2.0 blocker that needs fixing. So from the POV of actual Jenkins releases, we'll still be on time

Letting the ATH run for each PR would be ideal, of course. (I'm not sure if this is feasible since one run of the ATH is quite time consuming...)

Since all core changes require a code review it would make sense to make at least one ATH test case mandatory to get a 'bee' label in such a review (and a couple of unit tests

So there's 2 immediately obvious issues ...

1. Install wizard needs to be disabled for all/most tests.
2. No bundled plugins. Looks like we need to get these in there somehow.

#2 above is the trickier one of fix. Maybe we need the acceptance test harness to get all of these plugins automatically and install them in the $JENKINS_HOME/plugins ?

What I've been doing ...

1. Modified Jenkins core 2.0 to allow for an env based install-state override, allowing us to bypass the install wizard in the ATH (using LocalController.commonLaunchEnv to set the override in the test harness)
2. Testing against alpha4 for now because the new-item page changes cause tests to fail immediately.

Modified Jenkins core 2.0 to allow for an env based install-state override

Is this on top of the existing jenkins.install.runSetupWizard and hudson.Main.development System properties that control setup wizard appearance? Can they maybe be consolidated?

tfennelly, thanks for looking at it. Can you push what you have so we can collaborate?

olivergondza Hey Oliver. I pushed the small few changes I have made to ...

• https://github.com/jenkinsci/jenkins/pull/2196 (don't worry danielbeck ... we're not going to merge it )
• https://github.com/jenkinsci/acceptance-test-harness/pull/87

olivergondza I'm really stuck on an issue with the selection of the "Execute shell" option from the "Add build step" dropdown. I'm seeing an issue with it on WarningsPluginTest.should_detect_warnings_of_multiple_compilers_in_console_freestyle, where it's unable to find .//A[@href][@id='Execute shell' or normalize-space(.)='Execute shell' or @title='Execute shell' or .//img[@alt='Execute shell']].

When I run it in my IDE, setting a breakpoint where the findElement fails, I'm able to go into the dev console in the browser and execute...

$x(".//A[@href][@id='Execute shell' or normalize-space(.)='Execute shell' or @title='Execute shell' or .//img[@alt='Execute shell']]")

Sure enough, when I execute this the first time it returns an empy array. But ... when I manually click on the "Add build step" dropdown I can actually see the "Execute shell" option + when I then go back to the dev console and execute the above xpath it actually works the second time i.e. it finds it.

See maybe it's some sort of timing issue? Still not sure how the config page changes have caused this.

olivergondza this is funky I think part of the problem is that the Add build step dropdown button is no longer in view after a few steps have been added.

I see there are some features getting into our way. Security by default seems to be the first one. Is there any way to launch Jenkins without that or inject the key somehow? Cc danielbeck.

https://jenkins.ci.cloudbees.com/job/core/job/acceptance-test-harness-2.0/59/console

olivergondza I think I resolved my earlier issue with the Add build step. See 0b9dfaf. Maybe you know of the right way of doing this or if it's actually been caused by something else that can be fixed?

As for the security error you are seeing in the CI build ... is that not because it's not running against a Jenkins 2.0 that has https://github.com/jenkinsci/jenkins/pull/2196 ??

Code changed in jenkins
User: Oliver Gondža
Path:
core/src/main/java/jenkins/install/InstallUtil.java
http://jenkins-ci.org/commit/jenkins/9999516aa826dd34986204a508a6bda7124e889d
Log:
Merge pull request #2196 from tfennelly/2.0-JENKINS-33730

JENKINS-33730 Fix ATH for Jenkins 2.0

Compare: https://github.com/jenkinsci/jenkins/compare/4d01c07db60d...9999516aa826

This got the most of the ATH running. However, there are still things to fix. A lot of tests require presence of formerly bundled plugins. Th easiest fix seems to be just add the annotations or alternatively to make the dependency resolution mechanism aware of unbundling in similar way we do in core.

tfennelly, working on tools config page extraction support ...

olivergondza Another PR: https://github.com/jenkinsci/acceptance-test-harness/pull/88

Should fix some of those scrolling issues + sort out the missing dependencies.

olivergondza hey Oliver ... any luck extracting the tools ?

I just tried doing it myself here, but not really sure what I'm at since I don't know this code very well. I'm using the PmdPluginTest and it gets a bit further than it was getting, but stalls (and eventually times out) when doing the build just after extracting the maven zip. Will look again in the morning if you haven't had time to look into it.

olivergondza also note that I had to add a temporary tweak to make the tooling config go to the right page since 2.0-beta is actually older than VersionNumber("2"). We can reverty to what you added after the 2.0 release.

Code changed in jenkins
User: Oliver Gondža
Path:
src/main/java/org/jenkinsci/test/acceptance/controller/LocalController.java
src/main/java/org/jenkinsci/test/acceptance/po/ToolInstallation.java
src/main/java/org/jenkinsci/test/acceptance/selenium/Scroller.java
src/main/resources/org/jenkinsci/test/acceptance/selenium/scroller.js
http://jenkins-ci.org/commit/acceptance-test-harness/c1512e4da9dc3b4dd701988dae103eaec405ec25
Log:
Merge pull request #88 from tfennelly/2.0-JENKINS-33730

JENKINS-33730 More ATH tweaks for Jenkins 2.0

Compare: https://github.com/jenkinsci/acceptance-test-harness/compare/4de2dfc72cf2...c1512e4da9dc

I should be done with this today.

tfennelly, there seems to be some problem with Control#selectDropdownMenu(Unable to add build step, post build step, tool installation) but I am not able to reproduce that locally. I speculate it might take a little longer to appear in 2.0 so we are clicking the item before the menu appear but it is just a wild guess.

olivergondza where are you seeing the issue with Control#selectDropdownMenu ? Which test?

tfennelly for instance https://jenkins.ci.cloudbees.com/job/core/job/acceptance-test-harness-2.0/lastCompletedBuild/testReport/plugins/AnalysisCollectorPluginTest/should_compute_all_new_open_tasks/

Hmmm I've run that a few times now and it passed each time.

I am not able to reproduce that locally, either.

Closing this now. The only failures/errors I get now are tests that were already in fail/error when running against 1.x.