There is a SmokeTest category in the ATH which seems interesting, it covers:

• FreeStyle Job Creation
• Periodical builds of FreeStyle Jobs
• Parameterized builds of FreeStyle Jobs
• Build history correctness
• User log in and log out capabilities
• Execute system script
• Slave creation and tie jobs to specific labels
• Remote build trigger
• View creation and reg exp 
• Jobs renaming

The entire category run (including checkout, docker images run and all the needed set up) takes between 10 and 12 min in my test instance, according to Javadoc is meant to not have more than 10 tests and has been there for some quite time. DISCLAIMER: I am not running in my local instance using the ATH docker image, will do that after I have some data with plain runs just to compare results

Below is a list of the concrete tests executed and some stability measures

• FreestyleJobTest#archiveArtifacts (No failures recorded in history)
• FreestyleJobTest#buildParametrized (No failures recorded in history)
• FreestyleJobTest#buildPeriodically (No failures recorded in history)
• FreestyleJobTest#doNotDiscardSuccessfulBuilds (No failures recorded in history)
• JenkinsDatabaseSecurityRealmTest#login_and_logout (No failures recorded in history)
• ScriptTest#execute_system_script  (Not very stable according to  history and failing now in ci.jenkins.io even if working  perfectly fine in my test instance)
• SlaveTest#tie_job_to_specified_label (Not very stable according to history)
• TriggerRemoteBuildsTest#triggerBuildRemotely (No failures recorded in history)
• ViewTest#findJobThroughRegexp (No failures recorded in history)
• ViewTest#renameJob (No failures recorded in history)

I am currently performing runs of those tests on my test instance every half an hour to get more data about failures and timings, if  no problems are found I would recommend to create a new category in the ATH mostly based on the SmokeTest one without the tests that have shown any failure.

Will update this with data after 24 hours of runs. Any comment so far vilacides rtyler olivergondza ?

Thanks for looking into this. I have 2 comments:

• Currently, the ATH container is build every time. If we can get it pushed to the registry it can speed the execution significantly.
• Once you will purge this off flaky test, fee free to update the SmokeTest category instead of creating a new one.

olivergondza My main concern about using a released version of the ATH container instead of building from source is to make sure we use the latest fixes (same reason why we use master of the ATH itself instead of a released version) In your experience is this concern valid or the ATH container is stable enough to use a released version without worries?

Note that on first stages I am not using the ATH container (that means is not really needed), just starting up the selenium firefox one and using it, but I want to change that so for example if the ATH updates the selenium version I expect the container to be also updated and hence no changes in the job itself are needed

JenkinsDatabaseSecurityRealmTest#login_and_logout has shown some instability failing 6 times in the last 37 runs. The reason is always the same and I suspect that is a race condition after login, I am going to try to fix the test as I believe is important to make sure users can log in and out

expected:<jenkins-acceptance-tests-user (Full Name)> but was:<null (null)>

Testing now my attempt to remove flakiness, code is here

So, my initial set of changes did not worked, but it has confirmed the issue, for some reason sometimes the login takes more than ten seconds...

Timed out after 10 seconds: Element matching By.cssSelector: a[href='/user/jenkins-acceptance-tests-user'] is present

The changes I made did not entirely solve the issue but they have greatly reduced the frequency of it, two failures in 47 runs... I am going to continue with the same approach I suspect now the problem is a race condition related to the click on login button and the load of the next page

Gathering data with my latest changes....

Given the amount of integration testing related to security and login I am going to dedicate only today to this test, if I am not able to properly fix it (may it be a race condition on the UI, something related to how Jenkins notifies of login events, or a cache problem) my proposal will be to eliminate the test from the SmokeTest category   

Confirmed that I have not been able to remove the flakiness of JenkinsDatabaseSecurityRealmTest#login_and_logout in a reasonable amount of time, so my proposal is to remove it from the `SmokeTest` category, create a ticket to try to fix it properly and add it again when fixed

ScriptTest and SlaveTest have not shown flakiness in my runs, BUT they have shown some lack of robustness against changes in the UI in the history, so is better to remove them from the list for the first implementation

So my proposal is to run the SmokeTest Category "fixed" by #PR-410

Code changed in jenkins
User: Raul Arabaolaza
Path:
src/test/java/core/JenkinsDatabaseSecurityRealmTest.java
http://jenkins-ci.org/commit/acceptance-test-harness/3413c9ffc7093d014cf81327832d97545adfbf0c
Log:
JENKINS-49263 Remove login test from SmokeTest until is fixed

Code changed in jenkins
User: Raul Arabaolaza
Path:
src/test/java/core/ScriptTest.java
http://jenkins-ci.org/commit/acceptance-test-harness/f07816a49d38c7416e20865f02bebe76eb6fd467
Log:
JENKINS-49263 Remove ScriptTest from SmokeTest

• As it has shown symptoms of lack of robustness

Code changed in jenkins
User: Raul Arabaolaza
Path:
src/test/java/core/SlaveTest.java
http://jenkins-ci.org/commit/acceptance-test-harness/9098f4a8e368fe3daa3db98691781fdc10d9a459
Log:
JENKINS-49263 Remove SlaveTest from SmokeTest

• AS it has shown symptoms of lack of robustness

Code changed in jenkins
User: Oliver Gondža
Path:
src/test/java/core/JenkinsDatabaseSecurityRealmTest.java
src/test/java/core/ScriptTest.java
src/test/java/core/SlaveTest.java
http://jenkins-ci.org/commit/acceptance-test-harness/bdef2c394483ad7d530060ee2227162c3e285b10
Log:
Merge pull request #410 from raul-arabaolaza/JENKINS-49263

JENKINS-49263 Remove flaky and no robust enough tests from SmokeTest Category

No comments against using the SmokeTest so closing this now