James Dumay added a comment - 2017-04-19 00:29 - edited vivek I would like you to go through all our java modules and identify where we could improve test coverage. We should have tickets in your queue for improving that coverage so that we do not rely on the ATH to pick up problems. For example, creation backend has almost no tests and I would expect the coverage for that to be rather high. I expect that we want to set a minimum coverage (90%?) for each of our modules and have the build fail if we go below that amount. imeredith is working from this document to improve the ATH situation for logged in scenarios such as creation, editing and personalization.

Vivek Pandey added a comment - 2017-04-28 02:29 jamesdumay Following are the current coverage, looks like lot more work needed to reach 90%. I have opened PR to generate test report, currently I am not failing build if criteria is not met. After all modules meet the criteria we can enable failing on build if coverage is not met. See PR  https://github.com/jenkinsci/blueocean-plugin/pull/1013 . Current code coverages for each module and corresponding ticket. blueocean-commons 25%  JENKINS-43924 blueocean-config 0%  JENKINS-43917 blueocean-events 0%  JENKINS-43740 blueocean-git-pipeline 58%  JENKINS-43923 blueocean-github-pipeline 0%  JENKINS-43741 blueocean-i18n 37%  JENKINS-43921 blueocean-jwt 47%  JENKINS-43919 blueocean-pipeline-api-impl 58%  JENKINS-43916 blueocean-rest-impl 65%  JENKINS-43914 blueocean-web 11%  JENKINS-43920  

James Dumay added a comment - 2017-04-28 02:33 Thanks vivek

Isa Vilacides added a comment - 2017-05-11 10:38 Completing the coverage analysis with the ATH analysis: ATH Area Subareas Automated tests Notes Issues found during ET? Creating new pipelines With github       With git https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/creation/git/localRepo.js - SKIPPED Creates a pipeline with git, checks that creation finishes successfully and finally checks the activity tab "When trying to create a pipeline from git, credentials are mandatory and there is no way to do it as anonymous" Tests SKIPPED Editing/workflow with declarative pipelines Visual editor (convert) See the table below for coverage on this plugin       Edition       Interacting with running builds, capturing input, starting/stopping builds Queued state https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/queued.js - SKIPPED basic tests around the queued state with: Executors disabled Multibranch jobs Freestyle jobs Graph on multibranch job Logconsole on a freestyle job All the tests SKIPPED Failing stages and rerunning https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/failingStages.js Tests around a failing pipeline with stages, and re-running in place Check whether the resultItemas are collapsing as expected   Failing pipeline https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/failing.js basic tests around the failing pipeline. Test whether the result is not collapsed. Create Pipeline Job "noStagesFail", create a simple pipeline, that will produce a failure Check whether the resultItemas are collapsing as expected Collapse, expand, all code bocks visible, replay for failed ones As it has failed, we can rerun the job, check that it runs, and then result is still failure   not Multibranch freestyleRunning - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/notMultibranch/freestyleRunning.js pipelineRunning - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/notMultibranch/pipelineRunning.js Check can run non multibranch pipelines and freestyle jobs from activity screen   multibranch commit messages - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/multibranch/commitMessages.js Creates 2 commits and checks that the latest commit message is shown     filtering - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/multibranch/filtering.js Check that can filter the activity list     Folders - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/multibranch/folder.js Folders tests in BO, navigation from BO to classic and from classic to BO and multibranch     Run and open for multibranch - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/multibranch/multibranchOpening.js Check we run and open results screen for multibranch projects,     Test results - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/multibranch/testResults.js Validate that tests results can be expanded and are correct for a multi branch job   Using blue ocean to navigate a set of pipelines/jobs (perhaps pre existing)                 Artefact https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/artifacts/artifacts.js Create, build and assert that the artefact list shows up and it only shows the first 100 artefacts   authorCollapsed https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/edgeCases/authorCollapsed.js - SKIPPED Checks the condensed state of commit author in detail header All the tests skipped folders https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/edgeCases/folder.js - Some know as failing in CI Load multibranch projects in a folder Verify encoding and spacing of details Unable to load steps for multibranch pipelines with / in them Open Blue Ocean button should not try to load /activity for a folder make sure the open blue ocean button works. test open blueocean from classic - a normal folder page in classic jenkins Jobs can have the same name in different folders, they should show up in the gui Validate correct encoding, pipeline graph and steps deepLinks https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/edgeCases/runDetailsDeepLink.js test whether navigating to Run Details without specifying a tab allows the close button to work correctly. NOTE: it requires to Reload the job page and check that there was a build done   initialStage https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/edgeCases/initialStage.js test whether the stageGraph is rendered in the first run when the first step is not a stage   freestyle https://github.com/jenkinsci/blueocean-acceptance-test/tree/master/src/test/js/freestyle Branch and PR tabs show the correct display   i18n https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/i18n/loadGerman.js basic smoke test for i18n   smoke https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/smoke.js basic tests around the pipeline. This includes creating and running a pipeline job, validating   matrix https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/matrix.js Navigation from matric jobs to classic   404 pages https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/404.js Trying out different urls that should result in the same 404 page   Logs Freestyle - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/log-karaoke/freestyle.js logs tailing with a freestyle job - karaoke mode     Freestyle ping - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/log-karaoke/freestylePing.js We will use a simple script which pings 10 times a certain url. To provoke that the browser has to scroll we will resize the browser and then test the scrollHeight of the parent container. If that is bigger then 0, that means that we have scrolled and the karaoke works as expected.     Input - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/log-karaoke/input.js validate that the pause for user input is working. Check Job Blue Ocean Pipeline Activity Page has run - then go to the detail page and validate the input form     pipeline with no stages but with steps - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/log-karaoke/noStages.js logs tailing a pipeline job without stages, but with steps - karaoke mode     parallel stages - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/log-karaoke/parallelStages.js parallel karaoke not allowing branch selection or completing correctly     parametrised Pipeline - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/log-karaoke/parametrisedPipeline.js Covering basic parametres in run button     stages block - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/log-karaoke/stages-block.js and stages new - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/log-karaoke/stages-new.js and stages old - https://github.com/jenkinsci/blueocean-acceptance-test/blob/master/src/test/js/log-karaoke/stages-old.js Logs tailing with stages and steps in the different modes: Declarative pipelines Pipelines Old deprecated but supported Based on three different syntax files we will run the tests against each syntax to make sure they work all as expected  

Isa Vilacides added a comment - 2017-05-11 10:39 - edited Adding comprehensive information about the unit/integration tests at plugins level (we already have %, but this is more about functionality) both for blue ocean and the pipeline editor:   Blue ocean plugin Area Subarea Automated tests Notes blueocean-common   https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-commons/src/test/java/io/jenkins/blueocean/commons   blueocean-rest - https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-rest-impl/src/test/java/io/jenkins/blueocean/service/embedded   blueocean-pipeline-api-impl - https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-pipeline-api-impl/src   blueocean-personalization Dashboards and pipelines https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-personalization/src/test/js/components   blueocean-jwt - https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-jwt/src/test/java/io/jenkins/blueocean/auth/jwt/impl Authentication at API level blueocean-i18n - https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-i18n/src/test/java/io/jenkins/blueocean/i18n URLs and locales blueocean-github-pipeline - https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-github-pipeline/src/test/java/io/jenkins/blueocean/blueocean_github_pipeline API, authentication blueocean-git-pipeline   https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-git-pipeline/src/test/java/io/jenkins/blueocean/blueocean_git_pipeline Credentials, access token with good and bad credentials blueocean-events   https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-events 0 tests blueocean-dashboard   https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-dashboard/src/test/java/io/jenkins/blueocean URL manager   https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-dashboard/src/test/js Activity, braches, time, logs, PRs, run details, store, test results blueocean-core-js     https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-core-js/test/js/paths paths   https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-core-js/test/js/services services   https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-core-js/test/js/components Run/Replay buttons   https://github.com/jenkinsci/blueocean-plugin/tree/master/blueocean-core-js/test/js/capability   Pipeline editor plugin Area Subarea Automated tests Notes Metadata   https://github.com/jenkinsci/blueocean-pipeline-editor-plugin/blob/master/src/test/java/io/blueocean/rest/pipeline/editor/PipelineMetadataServiceTest.java Unit test for checking metadata PipelineSyntaxConverter   https://github.com/jenkinsci/blueocean-pipeline-editor-plugin/blob/master/src/test/js/services/PipelineSyntaxConverter-spec.js converts from JSON: agent any converts from JSON: agent docker converts from JSON: single stage converts from JSON: parallel stage converts from JSON: named parameter values properly converts from JSON: unnamed parameter values properly converts from JSON: nested steps converts to JSON: basic restores unknown sections from JSON handles unknown steps  

Vivek Pandey added a comment - 2017-05-11 15:37 vilacides  There is separate ATH ticket for that. In this ticket, backend functionality is tested  by writing backend integration tests and unit test to test each individual functionality in backend code, code coverage is going to be measure for that.