Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43083

Improve unit and integration tests and avoid relying on E2E (ATH) for front end code

    • Blue Ocean - 1.1-beta-1, Blue Ocean - 1.1-beta2, Blue Ocean 1.1-beta4, Blue Ocean 1.1, Blue Ocean 1.1

      Currently we have some unit tests for UI, a lot of unit and integration tests for API, but rely mostly on ATH for feature and integration testing. 

      This is perhaps not optimal, there must be ways where we can unit and even integration tests components without end to end testing with full browsers. Eg, could we aim for a pyramid of tests like: 

      https://2.bp.blogspot.com/-YTzv_O4TnkA/VTgexlumP1I/AAAAAAAAAJ8/57-rnwyvP6g/s1600/image02.png from: https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html 

      In scope: 

      • Look at a way of reporting test coverage (so we can report on it) and know where we stand
      • Consolidate test runners used (karma or mocha?) across modules
      • Improve code coverage (even if small improvement) for unit testable areas in javascript code where it makes sense, using mocks, component testing, whatever
        • Add unit tests for Activity, Branches and Pagers tab to ensure that data doesn't "blink" when pressing "Show More" (see JENKINS-43702). Note: this is currently a bug on Pull Requests tab.

          [JENKINS-43083] Improve unit and integration tests and avoid relying on E2E (ATH) for front end code

          Michael Neale added a comment -

          cc imeredith this might be worth researching too

          Michael Neale added a comment - cc imeredith this might be worth researching too

          Michael Neale added a comment -

          cliffmeyers IVan did some reasearch on this and was going to write it up. The start of the art for unit and integration testing of react apps is not terribly impressive it seems, would be good to get your feelings on it too. I am sure we can do better. 

          Michael Neale added a comment - cliffmeyers IVan did some reasearch on this and was going to write it up. The start of the art for unit and integration testing of react apps is not terribly impressive it seems, would be good to get your feelings on it too. I am sure we can do better. 

          Cliff Meyers added a comment -

          Couple of useful reads I found over the weekend that seem to be pro-Jest:
          https://www.codementor.io/vijayst/unit-testing-react-components-jest-or-enzyme-du1087lh8
          https://blog.grommet.io/post/2016/09/01/how-we-landed-on-jest-snapshot-testing-for-javascript

          And at least one contrarian opinion:
          https://www.reddit.com/r/reactjs/comments/46cwj3/whats_february_2016s_best_alternative_to_jest_i/

          I am not sold on it, although I do think it might be worth taking a look at. Happy to discuss more.

          Cliff Meyers added a comment - Couple of useful reads I found over the weekend that seem to be pro-Jest: https://www.codementor.io/vijayst/unit-testing-react-components-jest-or-enzyme-du1087lh8 https://blog.grommet.io/post/2016/09/01/how-we-landed-on-jest-snapshot-testing-for-javascript And at least one contrarian opinion: https://www.reddit.com/r/reactjs/comments/46cwj3/whats_february_2016s_best_alternative_to_jest_i/ I am not sold on it, although I do think it might be worth taking a look at. Happy to discuss more.

          Cliff Meyers added a comment -

          Perhaps rather than pure unit tests we could also look at "integration tests" where we mock out HTTP calls using nock. That could give us a lot more confidence that certain views are behaving appropriately when wired up to a more real world plumbing.

          Cliff Meyers added a comment - Perhaps rather than pure unit tests we could also look at "integration tests" where we mock out HTTP calls using nock. That could give us a lot more confidence that certain views are behaving appropriately when wired up to a more real world plumbing.

          James Dumay added a comment -

          This is very much related to JENKINS-43671

          James Dumay added a comment - This is very much related to JENKINS-43671

          Michael Neale added a comment -

          cliffmeyers yes - by "unit tests" really means "not end to end" - it may mean "integration" tests, for sure. 

          Michael Neale added a comment - cliffmeyers yes - by "unit tests" really means "not end to end" - it may mean "integration" tests, for sure. 

          John Hill added a comment - - edited

          Without a deep understanding of React testing (I come from an angularJS 1.x shop) I'd say that it's always a safe bet to chose the framework which maintained by the principal contributors to the project. In this case Facebook -> Jest.

          The Jest team has a healthy community which continually addresses issues and has specifically solved the issue which affected the user from feb 2016 in addition to a bunch of other first-time-user issues.

          John Hill added a comment - - edited Without a deep understanding of React testing (I come from an angularJS 1.x shop) I'd say that it's always a safe bet to chose the framework which maintained by the principal contributors to the project. In this case Facebook -> Jest. The Jest team has a healthy community which continually addresses issues and has specifically solved the issue which affected the user from feb 2016 in addition to a bunch of other first-time-user issues .

          Michael Neale added a comment -

          Jest looks good - but I will let cliffmeyers comment. When I looked it was about snapshot based testing which seems "odd" to me and not sure if it really helps people that much in practice (it just tells you that "things have changed") and encourages people to churn the tests all the time, vs writing things that are less variable with small changes. 

          Michael Neale added a comment - Jest looks good - but I will let cliffmeyers comment. When I looked it was about snapshot based testing which seems "odd" to me and not sure if it really helps people that much in practice (it just tells you that "things have changed") and encourages people to churn the tests all the time, vs writing things that are less variable with small changes. 

          Cliff Meyers added a comment -

          jhill thanks for that feedback. There are a few parts to Jest really, that need to be considered independently:

          • Jest is a test runner, which would replace the mocha runner (used by jenkins-js-builder) or the karma runner (used in some libs like JDL or blueocean-core-js). Jest runs in a browserless environment which has pros and cons. We don't have too many issues with the test runner itself, although it would be good to standardize on one across the board.
          • Just has some auto-mocking behavior which is interesting but also very complex under the hood.
          • Jest offers an API to do snapshotting which is optional; you can also use Enzyme w/ react-test-utils like we do today in most unit tests. The thrust of this ticket was mostly on this point here, as to whether it would better than Enzyme shallow render. Enzyme's html() method should also be explored.

          Cliff Meyers added a comment - jhill thanks for that feedback. There are a few parts to Jest really, that need to be considered independently: Jest is a test runner, which would replace the mocha runner (used by jenkins-js-builder) or the karma runner (used in some libs like JDL or blueocean-core-js). Jest runs in a browserless environment which has pros and cons. We don't have too many issues with the test runner itself, although it would be good to standardize on one across the board. Just has some auto-mocking behavior which is interesting but also very complex under the hood. Jest offers an API to do snapshotting which is optional; you can also use Enzyme w/ react-test-utils like we do today in most unit tests. The thrust of this ticket was mostly on this point here, as to whether it would better than Enzyme shallow render. Enzyme's html() method should also be explored.

          Cliff Meyers added a comment - - edited

          Added link to the PR. As it stands now it accomplishes the following:

          • Switch to Jest test runner for core-js standalone gulp build and dashboard / personalization via js-builder plugin.
          • Outputs coverage reports for all
          • Outputs junit test reports for all
          • Allows unit tests to be easily debugged

          Does not address improvement of test coverage in any particular place, but this should give us the tools to do that.

          Cliff Meyers added a comment - - edited Added link to the PR. As it stands now it accomplishes the following: Switch to Jest test runner for core-js standalone gulp build and dashboard / personalization via js-builder plugin. Outputs coverage reports for all Outputs junit test reports for all Allows unit tests to be easily debugged Does not address improvement of test coverage in any particular place, but this should give us the tools to do that.

            cliffmeyers Cliff Meyers
            michaelneale Michael Neale
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: