Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64038

ATH splitting tests incorrect if no previous successful builds

XMLWordPrintable

      in a PR to ATHs it will try to split tests based on previous runs if the run is not build#1 of the PR.

      however in the case the first build did not succeed, (and there are not successful builds) then as configured it createts 10 splits but will not create an exclusion file as there where no successful jobs (it was unstable).

      this results in running each test 20 times (as a minumim) once in each split (ten) for each of java8 and java11. (I say at least as we also re-run flaky tests which is in the opinion oif this user not a good practice and allows these flakes to creep in and consume even more resources),

        1. image-2020-10-26-13-32-21-896.png
          image-2020-10-26-13-32-21-896.png
          246 kB
        2. screenshot-1.png
          screenshot-1.png
          47 kB
        3. screenshot-2.png
          screenshot-2.png
          82 kB

          [JENKINS-64038] ATH splitting tests incorrect if no previous successful builds

          Jesse Glick added a comment -

          Is JENKINS-47206 not working?

          Jesse Glick added a comment - Is JENKINS-47206 not working?

          James Nord added a comment -

          some further investigation.
          it appears as though.

          build #1 does not actually split based on anything (it should be a special case)

          further builds correctly split but it looks like surefire is not correctly being passed the excludes file (it is in a profile to be activated.)

          James Nord added a comment - some further investigation. it appears as though. build #1 does not actually split based on anything (it should be a special case) further builds correctly split but it looks like surefire is not correctly being passed the excludes file (it is in a profile to be activated.)

          James Nord added a comment - - edited

          jglick I think it is more the if statement on line 11 as I do not see a checkout of the source happening.
          either that or the pipeline is not running that Jenkinsfile but something else (and I have no access to the config to verify)

          James Nord added a comment - - edited jglick I think it is more the if statement on line 11 as I do not see a checkout of the source happening. either that or the pipeline is not running that Jenkinsfile but something else (and I have no access to the config to verify)

          James Nord added a comment - - edited

          I wonder if the cause of this is because some builds that did not split had only 2 stages and we are splitting into more buckets.

          https://github.com/jenkinsci/parallel-test-executor-plugin/blob/6750151cd38e98d5a231c4da4ba2efcdea6ee87d/src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java#L220

              static List<InclusionExclusionPattern> findTestSplits(Parallelism parallelism, Run<?,?> build, TaskListener listener, boolean generateInclusions, @CheckForNull final String stageName, @CheckForNull FilePath workspace, boolean estimateTestsFromFiles) {
        TestResult tr = findPreviousTestResult(build, listener);
        Map<String/*fully qualified class name*/, TestClass> data = new TreeMap<>();
        if (tr != null) {
            Run<?,?> prevRun = tr.getRun();
            if (prevRun instanceof FlowExecutionOwner.Executable && stageName != null) {
                FlowExecutionOwner owner = ((FlowExecutionOwner.Executable)prevRun).asFlowExecutionOwner();
                if (owner != null) {
                    FlowExecution execution = owner.getOrNull();
                    if (execution != null) {
                        DepthFirstScanner scanner = new DepthFirstScanner();
                        FlowNode stageId = scanner.findFirstMatch(execution, new StageNamePredicate(stageName));
                        if (stageId != null) {
                            /*****
                             * the following line looks dubious
                             */
                              tr = ((hudson.tasks.junit.TestResult) tr).getResultForPipelineBlock(stageId.getId());
                        }

                    }
                }
            }
            collect(tr, data);

          James Nord added a comment - - edited I wonder if the cause of this is because some builds that did not split had only 2 stages and we are splitting into more buckets. https://github.com/jenkinsci/parallel-test-executor-plugin/blob/6750151cd38e98d5a231c4da4ba2efcdea6ee87d/src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java#L220 static List<InclusionExclusionPattern> findTestSplits(Parallelism parallelism, Run<?,?> build, TaskListener listener, boolean generateInclusions, @CheckForNull final String stageName, @CheckForNull FilePath workspace, boolean estimateTestsFromFiles) { TestResult tr = findPreviousTestResult(build, listener); Map< String /*fully qualified class name*/ , TestClass> data = new TreeMap<>(); if (tr != null ) { Run<?,?> prevRun = tr.getRun(); if (prevRun instanceof FlowExecutionOwner.Executable && stageName != null ) { FlowExecutionOwner owner = ((FlowExecutionOwner.Executable)prevRun).asFlowExecutionOwner(); if (owner != null ) { FlowExecution execution = owner.getOrNull(); if (execution != null ) { DepthFirstScanner scanner = new DepthFirstScanner(); FlowNode stageId = scanner.findFirstMatch(execution, new StageNamePredicate(stageName)); if (stageId != null ) { /***** * the following line looks dubious */ tr = ((hudson.tasks.junit.TestResult) tr).getResultForPipelineBlock(stageId.getId()); } } } } collect(tr, data);

          James Nord added a comment -

          build #2

          build #3

          James Nord added a comment - build #2 build #3

          James Nord added a comment -

          stagename should be null so this should not matter...

          James Nord added a comment - stagename should be null so this should not matter...

          James Nord added a comment -

          no idea why things sometimes see to run multiple times. can not reproduce in a local unit test. closing this as the original issue is resolved.

          James Nord added a comment - no idea why things sometimes see to run multiple times. can not reproduce in a local unit test. closing this as the original issue is resolved.

            Unassigned Unassigned
            teilo James Nord
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: