Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64038

ATH splitting tests incorrect if no previous successful builds

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      in a PR to ATHs it will try to split tests based on previous runs if the run is not build#1 of the PR.

      however in the case the first build did not succeed, (and there are not successful builds) then as configured it createts 10 splits but will not create an exclusion file as there where no successful jobs (it was unstable).

      this results in running each test 20 times (as a minumim) once in each split (ten) for each of java8 and java11. (I say at least as we also re-run flaky tests which is in the opinion oif this user not a good practice and allows these flakes to creep in and consume even more resources),

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            Is JENKINS-47206 not working?

            Show
            jglick Jesse Glick added a comment - Is JENKINS-47206 not working?
            Hide
            teilo James Nord added a comment -

            some further investigation.
            it appears as though.

            build #1 does not actually split based on anything (it should be a special case)

            further builds correctly split but it looks like surefire is not correctly being passed the excludes file (it is in a profile to be activated.)

            Show
            teilo James Nord added a comment - some further investigation. it appears as though. build #1 does not actually split based on anything (it should be a special case) further builds correctly split but it looks like surefire is not correctly being passed the excludes file (it is in a profile to be activated.)
            Hide
            teilo James Nord added a comment - - edited

            Jesse Glick I think it is more the if statement on line 11 as I do not see a checkout of the source happening.
            either that or the pipeline is not running that Jenkinsfile but something else (and I have no access to the config to verify)

            Show
            teilo James Nord added a comment - - edited Jesse Glick I think it is more the if statement on line 11 as I do not see a checkout of the source happening. either that or the pipeline is not running that Jenkinsfile but something else (and I have no access to the config to verify)
            Hide
            teilo James Nord added a comment - - edited

            I wonder if the cause of this is because some builds that did not split had only 2 stages and we are splitting into more buckets.

            https://github.com/jenkinsci/parallel-test-executor-plugin/blob/6750151cd38e98d5a231c4da4ba2efcdea6ee87d/src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java#L220

                static List<InclusionExclusionPattern> findTestSplits(Parallelism parallelism, Run<?,?> build, TaskListener listener, boolean generateInclusions, @CheckForNull final String stageName, @CheckForNull FilePath workspace, boolean estimateTestsFromFiles) {
                    TestResult tr = findPreviousTestResult(build, listener);
                    Map<String/*fully qualified class name*/, TestClass> data = new TreeMap<>();
                    if (tr != null) {
                        Run<?,?> prevRun = tr.getRun();
                        if (prevRun instanceof FlowExecutionOwner.Executable && stageName != null) {
                            FlowExecutionOwner owner = ((FlowExecutionOwner.Executable)prevRun).asFlowExecutionOwner();
                            if (owner != null) {
                                FlowExecution execution = owner.getOrNull();
                                if (execution != null) {
                                    DepthFirstScanner scanner = new DepthFirstScanner();
                                    FlowNode stageId = scanner.findFirstMatch(execution, new StageNamePredicate(stageName));
                                    if (stageId != null) {
                                        /*****
                                         * the following line looks dubious
                                         */
                                          tr = ((hudson.tasks.junit.TestResult) tr).getResultForPipelineBlock(stageId.getId());
                                    }
            
                                }
                            }
                        }
                        collect(tr, data);
            
            Show
            teilo James Nord added a comment - - edited I wonder if the cause of this is because some builds that did not split had only 2 stages and we are splitting into more buckets. https://github.com/jenkinsci/parallel-test-executor-plugin/blob/6750151cd38e98d5a231c4da4ba2efcdea6ee87d/src/main/java/org/jenkinsci/plugins/parallel_test_executor/ParallelTestExecutor.java#L220 static List<InclusionExclusionPattern> findTestSplits(Parallelism parallelism, Run<?,?> build, TaskListener listener, boolean generateInclusions, @CheckForNull final String stageName, @CheckForNull FilePath workspace, boolean estimateTestsFromFiles) { TestResult tr = findPreviousTestResult(build, listener); Map< String /*fully qualified class name*/ , TestClass> data = new TreeMap<>(); if (tr != null ) { Run<?,?> prevRun = tr.getRun(); if (prevRun instanceof FlowExecutionOwner.Executable && stageName != null ) { FlowExecutionOwner owner = ((FlowExecutionOwner.Executable)prevRun).asFlowExecutionOwner(); if (owner != null ) { FlowExecution execution = owner.getOrNull(); if (execution != null ) { DepthFirstScanner scanner = new DepthFirstScanner(); FlowNode stageId = scanner.findFirstMatch(execution, new StageNamePredicate(stageName)); if (stageId != null ) { /***** * the following line looks dubious */ tr = ((hudson.tasks.junit.TestResult) tr).getResultForPipelineBlock(stageId.getId()); } } } } collect(tr, data);
            Hide
            teilo James Nord added a comment -

            build #2

            build #3

            Show
            teilo James Nord added a comment - build #2 build #3
            Hide
            teilo James Nord added a comment -

            stagename should be null so this should not matter...

            Show
            teilo James Nord added a comment - stagename should be null so this should not matter...
            Hide
            teilo James Nord added a comment -

            no idea why things sometimes see to run multiple times. can not reproduce in a local unit test. closing this as the original issue is resolved.

            Show
            teilo James Nord added a comment - no idea why things sometimes see to run multiple times. can not reproduce in a local unit test. closing this as the original issue is resolved.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              teilo James Nord
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: