Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65482

Problem with matrix jobs: No removed found with id...

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • analysis-model
    • None
    • analysis-model 10.0.0
      matrix-project-plugin 1.18

      We have a matrix job with a JDK axis and two sub-jobs (JDK8 and JDK11).

      Since upgrading to analysis-model 10.0.0 it seems that "issues" are mixed up between the sub jobs and the parent (matrix-) job.

      E.g. we have four green builds without any code changes: #643, #644, #645, #646
      I think #643 was done with an older version of Jenkins and the analysis-model plugin.

      #643 has 42 spotbugs warnings
      #644 has twice as many (84) spotbugs warnings, 42 new
      #645 has 84 warnings, no new
      #646 has 84 warnings, 42 new

      Looking at the individual child jobs, they both have 42 spotbugs warnings, so it looks like the parent job somehow aggregates the results, but doesn't fully recognize them, so they are marked as new.

      The result of this mixup is that the matrix (parent) job repeatedly fails since build #647 at the end, because it cannot find an issue:

      FATAL: No removed found with id 0758f19a-939f-43ea-813d-9de068897527.
      edu.hm.hafner.util.NoSuchElementException: No removed found with id 0758f19a-939f-43ea-813d-9de068897527.
      	at edu.hm.hafner.analysis.Report.remove(Report.java:366)
      	at edu.hm.hafner.analysis.IssueDifference.remove(IssueDifference.java:92)
      	at edu.hm.hafner.analysis.IssueDifference.matchIssuesByEquals(IssueDifference.java:62)
      	at edu.hm.hafner.analysis.IssueDifference.<init>(IssueDifference.java:48)
      	at io.jenkins.plugins.analysis.core.model.DeltaReport.<init>(DeltaReport.java:49)
      	at io.jenkins.plugins.analysis.core.steps.IssuesPublisher.attachAction(IssuesPublisher.java:100)
      	at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.publishResult(IssuesRecorder.java:807)
      	at io.jenkins.plugins.analysis.core.steps.IssuesAggregator.lambda$endBuild$52c4a486$1(IssuesAggregator.java:91)
      	at org.eclipse.collections.impl.map.mutable.UnifiedMap.forEachKeyValue(UnifiedMap.java:998)
      	at org.eclipse.collections.impl.multimap.AbstractMultimap.forEachKeyMultiValues(AbstractMultimap.java:188)
      	at io.jenkins.plugins.analysis.core.steps.IssuesAggregator.endBuild(IssuesAggregator.java:89)
      	at hudson.matrix.MatrixBuild$MatrixBuildExecution.post2(MatrixBuild.java:433)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:698)
      	at hudson.model.Run.execute(Run.java:1932)
      	at hudson.matrix.MatrixBuild.run(MatrixBuild.java:323)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:429)
      

      Looking at the build output in the filesystem, I find the issue here in #646:

      grep -r 0758f19a-939f-43ea-813d-9de068897527
      builds/646/spotbugs-outstanding-issues.xml:      <id>0758f19a-939f-43ea-813d-9de068897527</id>
      configurations/axis-jdk/JDK11/builds/646/spotbugs-outstanding-issues.xml:      <id>0758f19a-939f-43ea-813d-9de068897527</id>
      

          [JENKINS-65482] Problem with matrix jobs: No removed found with id...

          Carsten Pfeiffer created issue -
          Carsten Pfeiffer made changes -
          Description Original: We have a matrix job with a JDK axis and two sub-jobs (JDK8 and JDK11).

          Since upgrading to analysis-model 10.0.0 it seems that "issues" are mixed up between the sub jobs and the parent (matrix-) job.

          E.g. we have four green builds without any code changes: #643, #644, #645, #646
          I think #643 was done with an older version of Jenkins and the analysis-model plugin.

          #643 has 42 spotbugs warnings
          #644 has twice as many (84) spotbugs warnings, 42 new
          #645 has 84 warnings, no new
          #646 has 84 warnings, 42 new

          Looking at the individual child jobs, they both have 42 spotbugs warnings, so it looks like the parent job somehow aggregates the results, but doesn't fully recognize them, so they are marked as new.

          The result of this mixup is that the matrix jobs repeatedly fails since build #647 at the end, because it cannot find an issue:
          {code}
          FATAL: No removed found with id 0758f19a-939f-43ea-813d-9de068897527.
          edu.hm.hafner.util.NoSuchElementException: No removed found with id 0758f19a-939f-43ea-813d-9de068897527.
          at edu.hm.hafner.analysis.Report.remove(Report.java:366)
          at edu.hm.hafner.analysis.IssueDifference.remove(IssueDifference.java:92)
          at edu.hm.hafner.analysis.IssueDifference.matchIssuesByEquals(IssueDifference.java:62)
          at edu.hm.hafner.analysis.IssueDifference.<init>(IssueDifference.java:48)
          at io.jenkins.plugins.analysis.core.model.DeltaReport.<init>(DeltaReport.java:49)
          at io.jenkins.plugins.analysis.core.steps.IssuesPublisher.attachAction(IssuesPublisher.java:100)
          at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.publishResult(IssuesRecorder.java:807)
          at io.jenkins.plugins.analysis.core.steps.IssuesAggregator.lambda$endBuild$52c4a486$1(IssuesAggregator.java:91)
          at org.eclipse.collections.impl.map.mutable.UnifiedMap.forEachKeyValue(UnifiedMap.java:998)
          at org.eclipse.collections.impl.multimap.AbstractMultimap.forEachKeyMultiValues(AbstractMultimap.java:188)
          at io.jenkins.plugins.analysis.core.steps.IssuesAggregator.endBuild(IssuesAggregator.java:89)
          at hudson.matrix.MatrixBuild$MatrixBuildExecution.post2(MatrixBuild.java:433)
          at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:698)
          at hudson.model.Run.execute(Run.java:1932)
          at hudson.matrix.MatrixBuild.run(MatrixBuild.java:323)
          at hudson.model.ResourceController.execute(ResourceController.java:97)
          at hudson.model.Executor.run(Executor.java:429)
          {code}

          Looking at the build output in the filesystem, I find the issue here in #646:
          {code}
          grep -r 0758f19a-939f-43ea-813d-9de068897527
          builds/646/spotbugs-outstanding-issues.xml: <id>0758f19a-939f-43ea-813d-9de068897527</id>
          configurations/axis-jdk/JDK11/builds/646/spotbugs-outstanding-issues.xml: <id>0758f19a-939f-43ea-813d-9de068897527</id>
          {code}
          New: We have a matrix job with a JDK axis and two sub-jobs (JDK8 and JDK11).

          Since upgrading to analysis-model 10.0.0 it seems that "issues" are mixed up between the sub jobs and the parent (matrix-) job.

          E.g. we have four green builds without any code changes: #643, #644, #645, #646
          I think #643 was done with an older version of Jenkins and the analysis-model plugin.

          #643 has 42 spotbugs warnings
          #644 has twice as many (84) spotbugs warnings, 42 new
          #645 has 84 warnings, no new
          #646 has 84 warnings, 42 new

          Looking at the individual child jobs, they both have 42 spotbugs warnings, so it looks like the parent job somehow aggregates the results, but doesn't fully recognize them, so they are marked as new.

          The result of this mixup is that the matrix (parent) job repeatedly fails since build #647 at the end, because it cannot find an issue:
          {code}
          FATAL: No removed found with id 0758f19a-939f-43ea-813d-9de068897527.
          edu.hm.hafner.util.NoSuchElementException: No removed found with id 0758f19a-939f-43ea-813d-9de068897527.
          at edu.hm.hafner.analysis.Report.remove(Report.java:366)
          at edu.hm.hafner.analysis.IssueDifference.remove(IssueDifference.java:92)
          at edu.hm.hafner.analysis.IssueDifference.matchIssuesByEquals(IssueDifference.java:62)
          at edu.hm.hafner.analysis.IssueDifference.<init>(IssueDifference.java:48)
          at io.jenkins.plugins.analysis.core.model.DeltaReport.<init>(DeltaReport.java:49)
          at io.jenkins.plugins.analysis.core.steps.IssuesPublisher.attachAction(IssuesPublisher.java:100)
          at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.publishResult(IssuesRecorder.java:807)
          at io.jenkins.plugins.analysis.core.steps.IssuesAggregator.lambda$endBuild$52c4a486$1(IssuesAggregator.java:91)
          at org.eclipse.collections.impl.map.mutable.UnifiedMap.forEachKeyValue(UnifiedMap.java:998)
          at org.eclipse.collections.impl.multimap.AbstractMultimap.forEachKeyMultiValues(AbstractMultimap.java:188)
          at io.jenkins.plugins.analysis.core.steps.IssuesAggregator.endBuild(IssuesAggregator.java:89)
          at hudson.matrix.MatrixBuild$MatrixBuildExecution.post2(MatrixBuild.java:433)
          at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:698)
          at hudson.model.Run.execute(Run.java:1932)
          at hudson.matrix.MatrixBuild.run(MatrixBuild.java:323)
          at hudson.model.ResourceController.execute(ResourceController.java:97)
          at hudson.model.Executor.run(Executor.java:429)
          {code}

          Looking at the build output in the filesystem, I find the issue here in #646:
          {code}
          grep -r 0758f19a-939f-43ea-813d-9de068897527
          builds/646/spotbugs-outstanding-issues.xml: <id>0758f19a-939f-43ea-813d-9de068897527</id>
          configurations/axis-jdk/JDK11/builds/646/spotbugs-outstanding-issues.xml: <id>0758f19a-939f-43ea-813d-9de068897527</id>
          {code}

          Ulli Hafner added a comment -

          This seems to be related to JENKINS-65382. Somehow the mixing of different results does not work.

          Ulli Hafner added a comment - This seems to be related to JENKINS-65382 . Somehow the mixing of different results does not work.
          Ulli Hafner made changes -
          Link New: This issue relates to JENKINS-65382 [ JENKINS-65382 ]

          Thanks for the quick reply!
          Looking at builds/645/spotbugs-outstanding-issues.xml and builds/646/spotbugs-new-issues.xml, and checking a "new issue" in #646 I notice that

          • both issues have the same fingerprint
          • but a different id
            I guess that they do not match, because one of them stems from the JDK8 child job and the other from the JDK11 child job.

          So (wild guess), maybe the issues are aggregated from both child jobs and somewhere hashed by the fingerprint, and it is indeterministic from which child job the issue is then taken. When they alternate between builds, the issue is marked as new.

          Carsten Pfeiffer added a comment - Thanks for the quick reply! Looking at builds/645/spotbugs-outstanding-issues.xml and builds/646/spotbugs-new-issues.xml, and checking a "new issue" in #646 I notice that both issues have the same fingerprint but a different id I guess that they do not match, because one of them stems from the JDK8 child job and the other from the JDK11 child job. So (wild guess), maybe the issues are aggregated from both child jobs and somewhere hashed by the fingerprint, and it is indeterministic from which child job the issue is then taken. When they alternate between builds, the issue is marked as new.

          To clarify, the difference comes from the pathName element:

          <pathName>/var/jenkins_home/workspace/foo/validation/bar-validation/jdk/JDK8</pathName>
          

          vs.

          <pathName>/var/jenkins_home/workspace/foo/validation/bar-validation/jdk/JDK11</pathName>
          

          Carsten Pfeiffer added a comment - To clarify, the difference comes from the pathName element: <pathName>/ var /jenkins_home/workspace/foo/validation/bar-validation/jdk/JDK8</pathName> vs. <pathName>/ var /jenkins_home/workspace/foo/validation/bar-validation/jdk/JDK11</pathName>

          Ulli Hafner added a comment -

          Thanks for your help! id and pathName are actually not part of equals, because they change from build to build (see https://github.com/jenkinsci/analysis-model/blob/master/src/main/java/edu/hm/hafner/analysis/Issue.java#L851).
          I think the underlying problem is that in 10.x the results are stored in two sub results while in 9.x everything was part of one collection. I need to prepare a test case that shows the problem...

          Ulli Hafner added a comment - Thanks for your help! id and pathName are actually not part of equals , because they change from build to build (see https://github.com/jenkinsci/analysis-model/blob/master/src/main/java/edu/hm/hafner/analysis/Issue.java#L851 ). I think the underlying problem is that in 10.x the results are stored in two sub results while in 9.x everything was part of one collection. I need to prepare a test case that shows the problem...

          Alright, thanks – let me know if you need any more information.

          Carsten Pfeiffer added a comment - Alright, thanks – let me know if you need any more information.
          Ulli Hafner made changes -
          Rank New: Ranked higher
          Ulli Hafner made changes -
          Rank New: Ranked higher

            drulli Ulli Hafner
            capf Carsten Pfeiffer
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: