Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69650

Fix test reporting of Security2779Test

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • core
    • 2.371

      Security2779Test#noXssInHelp() is flaky, according to the build logs: build log on ci.jenkins.io

      The first test passes, but the following five fail:

      [2022-09-20T02:56:55.267Z] [WARNING] Flakes: 
[2022-09-20T02:56:55.267Z] [WARNING] Security2779Test.noXssInHelp
[2022-09-20T02:56:55.267Z] [INFO]   Run 1: PASS
[2022-09-20T02:56:55.267Z] [ERROR]   Run 2: expected:<0> but was:<1>
[2022-09-20T02:56:55.267Z] [ERROR]   Run 3: expected:<0> but was:<1>
[2022-09-20T02:56:55.267Z] [ERROR]   Run 4: expected:<0> but was:<1>
[2022-09-20T02:56:55.267Z] [ERROR]   Run 5: expected:<0> but was:<1>
[2022-09-20T02:56:55.267Z] [ERROR]   Run 6: expected:<0> but was:<1>

      Improve the test so that it accurately reports the state of the code rather than incorrectly reporting a real failure as a "flaky test". The test is not unreliable. It was correctly identifying a failing condition. The Maven surefire plugin seems to incorrectly classify the test as unreliable.

          [JENKINS-69650] Fix test reporting of Security2779Test

          Mark Waite added a comment - - edited

          It appears that the parameterized test started failing for the '#icon-panel svg' with commit 3923e45a3 in pull request 7052.

          I ran the git bisect with the commands:

          $ git clean -xffd && git reset --hard HEAD
$ gh pr checkout 7134 # Creates branch fix-Security2779Test
$ git checkout master
$ git bisect start a643d8fefa72f1c4610f794920726411fcc73b9e f71495a63f8b861e8ca3a5fcf5cc931fce55bc57
$ git bisect run ../check-the-test

          using the script ../check-the-test:

          #!/bin/bash
git reset --hard HEAD
git checkout fix-Security2779Test -- test/src/test/java/jenkins/security/Security2779Test.java
mvn -am -pl war,bom -Pquick-build clean install && cd test && mvn -Dtest=Security2779Test test
result=$?
git reset --hard HEAD
exit $result

          Mark Waite added a comment - - edited It appears that the parameterized test started failing for the '#icon-panel svg' with commit 3923e45a3 in pull request 7052 . I ran the git bisect with the commands: $ git clean -xffd && git reset --hard HEAD $ gh pr checkout 7134 # Creates branch fix-Security2779Test $ git checkout master $ git bisect start a643d8fefa72f1c4610f794920726411fcc73b9e f71495a63f8b861e8ca3a5fcf5cc931fce55bc57 $ git bisect run ../check-the-test using the script ../check-the-test : #!/bin/bash git reset --hard HEAD git checkout fix-Security2779Test -- test/src/test/java/jenkins/security/Security2779Test.java mvn -am -pl war,bom -Pquick-build clean install && cd test && mvn -Dtest=Security2779Test test result=$? git reset --hard HEAD exit $result

          Daniel Beck added a comment -

          The "flaky" test identified a real regression that was fixed in 2.370 as SECURITY-2886: https://www.jenkins.io/security/advisory/2022-09-21/#SECURITY-2886

          This issue probably should be used to track improvements to the test.

          Daniel Beck added a comment - The "flaky" test identified a real regression that was fixed in 2.370 as SECURITY-2886: https://www.jenkins.io/security/advisory/2022-09-21/#SECURITY-2886 This issue probably should be used to track improvements to the test.

            markewaite Mark Waite
            notmyfault Alexander Brandes
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: