• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Component/s: core
• Labels:

  • tests

[JENKINS-69650] Fix test reporting of Security2779Test

Alexander Brandes created issue - 2022-09-20 13:42

Mark Waite made changes - 2022-09-20 22:22

Remote Link

New: This issue links to "PR 7134 for easier diagnosis of the failure (Web Link)" [ 28233 ]

Collapse comment: Mark Waite added a comment - 2022-09-21 04:18, Edited by Mark Waite - 2022-09-21 16:06

Mark Waite added a comment - 2022-09-21 04:18 - edited

It appears that the parameterized test started failing for the '#icon-panel svg' with commit 3923e45a3 in pull request 7052.

I ran the git bisect with the commands:

$ git clean -xffd && git reset --hard HEAD
$ gh pr checkout 7134 # Creates branch fix-Security2779Test
$ git checkout master
$ git bisect start a643d8fefa72f1c4610f794920726411fcc73b9e f71495a63f8b861e8ca3a5fcf5cc931fce55bc57
$ git bisect run ../check-the-test

using the script ../check-the-test:

#!/bin/bash
git reset --hard HEAD
git checkout fix-Security2779Test -- test/src/test/java/jenkins/security/Security2779Test.java
mvn -am -pl war,bom -Pquick-build clean install && cd test && mvn -Dtest=Security2779Test test
result=$?
git reset --hard HEAD
exit $result

Expand comment: Mark Waite added a comment - 2022-09-21 04:18, Edited by Mark Waite - 2022-09-21 16:06

Mark Waite added a comment - 2022-09-21 04:18 - edited It appears that the parameterized test started failing for the '#icon-panel svg' with commit 3923e45a3 in pull request 7052 . I ran the git bisect with the commands: $ git clean -xffd && git reset --hard HEAD $ gh pr checkout 7134 # Creates branch fix-Security2779Test $ git checkout master $ git bisect start a643d8fefa72f1c4610f794920726411fcc73b9e f71495a63f8b861e8ca3a5fcf5cc931fce55bc57 $ git bisect run ../check-the-test using the script ../check-the-test : #!/bin/bash git reset --hard HEAD git checkout fix-Security2779Test -- test/src/test/java/jenkins/security/Security2779Test.java mvn -am -pl war,bom -Pquick-build clean install && cd test && mvn -Dtest=Security2779Test test result=$? git reset --hard HEAD exit $result

Jesse Glick made changes - 2022-09-21 15:49

Assignee

New: Mark Waite [ markewaite ]

Jesse Glick made changes - 2022-09-21 15:49

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

Jesse Glick made changes - 2022-09-21 15:49

Status

Original: In Progress [ 3 ]

New: In Review [ 10005 ]

Collapse comment: Daniel Beck added a comment - 2022-09-21 17:40

Daniel Beck added a comment - 2022-09-21 17:40

The "flaky" test identified a real regression that was fixed in 2.370 as SECURITY-2886: https://www.jenkins.io/security/advisory/2022-09-21/#SECURITY-2886

This issue probably should be used to track improvements to the test.

Expand comment: Daniel Beck added a comment - 2022-09-21 17:40

Daniel Beck added a comment - 2022-09-21 17:40 The "flaky" test identified a real regression that was fixed in 2.370 as SECURITY-2886: https://www.jenkins.io/security/advisory/2022-09-21/#SECURITY-2886 This issue probably should be used to track improvements to the test.

Mark Waite made changes - 2022-09-21 17:56

Description

Original: Security2779Test#noXssInHelp() is flaky, according to the build logs: [build log on ci.jenkins.io|https://ci.jenkins.io/job/Core/job/jenkins/job/master/lastSuccessfulBuild/consoleText] The first test passes, but the following five fail: {code} [2022-09-20T02:56:55.267Z] [WARNING] Flakes: [2022-09-20T02:56:55.267Z] [WARNING] Security2779Test.noXssInHelp [2022-09-20T02:56:55.267Z] [INFO] Run 1: PASS [2022-09-20T02:56:55.267Z] [ERROR] Run 2: expected:<0> but was:<1> [2022-09-20T02:56:55.267Z] [ERROR] Run 3: expected:<0> but was:<1> [2022-09-20T02:56:55.267Z] [ERROR] Run 4: expected:<0> but was:<1> [2022-09-20T02:56:55.267Z] [ERROR] Run 5: expected:<0> but was:<1> [2022-09-20T02:56:55.267Z] [ERROR] Run 6: expected:<0> but was:<1> {code}

New: Security2779Test#noXssInHelp() is flaky, according to the build logs: [build log on ci.jenkins.io|https://ci.jenkins.io/job/Core/job/jenkins/job/master/lastSuccessfulBuild/consoleText] The first test passes, but the following five fail: {code} [2022-09-20T02:56:55.267Z] [WARNING] Flakes: [2022-09-20T02:56:55.267Z] [WARNING] Security2779Test.noXssInHelp [2022-09-20T02:56:55.267Z] [INFO] Run 1: PASS [2022-09-20T02:56:55.267Z] [ERROR] Run 2: expected:<0> but was:<1> [2022-09-20T02:56:55.267Z] [ERROR] Run 3: expected:<0> but was:<1> [2022-09-20T02:56:55.267Z] [ERROR] Run 4: expected:<0> but was:<1> [2022-09-20T02:56:55.267Z] [ERROR] Run 5: expected:<0> but was:<1> [2022-09-20T02:56:55.267Z] [ERROR] Run 6: expected:<0> but was:<1> {code} Improve the test so that it accurately reports the state of the code rather than incorrectly reporting a real failure as a "flaky test". The test is not unreliable. It was correctly identifying a failing condition. The Maven surefire plugin seems to incorrectly classify the test as unreliable.

Mark Waite made changes - 2022-09-21 17:57

Summary

Original: Flaky test in Security2779Test#noXssInHelp()

New: Fix test reporting of Security2779Test

Alexander Brandes made changes - 2022-09-27 12:23

Released As		New: 2.371
Resolution		New: Fixed [ 1 ]
Status	Original: In Review [ 10005 ]	New: Fixed but Unreleased [ 10203 ]

Load more newer events