Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49340

Be able to mark a job as succesful even if it has failing tests

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • pipeline-maven-plugin
    • None
    • Jenkins 2.89.3

      I'm using withMaven and running maven in a pipeline. withMaven junit publisher sets the build as UNSTABLE when there's a test.

      However we check for flakey tests and when only flakey tests are found, we'd like to mark the job as sucessful. We tried doing this (see https://github.com/xwiki/xwiki-jenkins-pipeline/blob/master/vars/xwikiBuild.groovy#L230) but the job is still reported as failing in the Jenkins UI and in the logs.

        1. screenshot-1.png
          screenshot-1.png
          33 kB

          [JENKINS-49340] Be able to mark a job as succesful even if it has failing tests

          Vincent Massol added a comment - - edited

          Note that I've now learnt that the issue is that using currentBuild.result to set a new result only works if you define a more restrictive result and that's why it currently doesn't work when setting SUCCESS if the current result is UNSTABLE.

          However, I believe that my use case is valid: if the only failing tests are flaky tests, I'd like to mark the build as successful.

          Note that currently, since the build is never marked as SUCCESS when we receive emails from the mailext plugin, we get a growing lists of "changes since the previous successful build", as in:

          Changes since the previous successful build:

Changes for Build #394
[olexande] Jan 29, 2018 9:28:29 PM XWIKI-14983 issue. obsolescence of the supplied documentation
(rev fbf1c49637a88f6540354fb94d6455246debf155)
[olexande] Jan 29, 2018 9:28:29 PM XWIKI-14983 issue. obsolescence of the supplied documentation
(rev fbf1c49637a88f6540354fb94d6455246debf155)
[olexande] Jan 29, 2018 9:28:29 PM XWIKI-14983 issue. obsolescence of the supplied documentation
(rev fbf1c49637a88f6540354fb94d6455246debf155)

Changes for Build #395
[Guillaume Delhumeau] Jan 30, 2018 12:47:22 PM XWIKI-14984: Even disabled, Watchlist Application still add modified
(rev 6a8095654284f3b48b2c9962ad56f9b6caca0583)
[Guillaume Delhumeau] Jan 30, 2018 12:47:22 PM XWIKI-14984: Even disabled, Watchlist Application still add modified
(rev 6a8095654284f3b48b2c9962ad56f9b6caca0583)

Changes for Build #396
[Thomas Mortagne] Jan 30, 2018 3:52:08 PM [misc] Fix typo
(rev 6ea21bae13c7dd91077faacdb8db38edf4c91a38)
[Thomas Mortagne] Jan 30, 2018 3:52:08 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent
(rev 56f4402c192d5952d49c8ca9146f919fe8babcc4)
[Guillaume Delhumeau] Jan 30, 2018 5:51:45 PM XWIKI-14782: "Failed to save your settings" when enabling the toggles
(rev 26c8f6836d7894f8a440024932d65bf6aa8fb01f)
[Thomas Mortagne] Jan 30, 2018 3:52:08 PM [misc] Fix typo
(rev 6ea21bae13c7dd91077faacdb8db38edf4c91a38)
[Thomas Mortagne] Jan 30, 2018 3:52:08 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent
(rev 56f4402c192d5952d49c8ca9146f919fe8babcc4)
[Guillaume Delhumeau] Jan 30, 2018 5:51:45 PM XWIKI-14782: "Failed to save your settings" when enabling the toggles
(rev 26c8f6836d7894f8a440024932d65bf6aa8fb01f)
[Thomas Mortagne] Jan 30, 2018 3:52:08 PM [misc] Fix typo
(rev 6ea21bae13c7dd91077faacdb8db38edf4c91a38)
[Thomas Mortagne] Jan 30, 2018 3:52:08 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent
(rev 56f4402c192d5952d49c8ca9146f919fe8babcc4)
[Guillaume Delhumeau] Jan 30, 2018 5:51:45 PM XWIKI-14782: "Failed to save your settings" when enabling the toggles
(rev 26c8f6836d7894f8a440024932d65bf6aa8fb01f)

Changes for Build #397
No changes

Changes for Build #398
No changes

Changes for Build #399
No changes

Changes for Build #400
[Thomas Mortagne] Jan 31, 2018 1:33:34 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent
(rev 0aeb3017f29d0e8fdc57b7ae6e086c802f9c84d4)
[Vincent Massol] Jan 31, 2018 6:11:02 PM [Misc] Try to fix apparently flickering test
(rev efc1ac851a358a665dec2685350edd0616e67979)
[Thomas Mortagne] Jan 31, 2018 1:33:34 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent
(rev 0aeb3017f29d0e8fdc57b7ae6e086c802f9c84d4)
[Vincent Massol] Jan 31, 2018 6:11:02 PM [Misc] Try to fix apparently flickering test
(rev efc1ac851a358a665dec2685350edd0616e67979)
[Thomas Mortagne] Jan 31, 2018 1:33:34 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent
(rev 0aeb3017f29d0e8fdc57b7ae6e086c802f9c84d4)
[Vincent Massol] Jan 31, 2018 6:11:02 PM [Misc] Try to fix apparently flickering test
(rev efc1ac851a358a665dec2685350edd0616e67979)

Changes for Build #401
[Vincent Massol] Feb 1, 2018 8:37:17 AM [Misc] Try to fix apparently flickering test (second try, no idea why
(rev 5f7dd138e904c7968d63c3883c8159b5841d83bc)
[Vincent Massol] Feb 1, 2018 8:37:17 AM [Misc] Try to fix apparently flickering test (second try, no idea why
(rev 5f7dd138e904c7968d63c3883c8159b5841d83bc)
[Vincent Massol] Feb 1, 2018 8:37:17 AM [Misc] Try to fix apparently flickering test (second try, no idea why
(rev 5f7dd138e904c7968d63c3883c8159b5841d83bc)

Changes for Build #402
[Thomas Mortagne] Feb 1, 2018 11:31:24 AM XWIKI-14979: Viewing the Admin user profile the first time generates a
(rev d3735a6b8ccf38c467baeedda6ba3d4f5056370b)
[Thomas Mortagne] Feb 1, 2018 11:31:24 AM [misc] Update XAR format
(rev 2f3e031a2c5f33651052735cede80e3bed598f59)
[Vincent Massol] Feb 1, 2018 3:37:18 PM [Misc] Try to fix flickering error "Could not start mail server
(rev f6a699aba8bf7fd3717731847535c94d027d5165)
[Guillaume Delhumeau] Feb 1, 2018 3:44:54 PM XWIKI-14710: Display all filters in the filter LT and not in some
(rev 64ae4cc483af1b4fc34ce54cc33d7c84e963571e)
[Guillaume Delhumeau] Feb 1, 2018 3:46:50 PM XWIKI-14710: Display all filters in the filter LT and not in some
(rev 58d742406cc4b42c1e18b9c5ed4b714f800a62bd)
[Thomas Mortagne] Feb 1, 2018 11:31:24 AM XWIKI-14979: Viewing the Admin user profile the first time generates a
(rev d3735a6b8ccf38c467baeedda6ba3d4f5056370b)
[Thomas Mortagne] Feb 1, 2018 11:31:24 AM [misc] Update XAR format
(rev 2f3e031a2c5f33651052735cede80e3bed598f59)
[Vincent Massol] Feb 1, 2018 3:37:18 PM [Misc] Try to fix flickering error "Could not start mail server
(rev f6a699aba8bf7fd3717731847535c94d027d5165)
[Guillaume Delhumeau] Feb 1, 2018 3:44:54 PM XWIKI-14710: Display all filters in the filter LT and not in some
(rev 64ae4cc483af1b4fc34ce54cc33d7c84e963571e)
[Guillaume Delhumeau] Feb 1, 2018 3:46:50 PM XWIKI-14710: Display all filters in the filter LT and not in some
(rev 58d742406cc4b42c1e18b9c5ed4b714f800a62bd)
[Thomas Mortagne] Feb 1, 2018 11:31:24 AM XWIKI-14979: Viewing the Admin user profile the first time generates a
(rev d3735a6b8ccf38c467baeedda6ba3d4f5056370b)
[Thomas Mortagne] Feb 1, 2018 11:31:24 AM [misc] Update XAR format
(rev 2f3e031a2c5f33651052735cede80e3bed598f59)
[Vincent Massol] Feb 1, 2018 3:37:18 PM [Misc] Try to fix flickering error "Could not start mail server
(rev f6a699aba8bf7fd3717731847535c94d027d5165)
[Guillaume Delhumeau] Feb 1, 2018 3:44:54 PM XWIKI-14710: Display all filters in the filter LT and not in some
(rev 64ae4cc483af1b4fc34ce54cc33d7c84e963571e)
[Guillaume Delhumeau] Feb 1, 2018 3:46:50 PM XWIKI-14710: Display all filters in the filter LT and not in some
(rev 58d742406cc4b42c1e18b9c5ed4b714f800a62bd)

Changes for Build #403
[Guillaume Delhumeau] Feb 1, 2018 4:23:38 PM XWIKI-14345: Wrong user displayed in notifications list when creating a
(rev 8650c38dfc2b40fc64f71df8093c81ba679b1541)
[Vincent Massol] Feb 1, 2018 4:34:52 PM [Misc] Build the distribution flavor tests only once!
(rev 7786fe946e2dca3972f11369c6dcd33a6f1ef7db)
[Eduard Moraru] Feb 1, 2018 8:31:02 PM XWIKI-14975: Finder in documentTree doesn't work when there is a space
(rev 1f82de8f68c7cbe62b431b22e1cc6f3d667e7d66)
[Guillaume Delhumeau] Feb 1, 2018 4:23:38 PM XWIKI-14345: Wrong user displayed in notifications list when creating a
(rev 8650c38dfc2b40fc64f71df8093c81ba679b1541)
[Vincent Massol] Feb 1, 2018 4:34:52 PM [Misc] Build the distribution flavor tests only once!
(rev 7786fe946e2dca3972f11369c6dcd33a6f1ef7db)
[Eduard Moraru] Feb 1, 2018 8:31:02 PM XWIKI-14975: Finder in documentTree doesn't work when there is a space
(rev 1f82de8f68c7cbe62b431b22e1cc6f3d667e7d66)
[Guillaume Delhumeau] Feb 1, 2018 4:23:38 PM XWIKI-14345: Wrong user displayed in notifications list when creating a
(rev 8650c38dfc2b40fc64f71df8093c81ba679b1541)
[Vincent Massol] Feb 1, 2018 4:34:52 PM [Misc] Build the distribution flavor tests only once!
(rev 7786fe946e2dca3972f11369c6dcd33a6f1ef7db)
[Eduard Moraru] Feb 1, 2018 8:31:02 PM XWIKI-14975: Finder in documentTree doesn't work when there is a space
(rev 1f82de8f68c7cbe62b431b22e1cc6f3d667e7d66)

          (this is brought by ${CHANGES_SINCE_LAST_SUCCESS, changesFormat = "[%a] %d %m (rev %r)\n" config in the mailext template).

          Thanks!

          Vincent Massol added a comment - - edited Note that I've now learnt that the issue is that using currentBuild.result  to set a new result only works if you define a more restrictive result and that's why it currently doesn't work when setting SUCCESS if the current result is UNSTABLE. However, I believe that my use case is valid: if the only failing tests are flaky tests, I'd like to mark the build as successful. Note that currently, since the build is never marked as SUCCESS when we receive emails from the mailext plugin, we get a growing lists of "changes since the previous successful build", as in: Changes since the previous successful build: Changes for Build #394 [olexande] Jan 29, 2018 9:28:29 PM XWIKI-14983 issue. obsolescence of the supplied documentation (rev fbf1c49637a88f6540354fb94d6455246debf155) [olexande] Jan 29, 2018 9:28:29 PM XWIKI-14983 issue. obsolescence of the supplied documentation (rev fbf1c49637a88f6540354fb94d6455246debf155) [olexande] Jan 29, 2018 9:28:29 PM XWIKI-14983 issue. obsolescence of the supplied documentation (rev fbf1c49637a88f6540354fb94d6455246debf155) Changes for Build #395 [Guillaume Delhumeau] Jan 30, 2018 12:47:22 PM XWIKI-14984: Even disabled, Watchlist Application still add modified (rev 6a8095654284f3b48b2c9962ad56f9b6caca0583) [Guillaume Delhumeau] Jan 30, 2018 12:47:22 PM XWIKI-14984: Even disabled, Watchlist Application still add modified (rev 6a8095654284f3b48b2c9962ad56f9b6caca0583) Changes for Build #396 [Thomas Mortagne] Jan 30, 2018 3:52:08 PM [misc] Fix typo (rev 6ea21bae13c7dd91077faacdb8db38edf4c91a38) [Thomas Mortagne] Jan 30, 2018 3:52:08 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent (rev 56f4402c192d5952d49c8ca9146f919fe8babcc4) [Guillaume Delhumeau] Jan 30, 2018 5:51:45 PM XWIKI-14782: "Failed to save your settings" when enabling the toggles (rev 26c8f6836d7894f8a440024932d65bf6aa8fb01f) [Thomas Mortagne] Jan 30, 2018 3:52:08 PM [misc] Fix typo (rev 6ea21bae13c7dd91077faacdb8db38edf4c91a38) [Thomas Mortagne] Jan 30, 2018 3:52:08 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent (rev 56f4402c192d5952d49c8ca9146f919fe8babcc4) [Guillaume Delhumeau] Jan 30, 2018 5:51:45 PM XWIKI-14782: "Failed to save your settings" when enabling the toggles (rev 26c8f6836d7894f8a440024932d65bf6aa8fb01f) [Thomas Mortagne] Jan 30, 2018 3:52:08 PM [misc] Fix typo (rev 6ea21bae13c7dd91077faacdb8db38edf4c91a38) [Thomas Mortagne] Jan 30, 2018 3:52:08 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent (rev 56f4402c192d5952d49c8ca9146f919fe8babcc4) [Guillaume Delhumeau] Jan 30, 2018 5:51:45 PM XWIKI-14782: "Failed to save your settings" when enabling the toggles (rev 26c8f6836d7894f8a440024932d65bf6aa8fb01f) Changes for Build #397 No changes Changes for Build #398 No changes Changes for Build #399 No changes Changes for Build #400 [Thomas Mortagne] Jan 31, 2018 1:33:34 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent (rev 0aeb3017f29d0e8fdc57b7ae6e086c802f9c84d4) [Vincent Massol] Jan 31, 2018 6:11:02 PM [Misc] Try to fix apparently flickering test (rev efc1ac851a358a665dec2685350edd0616e67979) [Thomas Mortagne] Jan 31, 2018 1:33:34 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent (rev 0aeb3017f29d0e8fdc57b7ae6e086c802f9c84d4) [Vincent Massol] Jan 31, 2018 6:11:02 PM [Misc] Try to fix apparently flickering test (rev efc1ac851a358a665dec2685350edd0616e67979) [Thomas Mortagne] Jan 31, 2018 1:33:34 PM XWiki-6022 : XWikiDocument toXML lists attachments in an inconsistent (rev 0aeb3017f29d0e8fdc57b7ae6e086c802f9c84d4) [Vincent Massol] Jan 31, 2018 6:11:02 PM [Misc] Try to fix apparently flickering test (rev efc1ac851a358a665dec2685350edd0616e67979) Changes for Build #401 [Vincent Massol] Feb 1, 2018 8:37:17 AM [Misc] Try to fix apparently flickering test (second try, no idea why (rev 5f7dd138e904c7968d63c3883c8159b5841d83bc) [Vincent Massol] Feb 1, 2018 8:37:17 AM [Misc] Try to fix apparently flickering test (second try, no idea why (rev 5f7dd138e904c7968d63c3883c8159b5841d83bc) [Vincent Massol] Feb 1, 2018 8:37:17 AM [Misc] Try to fix apparently flickering test (second try, no idea why (rev 5f7dd138e904c7968d63c3883c8159b5841d83bc) Changes for Build #402 [Thomas Mortagne] Feb 1, 2018 11:31:24 AM XWIKI-14979: Viewing the Admin user profile the first time generates a (rev d3735a6b8ccf38c467baeedda6ba3d4f5056370b) [Thomas Mortagne] Feb 1, 2018 11:31:24 AM [misc] Update XAR format (rev 2f3e031a2c5f33651052735cede80e3bed598f59) [Vincent Massol] Feb 1, 2018 3:37:18 PM [Misc] Try to fix flickering error "Could not start mail server (rev f6a699aba8bf7fd3717731847535c94d027d5165) [Guillaume Delhumeau] Feb 1, 2018 3:44:54 PM XWIKI-14710: Display all filters in the filter LT and not in some (rev 64ae4cc483af1b4fc34ce54cc33d7c84e963571e) [Guillaume Delhumeau] Feb 1, 2018 3:46:50 PM XWIKI-14710: Display all filters in the filter LT and not in some (rev 58d742406cc4b42c1e18b9c5ed4b714f800a62bd) [Thomas Mortagne] Feb 1, 2018 11:31:24 AM XWIKI-14979: Viewing the Admin user profile the first time generates a (rev d3735a6b8ccf38c467baeedda6ba3d4f5056370b) [Thomas Mortagne] Feb 1, 2018 11:31:24 AM [misc] Update XAR format (rev 2f3e031a2c5f33651052735cede80e3bed598f59) [Vincent Massol] Feb 1, 2018 3:37:18 PM [Misc] Try to fix flickering error "Could not start mail server (rev f6a699aba8bf7fd3717731847535c94d027d5165) [Guillaume Delhumeau] Feb 1, 2018 3:44:54 PM XWIKI-14710: Display all filters in the filter LT and not in some (rev 64ae4cc483af1b4fc34ce54cc33d7c84e963571e) [Guillaume Delhumeau] Feb 1, 2018 3:46:50 PM XWIKI-14710: Display all filters in the filter LT and not in some (rev 58d742406cc4b42c1e18b9c5ed4b714f800a62bd) [Thomas Mortagne] Feb 1, 2018 11:31:24 AM XWIKI-14979: Viewing the Admin user profile the first time generates a (rev d3735a6b8ccf38c467baeedda6ba3d4f5056370b) [Thomas Mortagne] Feb 1, 2018 11:31:24 AM [misc] Update XAR format (rev 2f3e031a2c5f33651052735cede80e3bed598f59) [Vincent Massol] Feb 1, 2018 3:37:18 PM [Misc] Try to fix flickering error "Could not start mail server (rev f6a699aba8bf7fd3717731847535c94d027d5165) [Guillaume Delhumeau] Feb 1, 2018 3:44:54 PM XWIKI-14710: Display all filters in the filter LT and not in some (rev 64ae4cc483af1b4fc34ce54cc33d7c84e963571e) [Guillaume Delhumeau] Feb 1, 2018 3:46:50 PM XWIKI-14710: Display all filters in the filter LT and not in some (rev 58d742406cc4b42c1e18b9c5ed4b714f800a62bd) Changes for Build #403 [Guillaume Delhumeau] Feb 1, 2018 4:23:38 PM XWIKI-14345: Wrong user displayed in notifications list when creating a (rev 8650c38dfc2b40fc64f71df8093c81ba679b1541) [Vincent Massol] Feb 1, 2018 4:34:52 PM [Misc] Build the distribution flavor tests only once! (rev 7786fe946e2dca3972f11369c6dcd33a6f1ef7db) [Eduard Moraru] Feb 1, 2018 8:31:02 PM XWIKI-14975: Finder in documentTree doesn't work when there is a space (rev 1f82de8f68c7cbe62b431b22e1cc6f3d667e7d66) [Guillaume Delhumeau] Feb 1, 2018 4:23:38 PM XWIKI-14345: Wrong user displayed in notifications list when creating a (rev 8650c38dfc2b40fc64f71df8093c81ba679b1541) [Vincent Massol] Feb 1, 2018 4:34:52 PM [Misc] Build the distribution flavor tests only once! (rev 7786fe946e2dca3972f11369c6dcd33a6f1ef7db) [Eduard Moraru] Feb 1, 2018 8:31:02 PM XWIKI-14975: Finder in documentTree doesn't work when there is a space (rev 1f82de8f68c7cbe62b431b22e1cc6f3d667e7d66) [Guillaume Delhumeau] Feb 1, 2018 4:23:38 PM XWIKI-14345: Wrong user displayed in notifications list when creating a (rev 8650c38dfc2b40fc64f71df8093c81ba679b1541) [Vincent Massol] Feb 1, 2018 4:34:52 PM [Misc] Build the distribution flavor tests only once! (rev 7786fe946e2dca3972f11369c6dcd33a6f1ef7db) [Eduard Moraru] Feb 1, 2018 8:31:02 PM XWIKI-14975: Finder in documentTree doesn't work when there is a space (rev 1f82de8f68c7cbe62b431b22e1cc6f3d667e7d66) (this is brought by ${CHANGES_SINCE_LAST_SUCCESS, changesFormat = " [%a] %d %m (rev %r)\n" config in the mailext template). Thanks!

            Unassigned Unassigned
            vmassol Vincent Massol
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: