Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6268

Make checking for modification time of junit results be configurable and off by default

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open (View Workflow)
    • Priority: Blocker
    • Resolution: Unresolved
    • Component/s: junit-plugin
    • Labels:
      None
    • Similar Issues:

      Description

      The JUnit Result Archiver will fail the build if the test result files that match the filemask do not have a very recent modification time. The error message it prints is this:
      "Test reports were found but none of them are new. Did tests run?"

      This can be inconvenient in several settings; notably, when the clocks on the master and slave fall out of sync, or when the test results were generated by some process that isn't part of the build itself – maybe they were scp'd from somewhere else, for instance. Or for testing the JUnitResultArchiver itself.

      In my opinion, the behavior should be changed such that check-for-recent-modifications should be off by default. Additionally, the user should be able to specify a "recent" threshold via a setting.

        Attachments

          Issue Links

            Activity

            Hide
            mhulboj mhulboj added a comment -

            I have to use Hudson on a virtual machine on which clock behaves in the following way (less than a second between successive date invocations):

            [pcwindci] /etc/sysconfig > date
            Wed Nov 17 09:37:23 CET 2010
            [pcwindci] /etc/sysconfig > date
            Wed Nov 17 09:34:46 CET 2010
            [pcwindci] /etc/sysconfig > date
            Wed Nov 17 09:37:23 CET 2010

            This makes using Hudson impossible in my environment. I know my setup and would like to completely disable the check for modifications.

            Show
            mhulboj mhulboj added a comment - I have to use Hudson on a virtual machine on which clock behaves in the following way (less than a second between successive date invocations): [pcwindci] /etc/sysconfig > date Wed Nov 17 09:37:23 CET 2010 [pcwindci] /etc/sysconfig > date Wed Nov 17 09:34:46 CET 2010 [pcwindci] /etc/sysconfig > date Wed Nov 17 09:37:23 CET 2010 This makes using Hudson impossible in my environment. I know my setup and would like to completely disable the check for modifications.
            Hide
            netappbluedevil NetAppBlueDevil added a comment -

            I just ran into this because we are using network storage and the storage system time skewed earlier than the server time and the JUnit output files ended up dated earlier than the build. This just happened after a year of running with no issue, so it took a while to track down.

            I'm not sure I really understand the use case for skipping files based on time anyway. In my case, this is just a single master build which starts with a clean, so all the files generated by the build are good to process. The system clock issues if they occur are not really relevant for me.

            Looking at TestResult.java, this issue is almost resolved in the logic. There is a flag for ignoring the timestamp issues, it needs to be elevated to a project configurable setting. Currently it is defined in that class as

            private static final boolean checkTimestamps = true; // TODO: change to System.getProperty

            Show
            netappbluedevil NetAppBlueDevil added a comment - I just ran into this because we are using network storage and the storage system time skewed earlier than the server time and the JUnit output files ended up dated earlier than the build. This just happened after a year of running with no issue, so it took a while to track down. I'm not sure I really understand the use case for skipping files based on time anyway. In my case, this is just a single master build which starts with a clean, so all the files generated by the build are good to process. The system clock issues if they occur are not really relevant for me. Looking at TestResult.java, this issue is almost resolved in the logic. There is a flag for ignoring the timestamp issues, it needs to be elevated to a project configurable setting. Currently it is defined in that class as private static final boolean checkTimestamps = true; // TODO: change to System.getProperty
            Hide
            ebann ebann added a comment -

            I have many Test jobs and one job that build up the whole result.

            As some jobs may take a long time to end, the test-ender job refuses to publish junit results.

            18:12:32 Test reports were found but none of them are new. Did tests run?
            18:12:32 For example, /tmp/jenkins_slave/workspace/TESTS_ENDER/TEST-Statistic.AllTests.xml is 16 hr old
            18:12:32
            18:12:32 Build step 'Publier le rapport des résultats des tests JUnit' changed build result to FAILURE

            Show
            ebann ebann added a comment - I have many Test jobs and one job that build up the whole result. As some jobs may take a long time to end, the test-ender job refuses to publish junit results. 18:12:32 Test reports were found but none of them are new. Did tests run? 18:12:32 For example, /tmp/jenkins_slave/workspace/TESTS_ENDER/TEST-Statistic.AllTests.xml is 16 hr old 18:12:32 18:12:32 Build step 'Publier le rapport des résultats des tests JUnit' changed build result to FAILURE
            Hide
            nirro Stanislav Tyurikov added a comment -

            I have multi-module maven project, which rebuilds only changed modules. I want to display test results for old modules too. But this issue is a blocker for this.

            Show
            nirro Stanislav Tyurikov added a comment - I have multi-module maven project, which rebuilds only changed modules. I want to display test results for old modules too. But this issue is a blocker for this.
            Hide
            markewaite Mark Waite added a comment -

            Unreliable clocks on build machines and on file systems which support build machines seem like they will create many more problems than these few that you've detected.

            If you have reliable clocks and are still seeing the issue, then you could consider the work around of adding a build step which updates the time stamp of the result files. The Linux command "touch" will update the time stamp of one or more files and similar commands are available for Windows and other platforms.

            Show
            markewaite Mark Waite added a comment - Unreliable clocks on build machines and on file systems which support build machines seem like they will create many more problems than these few that you've detected. If you have reliable clocks and are still seeing the issue, then you could consider the work around of adding a build step which updates the time stamp of the result files. The Linux command "touch" will update the time stamp of one or more files and similar commands are available for Windows and other platforms.
            Hide
            radimk Radim Kubacki added a comment -

            More valid use-case then shifted clocks across build machines is a Gradle build. If I commit a change that has no impact on tests than Gradle skips their execution. But Jenkins tries to be smarter and marks the build as failed. See a longer version with workarounds at http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/

            Sounds more like a bug than RFE to me.

            Show
            radimk Radim Kubacki added a comment - More valid use-case then shifted clocks across build machines is a Gradle build. If I commit a change that has no impact on tests than Gradle skips their execution. But Jenkins tries to be smarter and marks the build as failed. See a longer version with workarounds at http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/ Sounds more like a bug than RFE to me.
            Hide
            mheil Matthias Heil added a comment -

            Seems to be that this kind of feature is not wanted in the core, and that users should switch to the xunit plugin to publish junit results.

            Show
            mheil Matthias Heil added a comment - Seems to be that this kind of feature is not wanted in the core, and that users should switch to the xunit plugin to publish junit results.
            Hide
            n0mer n0mer added a comment -
            Show
            n0mer n0mer added a comment - Possible workaround for Gradle: http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/
            Hide
            revoltn Kent Bolton added a comment -

            Below is code related to this issue.

            Looks like the magic number Jenkins core has hardcoded insite it is 1000 miliseconds. Code below in a patch here:

            http://git.eclipse.org/c/hudson/org.eclipse.hudson.core.git/commit?id=97587dde35e862fabbd8b5b4b0136a786cfcff37 .
            This is an issue when using VMs where time can drift quite quickly. Would be good to disable this test, or to be able to change the magic number of seconds in configuration.

            @@ -151,21 +144,22 @@ public final class TestResult extends MetaTabulatedResult {
            }
            }
            if(!parsed) {
            + if (!parsed) {
            long localTime = System.currentTimeMillis();
            if(localTime < buildTime-1000) /margin/
            // build time is in the the future. clock on this slave must be running behind
            + if (localTime < buildTime - 1000) /margin/ // build time is in the the future. clock on this slave must be running behind
            +

            { throw new AbortException( - "Clock on this slave is out of sync with the master, and therefore \n" + - "I can't figure out what test results are new and what are old.\n" + - "Please keep the slave clock in sync with the master."); + "Clock on this slave is out of sync with the master, and therefore \n" + + "I can't figure out what test results are new and what are old.\n" + + "Please keep the slave clock in sync with the master."); + }
            Show
            revoltn Kent Bolton added a comment - Below is code related to this issue. Looks like the magic number Jenkins core has hardcoded insite it is 1000 miliseconds. Code below in a patch here: http://git.eclipse.org/c/hudson/org.eclipse.hudson.core.git/commit?id=97587dde35e862fabbd8b5b4b0136a786cfcff37 . This is an issue when using VMs where time can drift quite quickly. Would be good to disable this test, or to be able to change the magic number of seconds in configuration. @@ -151,21 +144,22 @@ public final class TestResult extends MetaTabulatedResult { } } if(!parsed) { + if (!parsed) { long localTime = System.currentTimeMillis(); if(localTime < buildTime-1000) /margin/ // build time is in the the future. clock on this slave must be running behind + if (localTime < buildTime - 1000) /margin/ // build time is in the the future. clock on this slave must be running behind + { throw new AbortException( - "Clock on this slave is out of sync with the master, and therefore \n" + - "I can't figure out what test results are new and what are old.\n" + - "Please keep the slave clock in sync with the master."); + "Clock on this slave is out of sync with the master, and therefore \n" + + "I can't figure out what test results are new and what are old.\n" + + "Please keep the slave clock in sync with the master."); + }
            Hide
            cmcginty Casey McGinty added a comment - - edited

            Please turn this check into a configuration option. This breaks any multi-project Gradle build running tests.

            Show
            cmcginty Casey McGinty added a comment - - edited Please turn this check into a configuration option. This breaks any multi-project Gradle build running tests.
            Hide
            kaltokri Rolf Geuenich added a comment -

            Hi,

            I want to run "JUnit Result Archiver" belated on a nightly build to test different options and need the option to configure the amount of time for difference, which is ok. Or an option to disable this check at all.
            This case is 5 years old and no one will fix it?
            I hope someone will fix it.

            Show
            kaltokri Rolf Geuenich added a comment - Hi, I want to run "JUnit Result Archiver" belated on a nightly build to test different options and need the option to configure the amount of time for difference, which is ok. Or an option to disable this check at all. This case is 5 years old and no one will fix it? I hope someone will fix it.
            Hide
            slawek Slawek Wieczorkowski added a comment - - edited

            Hi,

            If you work under linux systems, you could try with command:
            touch <your_junit_file.xml>
            e.g touch mytest-junit.xml

            then system will put just current date as modification date and JUnit publish plugin should accept.
            I already tested it and work. I think similar modification date/time command can be run under Windows.

            Show
            slawek Slawek Wieczorkowski added a comment - - edited Hi, If you work under linux systems, you could try with command: touch <your_junit_file.xml> e.g touch mytest-junit.xml then system will put just current date as modification date and JUnit publish plugin should accept. I already tested it and work. I think similar modification date/time command can be run under Windows.
            Hide
            mcerdeiro Mariano Cerdeiro added a comment - - edited

            Hi,

            @Slawek Wieczorkowski Chaining the timestamp with touch is a workaround not a solution.

            In my opinion the best solution is to make this check configurable with 2 parameters, one configuration parameter to indicate if enable or disable and the second one if enable to indicate the maximal allowed timestamp difference.

            Mariano.-

            Show
            mcerdeiro Mariano Cerdeiro added a comment - - edited Hi, @ Slawek Wieczorkowski Chaining the timestamp with touch is a workaround not a solution. In my opinion the best solution is to make this check configurable with 2 parameters, one configuration parameter to indicate if enable or disable and the second one if enable to indicate the maximal allowed timestamp difference. Mariano.-
            Hide
            ablenesi Attila Blenesi added a comment -

            I agree with Mariano Cerdeiro on the solution.

            I'm facing this issue when the code was not changed only configuration files that are not affecting the test.

            Show
            ablenesi Attila Blenesi added a comment - I agree with Mariano Cerdeiro on the solution. I'm facing this issue when the code was not changed only configuration files that are not affecting the test.
            Hide
            kajsa_a Kajsa Anderson added a comment -

            +1 for Mariano Cerdeiro's suggested solution. We've got builds failing because of 5 second differences.

            Show
            kajsa_a Kajsa Anderson added a comment - +1 for Mariano Cerdeiro's suggested solution. We've got builds failing because of 5 second differences.
            Hide
            alex_dubrouski Alex Dubrouski added a comment -

            Gradle 4.0 moved build cache from incubating state to production. Build cache among other things caches JUnit test results, which results in continuous failures. It actually blocks our implementation of pipelines.

            Show
            alex_dubrouski Alex Dubrouski added a comment - Gradle 4.0 moved build cache from incubating state to production. Build cache among other things caches JUnit test results, which results in continuous failures. It actually blocks our implementation of pipelines.
            Hide
            sourmonk Justin McMillan added a comment - - edited

            I'm also blocked by this – I use a single Jenkins pipeline job to execute tests (across multiple environments – we want reporting on a per-environment level), and then another job executes which pulls those JUnit results into the workspace, and builds a report. I am encountering this on XMLs that are ~5 seconds "old".

             

            This Powershell script works, but still not the greatest:

            $files = Get-ChildItem "${WORKSPACE}"
              foreach ($i in $files) {
              $i.LastWriteTime = $(Get-Date)
            }
            
            Show
            sourmonk Justin McMillan added a comment - - edited I'm also blocked by this – I use a single Jenkins pipeline job to execute tests (across multiple environments – we want reporting on a per-environment level), and then another job executes which pulls those JUnit results into the workspace, and builds a report. I am encountering this on XMLs that are ~5 seconds "old".   This Powershell script works, but still not the greatest: $files = Get-ChildItem "${WORKSPACE}" foreach ($i in $files) { $i.LastWriteTime = $(Get-Date) }
            Hide
            mkobit Mike Kobit added a comment -

            Echoing what Alex Dubrouski said, this totally breaks incremental builds from tools like Gradle.

            Show
            mkobit Mike Kobit added a comment - Echoing what Alex Dubrouski said, this totally breaks incremental builds from tools like Gradle.
            Hide
            laeubi Christoph Läubrich added a comment -

            Is there any progress on this? We are also hit by this, so currently we have only the option to disable junit recording at all...

            Show
            laeubi Christoph Läubrich added a comment - Is there any progress on this? We are also hit by this, so currently we have only the option to disable junit recording at all...
            Hide
            postman Per Östman added a comment -

            This one is nine (9) years old. Come on, please fix this!

            Show
            postman Per Östman added a comment - This one is nine (9) years old. Come on, please fix this!
            Hide
            zaktaccardi Zak Taccardi added a comment -

            Note: I filed this too on GitHub which is related https://github.com/jenkinsci/junit-plugin/issues/201

            Show
            zaktaccardi Zak Taccardi added a comment - Note: I filed this too on GitHub which is related  https://github.com/jenkinsci/junit-plugin/issues/201

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              bshine bshine
              Votes:
              51 Vote for this issue
              Watchers:
              47 Start watching this issue

                Dates

                Created:
                Updated: