Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6268

Make checking for modification time of junit results be configurable and off by default

    XMLWordPrintable

Details

    • Improvement
    • Status: Open (View Workflow)
    • Blocker
    • Resolution: Unresolved
    • junit-plugin
    • None

    Description

      The JUnit Result Archiver will fail the build if the test result files that match the filemask do not have a very recent modification time. The error message it prints is this:
      "Test reports were found but none of them are new. Did tests run?"

      This can be inconvenient in several settings; notably, when the clocks on the master and slave fall out of sync, or when the test results were generated by some process that isn't part of the build itself – maybe they were scp'd from somewhere else, for instance. Or for testing the JUnitResultArchiver itself.

      In my opinion, the behavior should be changed such that check-for-recent-modifications should be off by default. Additionally, the user should be able to specify a "recent" threshold via a setting.

      Attachments

        Issue Links

          Activity

            mhulboj mhulboj added a comment -

            I have to use Hudson on a virtual machine on which clock behaves in the following way (less than a second between successive date invocations):

            [pcwindci] /etc/sysconfig > date
            Wed Nov 17 09:37:23 CET 2010
            [pcwindci] /etc/sysconfig > date
            Wed Nov 17 09:34:46 CET 2010
            [pcwindci] /etc/sysconfig > date
            Wed Nov 17 09:37:23 CET 2010

            This makes using Hudson impossible in my environment. I know my setup and would like to completely disable the check for modifications.

            mhulboj mhulboj added a comment - I have to use Hudson on a virtual machine on which clock behaves in the following way (less than a second between successive date invocations): [pcwindci] /etc/sysconfig > date Wed Nov 17 09:37:23 CET 2010 [pcwindci] /etc/sysconfig > date Wed Nov 17 09:34:46 CET 2010 [pcwindci] /etc/sysconfig > date Wed Nov 17 09:37:23 CET 2010 This makes using Hudson impossible in my environment. I know my setup and would like to completely disable the check for modifications.

            I just ran into this because we are using network storage and the storage system time skewed earlier than the server time and the JUnit output files ended up dated earlier than the build. This just happened after a year of running with no issue, so it took a while to track down.

            I'm not sure I really understand the use case for skipping files based on time anyway. In my case, this is just a single master build which starts with a clean, so all the files generated by the build are good to process. The system clock issues if they occur are not really relevant for me.

            Looking at TestResult.java, this issue is almost resolved in the logic. There is a flag for ignoring the timestamp issues, it needs to be elevated to a project configurable setting. Currently it is defined in that class as

            private static final boolean checkTimestamps = true; // TODO: change to System.getProperty

            netappbluedevil NetAppBlueDevil added a comment - I just ran into this because we are using network storage and the storage system time skewed earlier than the server time and the JUnit output files ended up dated earlier than the build. This just happened after a year of running with no issue, so it took a while to track down. I'm not sure I really understand the use case for skipping files based on time anyway. In my case, this is just a single master build which starts with a clean, so all the files generated by the build are good to process. The system clock issues if they occur are not really relevant for me. Looking at TestResult.java, this issue is almost resolved in the logic. There is a flag for ignoring the timestamp issues, it needs to be elevated to a project configurable setting. Currently it is defined in that class as private static final boolean checkTimestamps = true; // TODO: change to System.getProperty
            ebann ebann added a comment -

            I have many Test jobs and one job that build up the whole result.

            As some jobs may take a long time to end, the test-ender job refuses to publish junit results.

            18:12:32 Test reports were found but none of them are new. Did tests run?
            18:12:32 For example, /tmp/jenkins_slave/workspace/TESTS_ENDER/TEST-Statistic.AllTests.xml is 16 hr old
            18:12:32
            18:12:32 Build step 'Publier le rapport des résultats des tests JUnit' changed build result to FAILURE

            ebann ebann added a comment - I have many Test jobs and one job that build up the whole result. As some jobs may take a long time to end, the test-ender job refuses to publish junit results. 18:12:32 Test reports were found but none of them are new. Did tests run? 18:12:32 For example, /tmp/jenkins_slave/workspace/TESTS_ENDER/TEST-Statistic.AllTests.xml is 16 hr old 18:12:32 18:12:32 Build step 'Publier le rapport des résultats des tests JUnit' changed build result to FAILURE

            I have multi-module maven project, which rebuilds only changed modules. I want to display test results for old modules too. But this issue is a blocker for this.

            nirro Stanislav Tyurikov added a comment - I have multi-module maven project, which rebuilds only changed modules. I want to display test results for old modules too. But this issue is a blocker for this.
            markewaite Mark Waite added a comment -

            Unreliable clocks on build machines and on file systems which support build machines seem like they will create many more problems than these few that you've detected.

            If you have reliable clocks and are still seeing the issue, then you could consider the work around of adding a build step which updates the time stamp of the result files. The Linux command "touch" will update the time stamp of one or more files and similar commands are available for Windows and other platforms.

            markewaite Mark Waite added a comment - Unreliable clocks on build machines and on file systems which support build machines seem like they will create many more problems than these few that you've detected. If you have reliable clocks and are still seeing the issue, then you could consider the work around of adding a build step which updates the time stamp of the result files. The Linux command "touch" will update the time stamp of one or more files and similar commands are available for Windows and other platforms.
            radimk Radim Kubacki added a comment -

            More valid use-case then shifted clocks across build machines is a Gradle build. If I commit a change that has no impact on tests than Gradle skips their execution. But Jenkins tries to be smarter and marks the build as failed. See a longer version with workarounds at http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/

            Sounds more like a bug than RFE to me.

            radimk Radim Kubacki added a comment - More valid use-case then shifted clocks across build machines is a Gradle build. If I commit a change that has no impact on tests than Gradle skips their execution. But Jenkins tries to be smarter and marks the build as failed. See a longer version with workarounds at http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/ Sounds more like a bug than RFE to me.
            mheil Matthias Heil added a comment -

            Seems to be that this kind of feature is not wanted in the core, and that users should switch to the xunit plugin to publish junit results.

            mheil Matthias Heil added a comment - Seems to be that this kind of feature is not wanted in the core, and that users should switch to the xunit plugin to publish junit results.
            n0mer n0mer added a comment - Possible workaround for Gradle: http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/
            revoltn Kent Bolton added a comment -

            Below is code related to this issue.

            Looks like the magic number Jenkins core has hardcoded insite it is 1000 miliseconds. Code below in a patch here:

            http://git.eclipse.org/c/hudson/org.eclipse.hudson.core.git/commit?id=97587dde35e862fabbd8b5b4b0136a786cfcff37 .
            This is an issue when using VMs where time can drift quite quickly. Would be good to disable this test, or to be able to change the magic number of seconds in configuration.

            @@ -151,21 +144,22 @@ public final class TestResult extends MetaTabulatedResult {
            }
            }
            if(!parsed) {
            + if (!parsed) {
            long localTime = System.currentTimeMillis();
            if(localTime < buildTime-1000) /margin/
            // build time is in the the future. clock on this slave must be running behind
            + if (localTime < buildTime - 1000) /margin/ // build time is in the the future. clock on this slave must be running behind
            +

            { throw new AbortException( - "Clock on this slave is out of sync with the master, and therefore \n" + - "I can't figure out what test results are new and what are old.\n" + - "Please keep the slave clock in sync with the master."); + "Clock on this slave is out of sync with the master, and therefore \n" + + "I can't figure out what test results are new and what are old.\n" + + "Please keep the slave clock in sync with the master."); + }
            revoltn Kent Bolton added a comment - Below is code related to this issue. Looks like the magic number Jenkins core has hardcoded insite it is 1000 miliseconds. Code below in a patch here: http://git.eclipse.org/c/hudson/org.eclipse.hudson.core.git/commit?id=97587dde35e862fabbd8b5b4b0136a786cfcff37 . This is an issue when using VMs where time can drift quite quickly. Would be good to disable this test, or to be able to change the magic number of seconds in configuration. @@ -151,21 +144,22 @@ public final class TestResult extends MetaTabulatedResult { } } if(!parsed) { + if (!parsed) { long localTime = System.currentTimeMillis(); if(localTime < buildTime-1000) /margin/ // build time is in the the future. clock on this slave must be running behind + if (localTime < buildTime - 1000) /margin/ // build time is in the the future. clock on this slave must be running behind + { throw new AbortException( - "Clock on this slave is out of sync with the master, and therefore \n" + - "I can't figure out what test results are new and what are old.\n" + - "Please keep the slave clock in sync with the master."); + "Clock on this slave is out of sync with the master, and therefore \n" + + "I can't figure out what test results are new and what are old.\n" + + "Please keep the slave clock in sync with the master."); + }
            cmcginty Casey McGinty added a comment - - edited

            Please turn this check into a configuration option. This breaks any multi-project Gradle build running tests.

            cmcginty Casey McGinty added a comment - - edited Please turn this check into a configuration option. This breaks any multi-project Gradle build running tests.
            kaltokri Rolf Geuenich added a comment -

            Hi,

            I want to run "JUnit Result Archiver" belated on a nightly build to test different options and need the option to configure the amount of time for difference, which is ok. Or an option to disable this check at all.
            This case is 5 years old and no one will fix it?
            I hope someone will fix it.

            kaltokri Rolf Geuenich added a comment - Hi, I want to run "JUnit Result Archiver" belated on a nightly build to test different options and need the option to configure the amount of time for difference, which is ok. Or an option to disable this check at all. This case is 5 years old and no one will fix it? I hope someone will fix it.
            slawek Slawek Wieczorkowski added a comment - - edited

            Hi,

            If you work under linux systems, you could try with command:
            touch <your_junit_file.xml>
            e.g touch mytest-junit.xml

            then system will put just current date as modification date and JUnit publish plugin should accept.
            I already tested it and work. I think similar modification date/time command can be run under Windows.

            slawek Slawek Wieczorkowski added a comment - - edited Hi, If you work under linux systems, you could try with command: touch <your_junit_file.xml> e.g touch mytest-junit.xml then system will put just current date as modification date and JUnit publish plugin should accept. I already tested it and work. I think similar modification date/time command can be run under Windows.
            mcerdeiro Mariano Cerdeiro added a comment - - edited

            Hi,

            @slawek Chaining the timestamp with touch is a workaround not a solution.

            In my opinion the best solution is to make this check configurable with 2 parameters, one configuration parameter to indicate if enable or disable and the second one if enable to indicate the maximal allowed timestamp difference.

            Mariano.-

            mcerdeiro Mariano Cerdeiro added a comment - - edited Hi, @ slawek Chaining the timestamp with touch is a workaround not a solution. In my opinion the best solution is to make this check configurable with 2 parameters, one configuration parameter to indicate if enable or disable and the second one if enable to indicate the maximal allowed timestamp difference. Mariano.-

            I agree with mcerdeiro on the solution.

            I'm facing this issue when the code was not changed only configuration files that are not affecting the test.

            ablenesi Attila Blenesi added a comment - I agree with mcerdeiro on the solution. I'm facing this issue when the code was not changed only configuration files that are not affecting the test.

            +1 for Mariano Cerdeiro's suggested solution. We've got builds failing because of 5 second differences.

            kajsa_a Kajsa Anderson added a comment - +1 for Mariano Cerdeiro's suggested solution. We've got builds failing because of 5 second differences.

            Gradle 4.0 moved build cache from incubating state to production. Build cache among other things caches JUnit test results, which results in continuous failures. It actually blocks our implementation of pipelines.

            alex_dubrouski Alex Dubrouski added a comment - Gradle 4.0 moved build cache from incubating state to production. Build cache among other things caches JUnit test results, which results in continuous failures. It actually blocks our implementation of pipelines.
            sourmonk Justin McMillan added a comment - - edited

            I'm also blocked by this – I use a single Jenkins pipeline job to execute tests (across multiple environments – we want reporting on a per-environment level), and then another job executes which pulls those JUnit results into the workspace, and builds a report. I am encountering this on XMLs that are ~5 seconds "old".

             

            This Powershell script works, but still not the greatest:

            $files = Get-ChildItem "${WORKSPACE}"
              foreach ($i in $files) {
              $i.LastWriteTime = $(Get-Date)
            }
            
            sourmonk Justin McMillan added a comment - - edited I'm also blocked by this – I use a single Jenkins pipeline job to execute tests (across multiple environments – we want reporting on a per-environment level), and then another job executes which pulls those JUnit results into the workspace, and builds a report. I am encountering this on XMLs that are ~5 seconds "old".   This Powershell script works, but still not the greatest: $files = Get-ChildItem "${WORKSPACE}" foreach ($i in $files) { $i.LastWriteTime = $(Get-Date) }
            mkobit Mike Kobit added a comment -

            Echoing what alex_dubrouski said, this totally breaks incremental builds from tools like Gradle.

            mkobit Mike Kobit added a comment - Echoing what alex_dubrouski said, this totally breaks incremental builds from tools like Gradle.

            Is there any progress on this? We are also hit by this, so currently we have only the option to disable junit recording at all...

            laeubi Christoph Läubrich added a comment - Is there any progress on this? We are also hit by this, so currently we have only the option to disable junit recording at all...
            postman Per Östman added a comment -

            This one is nine (9) years old. Come on, please fix this!

            postman Per Östman added a comment - This one is nine (9) years old. Come on, please fix this!
            zaktaccardi Zak Taccardi added a comment -

            Note: I filed this too on GitHub which is related https://github.com/jenkinsci/junit-plugin/issues/201

            zaktaccardi Zak Taccardi added a comment - Note: I filed this too on GitHub which is related  https://github.com/jenkinsci/junit-plugin/issues/201

            People

              Unassigned Unassigned
              bshine bshine
              Votes:
              54 Vote for this issue
              Watchers:
              49 Start watching this issue

              Dates

                Created:
                Updated: