Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-6268

Make checking for modification time of junit results be configurable and off by default

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • junit-plugin
    • None
    • 1119.va_a_5e9068da_d7

      The JUnit Result Archiver will fail the build if the test result files that match the filemask do not have a very recent modification time. The error message it prints is this:
      "Test reports were found but none of them are new. Did tests run?"

      This can be inconvenient in several settings; notably, when the clocks on the master and slave fall out of sync, or when the test results were generated by some process that isn't part of the build itself – maybe they were scp'd from somewhere else, for instance. Or for testing the JUnitResultArchiver itself.

      In my opinion, the behavior should be changed such that check-for-recent-modifications should be off by default. Additionally, the user should be able to specify a "recent" threshold via a setting.

          [JENKINS-6268] Make checking for modification time of junit results be configurable and off by default

          mhulboj added a comment -

          I have to use Hudson on a virtual machine on which clock behaves in the following way (less than a second between successive date invocations):

          [pcwindci] /etc/sysconfig > date
          Wed Nov 17 09:37:23 CET 2010
          [pcwindci] /etc/sysconfig > date
          Wed Nov 17 09:34:46 CET 2010
          [pcwindci] /etc/sysconfig > date
          Wed Nov 17 09:37:23 CET 2010

          This makes using Hudson impossible in my environment. I know my setup and would like to completely disable the check for modifications.

          mhulboj added a comment - I have to use Hudson on a virtual machine on which clock behaves in the following way (less than a second between successive date invocations): [pcwindci] /etc/sysconfig > date Wed Nov 17 09:37:23 CET 2010 [pcwindci] /etc/sysconfig > date Wed Nov 17 09:34:46 CET 2010 [pcwindci] /etc/sysconfig > date Wed Nov 17 09:37:23 CET 2010 This makes using Hudson impossible in my environment. I know my setup and would like to completely disable the check for modifications.

          I just ran into this because we are using network storage and the storage system time skewed earlier than the server time and the JUnit output files ended up dated earlier than the build. This just happened after a year of running with no issue, so it took a while to track down.

          I'm not sure I really understand the use case for skipping files based on time anyway. In my case, this is just a single master build which starts with a clean, so all the files generated by the build are good to process. The system clock issues if they occur are not really relevant for me.

          Looking at TestResult.java, this issue is almost resolved in the logic. There is a flag for ignoring the timestamp issues, it needs to be elevated to a project configurable setting. Currently it is defined in that class as

          private static final boolean checkTimestamps = true; // TODO: change to System.getProperty

          NetAppBlueDevil added a comment - I just ran into this because we are using network storage and the storage system time skewed earlier than the server time and the JUnit output files ended up dated earlier than the build. This just happened after a year of running with no issue, so it took a while to track down. I'm not sure I really understand the use case for skipping files based on time anyway. In my case, this is just a single master build which starts with a clean, so all the files generated by the build are good to process. The system clock issues if they occur are not really relevant for me. Looking at TestResult.java, this issue is almost resolved in the logic. There is a flag for ignoring the timestamp issues, it needs to be elevated to a project configurable setting. Currently it is defined in that class as private static final boolean checkTimestamps = true; // TODO: change to System.getProperty

          ebann added a comment -

          I have many Test jobs and one job that build up the whole result.

          As some jobs may take a long time to end, the test-ender job refuses to publish junit results.

          18:12:32 Test reports were found but none of them are new. Did tests run?
          18:12:32 For example, /tmp/jenkins_slave/workspace/TESTS_ENDER/TEST-Statistic.AllTests.xml is 16 hr old
          18:12:32
          18:12:32 Build step 'Publier le rapport des résultats des tests JUnit' changed build result to FAILURE

          ebann added a comment - I have many Test jobs and one job that build up the whole result. As some jobs may take a long time to end, the test-ender job refuses to publish junit results. 18:12:32 Test reports were found but none of them are new. Did tests run? 18:12:32 For example, /tmp/jenkins_slave/workspace/TESTS_ENDER/TEST-Statistic.AllTests.xml is 16 hr old 18:12:32 18:12:32 Build step 'Publier le rapport des résultats des tests JUnit' changed build result to FAILURE

          I have multi-module maven project, which rebuilds only changed modules. I want to display test results for old modules too. But this issue is a blocker for this.

          Stanislav Tyurikov added a comment - I have multi-module maven project, which rebuilds only changed modules. I want to display test results for old modules too. But this issue is a blocker for this.

          Mark Waite added a comment -

          Unreliable clocks on build machines and on file systems which support build machines seem like they will create many more problems than these few that you've detected.

          If you have reliable clocks and are still seeing the issue, then you could consider the work around of adding a build step which updates the time stamp of the result files. The Linux command "touch" will update the time stamp of one or more files and similar commands are available for Windows and other platforms.

          Mark Waite added a comment - Unreliable clocks on build machines and on file systems which support build machines seem like they will create many more problems than these few that you've detected. If you have reliable clocks and are still seeing the issue, then you could consider the work around of adding a build step which updates the time stamp of the result files. The Linux command "touch" will update the time stamp of one or more files and similar commands are available for Windows and other platforms.

          Radim Kubacki added a comment -

          More valid use-case then shifted clocks across build machines is a Gradle build. If I commit a change that has no impact on tests than Gradle skips their execution. But Jenkins tries to be smarter and marks the build as failed. See a longer version with workarounds at http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/

          Sounds more like a bug than RFE to me.

          Radim Kubacki added a comment - More valid use-case then shifted clocks across build machines is a Gradle build. If I commit a change that has no impact on tests than Gradle skips their execution. But Jenkins tries to be smarter and marks the build as failed. See a longer version with workarounds at http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/ Sounds more like a bug than RFE to me.

          Matthias Heil added a comment -

          Seems to be that this kind of feature is not wanted in the core, and that users should switch to the xunit plugin to publish junit results.

          Matthias Heil added a comment - Seems to be that this kind of feature is not wanted in the core, and that users should switch to the xunit plugin to publish junit results.

          n0mer added a comment -

          n0mer added a comment - Possible workaround for Gradle: http://www.practicalgradle.org/blog/2011/06/incremental-tests-with-jenkins/

          Kent Bolton added a comment -

          Below is code related to this issue.

          Looks like the magic number Jenkins core has hardcoded insite it is 1000 miliseconds. Code below in a patch here:

          http://git.eclipse.org/c/hudson/org.eclipse.hudson.core.git/commit?id=97587dde35e862fabbd8b5b4b0136a786cfcff37 .
          This is an issue when using VMs where time can drift quite quickly. Would be good to disable this test, or to be able to change the magic number of seconds in configuration.

          @@ -151,21 +144,22 @@ public final class TestResult extends MetaTabulatedResult {
          }
          }
          if(!parsed) {
          + if (!parsed) {
          long localTime = System.currentTimeMillis();
          if(localTime < buildTime-1000) /margin/
          // build time is in the the future. clock on this slave must be running behind
          + if (localTime < buildTime - 1000) /margin/ // build time is in the the future. clock on this slave must be running behind
          +

          { throw new AbortException( - "Clock on this slave is out of sync with the master, and therefore \n" + - "I can't figure out what test results are new and what are old.\n" + - "Please keep the slave clock in sync with the master."); + "Clock on this slave is out of sync with the master, and therefore \n" + + "I can't figure out what test results are new and what are old.\n" + + "Please keep the slave clock in sync with the master."); + }

          Kent Bolton added a comment - Below is code related to this issue. Looks like the magic number Jenkins core has hardcoded insite it is 1000 miliseconds. Code below in a patch here: http://git.eclipse.org/c/hudson/org.eclipse.hudson.core.git/commit?id=97587dde35e862fabbd8b5b4b0136a786cfcff37 . This is an issue when using VMs where time can drift quite quickly. Would be good to disable this test, or to be able to change the magic number of seconds in configuration. @@ -151,21 +144,22 @@ public final class TestResult extends MetaTabulatedResult { } } if(!parsed) { + if (!parsed) { long localTime = System.currentTimeMillis(); if(localTime < buildTime-1000) /margin/ // build time is in the the future. clock on this slave must be running behind + if (localTime < buildTime - 1000) /margin/ // build time is in the the future. clock on this slave must be running behind + { throw new AbortException( - "Clock on this slave is out of sync with the master, and therefore \n" + - "I can't figure out what test results are new and what are old.\n" + - "Please keep the slave clock in sync with the master."); + "Clock on this slave is out of sync with the master, and therefore \n" + + "I can't figure out what test results are new and what are old.\n" + + "Please keep the slave clock in sync with the master."); + }

          Casey McGinty added a comment - - edited

          Please turn this check into a configuration option. This breaks any multi-project Gradle build running tests.

          Casey McGinty added a comment - - edited Please turn this check into a configuration option. This breaks any multi-project Gradle build running tests.

          Rolf Geuenich added a comment -

          Hi,

          I want to run "JUnit Result Archiver" belated on a nightly build to test different options and need the option to configure the amount of time for difference, which is ok. Or an option to disable this check at all.
          This case is 5 years old and no one will fix it?
          I hope someone will fix it.

          Rolf Geuenich added a comment - Hi, I want to run "JUnit Result Archiver" belated on a nightly build to test different options and need the option to configure the amount of time for difference, which is ok. Or an option to disable this check at all. This case is 5 years old and no one will fix it? I hope someone will fix it.

          Slawek Wieczorkowski added a comment - - edited

          Hi,

          If you work under linux systems, you could try with command:
          touch <your_junit_file.xml>
          e.g touch mytest-junit.xml

          then system will put just current date as modification date and JUnit publish plugin should accept.
          I already tested it and work. I think similar modification date/time command can be run under Windows.

          Slawek Wieczorkowski added a comment - - edited Hi, If you work under linux systems, you could try with command: touch <your_junit_file.xml> e.g touch mytest-junit.xml then system will put just current date as modification date and JUnit publish plugin should accept. I already tested it and work. I think similar modification date/time command can be run under Windows.

          Mariano Cerdeiro added a comment - - edited

          Hi,

          @slawek Chaining the timestamp with touch is a workaround not a solution.

          In my opinion the best solution is to make this check configurable with 2 parameters, one configuration parameter to indicate if enable or disable and the second one if enable to indicate the maximal allowed timestamp difference.

          Mariano.-

          Mariano Cerdeiro added a comment - - edited Hi, @ slawek Chaining the timestamp with touch is a workaround not a solution. In my opinion the best solution is to make this check configurable with 2 parameters, one configuration parameter to indicate if enable or disable and the second one if enable to indicate the maximal allowed timestamp difference. Mariano.-

          I agree with mcerdeiro on the solution.

          I'm facing this issue when the code was not changed only configuration files that are not affecting the test.

          Attila Blenesi added a comment - I agree with mcerdeiro on the solution. I'm facing this issue when the code was not changed only configuration files that are not affecting the test.

          +1 for Mariano Cerdeiro's suggested solution. We've got builds failing because of 5 second differences.

          Kajsa Anderson added a comment - +1 for Mariano Cerdeiro's suggested solution. We've got builds failing because of 5 second differences.

          Gradle 4.0 moved build cache from incubating state to production. Build cache among other things caches JUnit test results, which results in continuous failures. It actually blocks our implementation of pipelines.

          Alex Dubrouski added a comment - Gradle 4.0 moved build cache from incubating state to production. Build cache among other things caches JUnit test results, which results in continuous failures. It actually blocks our implementation of pipelines.

          Justin McMillan added a comment - - edited

          I'm also blocked by this – I use a single Jenkins pipeline job to execute tests (across multiple environments – we want reporting on a per-environment level), and then another job executes which pulls those JUnit results into the workspace, and builds a report. I am encountering this on XMLs that are ~5 seconds "old".

           

          This Powershell script works, but still not the greatest:

          $files = Get-ChildItem "${WORKSPACE}"
            foreach ($i in $files) {
            $i.LastWriteTime = $(Get-Date)
          }
          

          Justin McMillan added a comment - - edited I'm also blocked by this – I use a single Jenkins pipeline job to execute tests (across multiple environments – we want reporting on a per-environment level), and then another job executes which pulls those JUnit results into the workspace, and builds a report. I am encountering this on XMLs that are ~5 seconds "old".   This Powershell script works, but still not the greatest: $files = Get-ChildItem "${WORKSPACE}" foreach ($i in $files) { $i.LastWriteTime = $(Get-Date) }

          Mike Kobit added a comment -

          Echoing what alex_dubrouski said, this totally breaks incremental builds from tools like Gradle.

          Mike Kobit added a comment - Echoing what alex_dubrouski said, this totally breaks incremental builds from tools like Gradle.

          Is there any progress on this? We are also hit by this, so currently we have only the option to disable junit recording at all...

          Christoph Läubrich added a comment - Is there any progress on this? We are also hit by this, so currently we have only the option to disable junit recording at all...

          Per Östman added a comment -

          This one is nine (9) years old. Come on, please fix this!

          Per Östman added a comment - This one is nine (9) years old. Come on, please fix this!

          Zak Taccardi added a comment -

          Note: I filed this too on GitHub which is related https://github.com/jenkinsci/junit-plugin/issues/201

          Zak Taccardi added a comment - Note: I filed this too on GitHub which is related  https://github.com/jenkinsci/junit-plugin/issues/201

          Basil Crow added a comment -

          Basil Crow added a comment - Fixed in jenkinsci/junit-plugin#384 . Released in 1119.va_a_5e9068da_d7 .

            olamy Olivier Lamy
            bshine bshine
            Votes:
            54 Vote for this issue
            Watchers:
            50 Start watching this issue

              Created:
              Updated:
              Resolved: