-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
After recently upgrading to Jenkins LTS v1.596.3 we noticed that dozens, if not hundreds of builds per day are being reported as having "inconsistent" time stamps. Here are a few specific details that may help isolate the problem:
- I have confirmed that the build number (ie: integer values) for the offending builds as reported on the Jenkins dashboard is consistent with what is recorded in the build.xml files on the server, which are also consistent with the symbolic links stored on disk that point to those builds
- I have confirmed that the build IDs (ie: the time-stamped formatted identifiers) for the offending builds as reported on the Jenkins dashboard are consistent with what is recorded in the build.xml files on the server, which are also consistent with the names of the log folders for each of these builds
- I have confirmed that the <startTime> elements in the build.xml files resolve to the same build IDs used to store and organize the builds
- I have confirmed that none of these builds interfere or overlap with previous or subsequent builds of the same jobs (ie: if offending build is #536, I have confirmed that it does in fact run after build #535 and before build #537, and that all 3 such builds have consistent references to build numbers and IDs throughout the logs)
- After upgrading to this version I discovered that all build agents connected to our master are now reporting their clocks as being out of sync, despite the fact that I have confirmed that this is not actually correct (ie: as described under
JENKINS-18671). I suspect that this incorrect time reporting may be somehow confusing Jenkins into thinking builds have incorrect timestamps when in fact they don't.
I probably should also mention that we've attempted to upgrade to the latest LTS edition, v1.609.1, in the off chance there may be some improvements included there that may resolve this issue (ie: since it appears to have re-structured how builds are stored on disk, eliminating the 'time stamp' labelling mechanism, which may correct this problem) however due to other production-stop defects included in that release (ie: JENKINS-28513) we were forced to remain on the 1.596.3 version for the time being. 
- is related to
-
-
- Resolved
-
-
-
- Closed
-
-
JENKINS-18289 Detect & repair out-of-order build records
-
- Resolved
-
[JENKINS-29060] Jenkins incorrectly reports timestamps are inconsistent
Kevin Phillips
created issue -
Kevin Phillips
made changes -
| Link |
New:
This issue is related to |
Kevin Phillips
made changes -
| Link |
New:
This issue is related to |
Kevin Phillips
made changes -
| Link |
New:
This issue is related to |
Kevin Phillips
made changes -
| Description |
Original:
After recently upgrading to Jenkins LTS v1.596.3 we noticed that dozens, if not hundreds of builds per day are being reported as having "inconsistent" time stamps. Here are a few specific details that may help isolate the problem: * I have confirmed that the build number (ie: integer values) for the offending builds as reported on the Jenkins dashboard is consistent with what is recorded in the build.xml files on the server, which are also consistent with the symbolic links stored on disk that point to those builds * I have confirmed that the build IDs (ie: the time-stamped formatted identifiers) for the offending builds as reported on the Jenkins dashboard are consistent with what is recorded in the build.xml files on the server, which are also consistent with the names of the log folders for each of these builds * I have confirmed that the <startTime> elements in the build.xml files resolve to the same build IDs used to store and organize the builds * I have confirmed that none of these builds interfere or overlap with previous or subsequent builds of the same jobs (ie: if offending build is #536, I have confirmed that it does in fact run after build #535 and before build #537, and that all 3 such builds have consistent references to build numbers and IDs throughout the logs) * After upgrading to this version I discovered that all build agents connected to our master are now reporting their clocks as being out of sync, despite the fact that I have confirmed that this is not actually correct (ie: as described under I probably should also mention that we've attempted to upgrade to the latest LTS edition, v1.609.1, in the off chance there may be some improvements included there that may resolve this issue (ie: since it appears to have re-structured how builds are stored on disk, eliminating the 'time stamp' labelling mechanism, which may correct this problem) however due to other critical breaking changes included in that release (ie: |
New:
After recently upgrading to Jenkins LTS v1.596.3 we noticed that dozens, if not hundreds of builds per day are being reported as having "inconsistent" time stamps. Here are a few specific details that may help isolate the problem: * I have confirmed that the build number (ie: integer values) for the offending builds as reported on the Jenkins dashboard is consistent with what is recorded in the build.xml files on the server, which are also consistent with the symbolic links stored on disk that point to those builds * I have confirmed that the build IDs (ie: the time-stamped formatted identifiers) for the offending builds as reported on the Jenkins dashboard are consistent with what is recorded in the build.xml files on the server, which are also consistent with the names of the log folders for each of these builds * I have confirmed that the <startTime> elements in the build.xml files resolve to the same build IDs used to store and organize the builds * I have confirmed that none of these builds interfere or overlap with previous or subsequent builds of the same jobs (ie: if offending build is #536, I have confirmed that it does in fact run after build #535 and before build #537, and that all 3 such builds have consistent references to build numbers and IDs throughout the logs) * After upgrading to this version I discovered that all build agents connected to our master are now reporting their clocks as being out of sync, despite the fact that I have confirmed that this is not actually correct (ie: as described under I probably should also mention that we've attempted to upgrade to the latest LTS edition, v1.609.1, in the off chance there may be some improvements included there that may resolve this issue (ie: since it appears to have re-structured how builds are stored on disk, eliminating the 'time stamp' labelling mechanism, which may correct this problem) however due to other production-stop defects included in that release (ie: |
Kevin Phillips
added a comment - I probably should also pose the question: what criteria does Jenkins use to decide whether the timestamps on a job are "inconsistent" or not? (ie: what are they inconsistent with?) As described above, so far as I can tell all of the data stored in the actual build logs looks correct to me, however I may be overlooking something.
Kevin Phillips
added a comment - I probably should also pose the question: what criteria does Jenkins use to decide whether the timestamps on a job are "inconsistent" or not? (ie: what are they inconsistent with?) As described above, so far as I can tell all of the data stored in the actual build logs looks correct to me, however I may be overlooking something. Unfortunately, 1.596.x LTS is no longer supported so I'll have to reject this bug report. Since 1.597 changed the storage layout of builds on disk, the reported issue should no longer occur there.
Consider asking for advice on the jenkinsci-users mailing list, or in #jenkins on Freenode.
| Resolution | New: Cannot Reproduce [ 5 ] | |
| Status | Original: Open [ 1 ] | New: Resolved [ 5 ] |
The out-of-order monitoring code is at https://github.com/jenkinsci/jenkins/tree/stable-1.596/core/src/main/java/jenkins/diagnostics/ooom, maybe that helps.
Basically this occurs when a build with a higher build number has an earlier start date. Note that other bugs may result in the same build number being assigned multiple times (and only one of those can be loaded by Jenkins). You need to compare the files on disk, not the loaded build records.
This problem seems very similar, although slightly different in it's description, to
JENKINS-18289. I've added a related link for reference in case it's helpful.