Details
-
Type:
Bug
-
Status: Resolved (View Workflow)
-
Priority:
Major
-
Resolution: Cannot Reproduce
-
Component/s: core
-
Labels:None
-
Similar Issues:
Description
After recently upgrading to Jenkins LTS v1.596.3 we noticed that dozens, if not hundreds of builds per day are being reported as having "inconsistent" time stamps. Here are a few specific details that may help isolate the problem:
- I have confirmed that the build number (ie: integer values) for the offending builds as reported on the Jenkins dashboard is consistent with what is recorded in the build.xml files on the server, which are also consistent with the symbolic links stored on disk that point to those builds
- I have confirmed that the build IDs (ie: the time-stamped formatted identifiers) for the offending builds as reported on the Jenkins dashboard are consistent with what is recorded in the build.xml files on the server, which are also consistent with the names of the log folders for each of these builds
- I have confirmed that the <startTime> elements in the build.xml files resolve to the same build IDs used to store and organize the builds
- I have confirmed that none of these builds interfere or overlap with previous or subsequent builds of the same jobs (ie: if offending build is #536, I have confirmed that it does in fact run after build #535 and before build #537, and that all 3 such builds have consistent references to build numbers and IDs throughout the logs)
- After upgrading to this version I discovered that all build agents connected to our master are now reporting their clocks as being out of sync, despite the fact that I have confirmed that this is not actually correct (ie: as described under
JENKINS-18671). I suspect that this incorrect time reporting may be somehow confusing Jenkins into thinking builds have incorrect timestamps when in fact they don't.
I probably should also mention that we've attempted to upgrade to the latest LTS edition, v1.609.1, in the off chance there may be some improvements included there that may resolve this issue (ie: since it appears to have re-structured how builds are stored on disk, eliminating the 'time stamp' labelling mechanism, which may correct this problem) however due to other production-stop defects included in that release (ie: JENKINS-28513) we were forced to remain on the 1.596.3 version for the time being.
Attachments
Issue Links
- is related to
-
JENKINS-18671 Clock Difference broken on Manage Nodes page
-
- Resolved
-
-
JENKINS-23316 Some projects have builds whose timestamps are inconsistent
-
- Closed
-
-
JENKINS-18289 Detect & repair out-of-order build records
-
- Resolved
-
NOTE: Based on my brief review of the source code in the link provided above, the 'inconsistent' build labels are defined as build numbers with ascending values that have descending time stamps, or vice versa. As I mentioned in my bug report I have confirmed that none of the builds being reported by our Jenkins instance fall into that category.
When I examine the build.xml files on disk on the Jenkins master, the build numbers, time stamps, folder names and symbolic links are all consistent with one another, and all of the values are always in ascending order (ie: build 123 has a time stamp larger than build 122 and smaller than build 124, in every reported case).
I suppose this point may be moot if the underlying architecture has been changed in such a way that prevents any of these kinds of problems from happening, but I just thought I should mention it in case a similar bug may still be present in the latest version as well.