NOTE: Based on my brief review of the source code in the link provided above, the 'inconsistent' build labels are defined as build numbers with ascending values that have descending time stamps, or vice versa. As I mentioned in my bug report I have confirmed that none of the builds being reported by our Jenkins instance fall into that category.

When I examine the build.xml files on disk on the Jenkins master, the build numbers, time stamps, folder names and symbolic links are all consistent with one another, and all of the values are always in ascending order (ie: build 123 has a time stamp larger than build 122 and smaller than build 124, in every reported case).

I suppose this point may be moot if the underlying architecture has been changed in such a way that prevents any of these kinds of problems from happening, but I just thought I should mention it in case a similar bug may still be present in the latest version as well.

Hopefully the other product-stop bugs that were introduced by the 1.609.1 update can be fixed sooner rather than later so we can get those fixes as well.

I know about the queue/build blocker/block while something else is building issues, and expect 1.609.3 to have them resolved. Anything else a blocker for you?

No idea re JENKINS-18671. I doubt you're seeing the same issue. Make sure your nodes are all in synced time.

Anything else a blocker for you?

One other behavioral change in 1.609.1 is that the BUILD_ID property was changed from a time stamp to a numeric identifier. All of our build configurations and releases processes were dependent upon a builds identifier being a time stamp, and we more-or-less use that BUILD_ID variable everywhere (scripts, code, etc.). I did find a plugin that allows one to create a new time-stamp based environment variable for each build (Zen Time Stamp) but we'll need to reconfigure our infrastructure to use the new variable name throughout.

We hit a LOT of problems with the SVN plugin as well but I believe we've finally sorted all those out.

I'm also currently investigating some bugs with the way the new Jenkins version handles jobs that are enqueued for builds. There are a couple of cases where jobs may get deadlocked in the queue for various reasons. Still trying to isolate specific use cases though before I file any more bugs.

NOTE: We may have to stick with v1.596.3 for the time being. Assuming there are a great number of new changes / improvements / fixes included in the next LTS release, we'll likely have to invest considerable time and expense making sure none of those additional changes break anything else anew.

No idea re JENKINS-18671. I doubt you're seeing the same issue. Make sure your nodes are all in synced time.

All of our agents, and the master, are synchronized using the same NTP server and I have confirmed via direct examination of the agents that all the physical clocks on all machines are in fact correct and in sync with each other (to within < 1S variance at least). It's only the Jenkins dashboard that suggests otherwise. If you believe JENKINS-18671 (or the underlying code change from JENKINS-18438) just let me know and I'll create a new defect for that.

If this problem is in fact a new / different bug, and it is a contributing factor to the inconsistent time stamps being reported, then I would say that would be a significant issue that we'd also want fixed in the next LTS release as well. We can't have build logs getting corrupted even if the problem is intermittent. We use Jenkins as a central service now for all of our production work and we need to be able to rely on its ability to preserve the details of those build operations for auditing purposes. (although, I guess this is a problem we're currently stuck with on 1.596.3 so it hopefully wouldn't get any worse if we were to move to a newer LTS version at least).

We hit a LOT of problems with the SVN plugin as well but I believe we've finally sorted all those out.

More information would be nice, but be aware of JENKINS-21785 etc. (i.e. 'Additional Credentials' is deliberately needed in many situations for security concerns).

I'm also currently investigating some bugs with the way the new Jenkins version handles jobs that are enqueued for builds.

I was referring to those in my earlier comment. I expect further fixes in 1.609.3.

be aware of JENKINS-21785 etc

That was the issue that help us resolve the major SVN issues we were experiencing. Thanks for the link though.

The difficulty involved updating our 1000+ jobs to include the appropriate credentials definitions, which were not a requirement in previous LTS releases. Most, but not all, of our jobs monitor SVN repositories but it was a bit tricky to ensure that all the affected jobs, and only those affected by this change, got updated. And unfortunately we gleaned no benefit from the improved security as all of our Jenkins instances are run behind a firewall and operate within a Windows domain so only trusted system have access to our key services. The net result for us was a lot of superfluous work for little to no gain.

For future reference, any time that you include an invasive change like this in an LTS edition you probably should provide an easier way for users to upgrade their configurations instead of leaving them up to their own vices to hack around the changes. It would greatly reduce the pain of upgrading.

I was referring to those in my earlier comment.

Sorry - my misunderstanding.