When you select a job, are the dates in the build history widget correct? Is the date correct when viewing a single build?

Old builds' dates are incorrect (01.01.1970), new builds (after having updated to 1.597) are correct. Same for viewing a single build.

This seems pretty bad. Assigning to Jesse Glick speculatively.

Hmm, I definitely wrote (and tested) code to create <timestamp> in migrated build.xml records. Can you attach an example of the $JENKINS_HOME/jobs/*/builds/*/build.xml for one of the migrated builds showing a zero timestamp? Also if you happen to have your Jenkins log from the first startup after 1.597, attach that, in case there were any warnings. And attach the legacyIds file from the builds parent directory.

tdtappe: Did you configure a nondefault build record directory in the global configuration? I.e. are the builds for a job NOT in .../job/foo/builds ?

Did you configure a nondefault build record directory in the global configuration?

This should not matter, since the call to RunIdMigrator takes into account Job.getBuildDir, but of course this would be interesting information if so.

(The reverse migration tool cannot handle nondefault build directories, unfortunately.)

No, I did not configure non-default build directories. All my projects seem to have their builds in the default build directory.

AFAICS all legacyIds files have size 0.

Could it be a matter of whitespaces in the path?

Sent a build.xml to Jesse privately.

This might be an important part of the jenkins log:

INFO: Downloading jenkins.war
...
...
INFO: Augmented all extensions
...
...
WARNUNG: Build record migration (https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-24380+Migration) is one-way. If you need to downgrade Jenkins, run: java -classpath "D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\jenkins-core-1.597.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\commons-io-2.4.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\stapler-1.234.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\ant-1.8.4.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\commons-lang-2.6.jar" jenkins.model.RunIdMigrator "d:\NightlyBuilds\Jenkins"
...
...
20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate
INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 13.00-dev\builds
20.01.2015 16:22:20 jenkins.model.RunIdMigrator doMigrate
WARNUNG: failed to rename 2013-09-04_01-36-36 to 275
20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate
INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.01.00-patched\builds
20.01.2015 16:22:20 jenkins.model.RunIdMigrator doMigrate
WARNUNG: failed to rename 2015-01-16_01-03-35 to 175
20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate
INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 13.02-fibu\builds
20.01.2015 16:22:20 jenkins.model.RunIdMigrator doMigrate
WARNUNG: failed to rename 2014-08-29_21-30-00 to 176
20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate

If the rename is performed without first deleting the build number symlink, maybe this fails on Windows?

I glanced at one of the build.xml files and it was in the old format, which is clearly the problem. I do recall testing migration on Windows but there are a thousand and one things that can go wrong on a Windows filesystem and it looks like you hit one of them. Will see if I can find a cause.

The build number symlinks are deleted in one pass, then the build.xml modifications and directory renames happen in a second pass. There are no could not delete build number symlink warnings so it does not seem that this is the problem.

Can you show me a long directory listing (IIRC dir suffices in a command shell) of one of the affected builds directories?

I suspect that the problem is that on some Windows systems, build number “symlinks” are really text files pointing to the target, and that these are not handled.

Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
core/src/main/java/jenkins/model/RunIdMigrator.java
core/src/test/java/jenkins/model/RunIdMigratorTest.java
http://jenkins-ci.org/commit/jenkins/056b446480d9f24619c2c30ebdf9df6122a9b653
Log:
[FIXED JENKINS-26519] Accept Windows text file quasi-symlinks for build numbers.

In the interest of expediency I pushed what I think is a fix to the rc branch for 1.598 (which will I guess come out on Monday as usual), and am validating a merge to master. It would not be a bad idea to grab the jenkins.war from https://jenkins.ci.cloudbees.com/job/core/job/jenkins_rc_branch/317/ once it completes and try running it.

In order to force a rerun of migration when using the patched version of Jenkins:

1. Shut down Jenkins.
2. Delete all %JENKINS_HOME%\jobs*\builds\legacyId files. (They are just caches and will be recreated as needed.)
3. Start Jenkins again.
4. Check the log file. You should see the Migrating build records messages again, but should not see the warnings.

Code changed in jenkins
User: Jesse Glick
Path:
core/src/test/java/jenkins/model/RunIdMigratorTest.java
http://jenkins-ci.org/commit/jenkins/c997ac0c865ca4866ffda0e9ae93fe818c311cba
Log:
JENKINS-26519 Confirming that it is safe to rerun migration by just deleting legacyId files.

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/jenkins/model/RunIdMigrator.java
http://jenkins-ci.org/commit/jenkins/fccc34987f03a2716f3744a998cfe7d9127abafc
Log:
JENKINS-26519 Unreproducible NPE from unmigrateJobsDir.

Integrated in jenkins_main_trunk #3940
JENKINS-26519 Confirming that it is safe to rerun migration by just deleting legacyId files. (Revision c997ac0c865ca4866ffda0e9ae93fe818c311cba)
[FIXED JENKINS-26519] Accept Windows text file quasi-symlinks for build numbers. (Revision 056b446480d9f24619c2c30ebdf9df6122a9b653)

Result = SUCCESS
jesse glick : c997ac0c865ca4866ffda0e9ae93fe818c311cba
Files :

• core/src/test/java/jenkins/model/RunIdMigratorTest.java

jesse glick : 056b446480d9f24619c2c30ebdf9df6122a9b653
Files :

• changelog.html
• core/src/test/java/jenkins/model/RunIdMigratorTest.java
• core/src/main/java/jenkins/model/RunIdMigrator.java

Integrated in jenkins_main_trunk #3941
JENKINS-26519 Unreproducible NPE from unmigrateJobsDir. (Revision fccc34987f03a2716f3744a998cfe7d9127abafc)

Result = SUCCESS
jesse glick : fccc34987f03a2716f3744a998cfe7d9127abafc
Files :

• core/src/main/java/jenkins/model/RunIdMigrator.java

Switched to Jenkins ver. 1.598-SNAPSHOT (rc-01/21/2015 18:23 GMT-jenkins) after having deleted the legacyIds files. But still:

...
...
22.01.2015 08:13:43 jenkins.model.RunIdMigrator migrate
INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02-dev\builds
22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate
WARNUNG: failed to rename 2014-08-30_00-58-41 to 79
22.01.2015 08:13:43 jenkins.model.RunIdMigrator migrate
INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 12.01.02-patched\builds
22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate
WARNUNG: found no build.xml in 2012-12-07_00-13-31
22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate
WARNUNG: failed to rename 2013-03-01_00-22-07 to 271
22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate
WARNUNG: failed to rename 2015-01-20_01-45-40 to 82
22.01.2015 08:13:43 jenkins.model.RunIdMigrator migrate
...
...

Listings of job and job/builds directories.

Tried the new war (jenkins_main_trunk #3941) it didn't work for me. I have the same set-up Windows running under a service with many legacy builds.
see err log file attached (jenkins.err.log_on-startup.txt)

NOTE: What did work was adding
<timestamp>1420730669680</timestamp>
To the build.xml file. This resolved all my issues for this build.

This value was missing the only value present was
<startTime>1420730669680</startTime>

Obviously doing this for all 10000+ builds will require a script, but that seems to resolve it.

Steve Haworth timestamp is never present in old builds. The migrator adds it. Your case looks to have at least two problems:

1. failed to rename despite no prior error reported deleting the symlink. (The directory listing confirms that this is really a symlink, not a text file, so it is a distinct problem from the one I just fixed.) I would recommend stepping through RunIdMigrator in a debugger, starting with your original build directory (probably recoverable just be deleting all legacyId files).
2. And then in a few cases could not find <number>, suggesting an unexpected or corrupt build.xml format.

If you are willing to try up-to-the-minute dev builds, there are a few places where I think I could add better diagnostics.

Sure!

Jesse if the question was aimed at me "yes I am".

I am also still experiencing this issue with 1.598.

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/jenkins/model/RunIdMigrator.java
core/src/test/java/jenkins/model/RunIdMigratorTest.java
http://jenkins-ci.org/commit/jenkins/0633beef0868c59a7e853731aee5f5c0745fa3fc
Log:
JENKINS-26519 Attempting to improve diagnostics in case rename fails.

To anyone still seeing issues, this build ought to have more informative error messages for the “failure to rename”.

Steve Haworth your could not find <number> could only be diagnosed by seeing the named build.xml files.

Integrated in jenkins_main_trunk #3953
JENKINS-26519 Attempting to improve diagnostics in case rename fails. (Revision 0633beef0868c59a7e853731aee5f5c0745fa3fc)

Result = SUCCESS
jesse glick : 0633beef0868c59a7e853731aee5f5c0745fa3fc
Files :

• core/src/test/java/jenkins/model/RunIdMigratorTest.java
• core/src/main/java/jenkins/model/RunIdMigrator.java

So here is part of the log with version #3953. Or what do you need? Do I have to enable some logging?

28.01.2015 08:04:37 jenkins.model.RunIdMigrator migrate
INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds
28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate
WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-10-16_10-32-01
java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\68 already exists
at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285)
at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262)
at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
at hudson.model.Job.onLoad(Job.java:194)
at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
at hudson.model.Project.onLoad(Project.java:98)
at hudson.model.Items.load(Items.java:279)
at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate
WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-05_16-13-26
java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\69 already exists
at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285)
at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262)
at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
at hudson.model.Job.onLoad(Job.java:194)
at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
at hudson.model.Project.onLoad(Project.java:98)
at hudson.model.Items.load(Items.java:279)
at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate
WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-05_16-18-56
java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\70 already exists
at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285)
at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262)
at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
at hudson.model.Job.onLoad(Job.java:194)
at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
at hudson.model.Project.onLoad(Project.java:98)
at hudson.model.Items.load(Items.java:279)
at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate
WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-19_17-16-21
java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\71 already exists
at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285)
at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262)
at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
at hudson.model.Job.onLoad(Job.java:194)
at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
at hudson.model.Project.onLoad(Project.java:98)
at hudson.model.Items.load(Items.java:279)
at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate
WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-19_17-20-07
java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\72 already exists
at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285)
at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262)
at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
at hudson.model.Job.onLoad(Job.java:194)
at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
at hudson.model.Project.onLoad(Project.java:98)
at hudson.model.Items.load(Items.java:279)
at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
28.01.2015 08:04:37 jenkins.InitReactorRunner$1 onAttained
INFO: Loaded all jobs

Log attached - Error Log SH - 3953 .txt. Running with build #3953.

Yes Understand the issue with "could not find <number>", a couple of disassociated builds with no build.xml, so I'll manually delete those.
Main error persists though looks to be same as tdtappe's debug output

Attached a jpeg of the builds directory for a build (builds directory.jpg)
NOTE: Build 1417 seems to have been migrated
Builds 1418-1425 fail to migrate see log above.
Builds 1426-1427 have been built with new format records

Heiko Tappe so we have confirmation that the problem involves the target of the move already existing, yet the migrator apparently did not even attempt to delete it. Can you give me a directory listing of d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds please? (dir) And if for example 71 is a regular file, can you please attach it? I will attempt to strengthen the migrator here.

Steve Haworth:

Understand the issue with "could not find <number>", a couple of disassociated builds with no build.xml

This warning ought to be printed only when build.xml does exist, but has unexpected content. Else you would see found no build.xml in … in this spot. So please attach the build.xml.

I am pushing another commit which tries harder to clean out existing numeric rename targets. Will post a download link when available. This is looking like some weirdness in how Windows handles symbolic links, which varies not only according to Windows version but also depending on filesystem and system configuration. Unfortunately Jenkins makes heavy use of the filesystem to store data, and lots of things behave unpredictably on Windows servers.

You are encouraged to use Java 7 (or 8) rather than Java 6, as you will get better diagnostics: the file APIs available in Java 6- are poor.

You should also go to /log/ (Log Manager) and create a custom log recording jenkins.model.RunIdMigrator at FINE. After the migrator runs at startup, logs are available in that URL. If they are truncated (only latest messages shown), you can install the Support Core plugin to capture fuller logs.

Directory listing of "KomalogWin - tag\builds"

Are you running Java 6, or 7+?

Java 6

OK, guess I need to retest everything in Windows on Java 6 then.

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/jenkins/model/RunIdMigrator.java
http://jenkins-ci.org/commit/jenkins/75c8c31f2fdc98081a0446dd3ee03da7243304a1
Log:
JENKINS-26519 More diagnostics in case a numeric rename target already exists.

Here is the latest round of fixes. In parallel I will see if I can reproduce the issue using Java 6.

Excerpt from custom logger with your newest build:

Jan 28, 2015 4:12:36 PM INFO jenkins.model.RunIdMigrator migrate
Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02.00-patched\builds
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping deletion of directory 116
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping deletion of directory 121
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping deletion of directory 122
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping deletion of directory 123
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping non-directory lastStableBuild
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping deletion of directory 4
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping new build dir 116
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping new build dir 121
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping new build dir 122
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping new build dir 123
Jan 28, 2015 4:12:36 PM WARNUNG jenkins.model.RunIdMigrator doMigrate
found no build.xml in 2014-08-29_03-51-39
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping new build dir 4
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping non-directory lastSuccessfulBuild
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping non-directory lastUnstableBuild
Jan 28, 2015 4:12:36 PM INFO jenkins.model.RunIdMigrator migrate
Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping deletion of directory 68
Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
skipping deletion of directory 69
Jan 28, 2015 4:12:36 PM INFO jenkins.model.RunIdMigrator migrate
Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 12.02-dev\builds

I am guessing that d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02.00-patched\builds\123 etc. are in fact SYMLINKD, not regular directories. The readSymlink call which would normally determine this cannot work on Java 6, and then apparently Windows claims that File.isDirectory based on the target. So I will see if I can reproduce and find a way to fix this case.

Actually not:

Datentr„ger in Laufwerk D: ist DATEN
Volumeseriennummer: 1225-FEC9

Verzeichnis von D:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02.00-patched\builds

Mi 28.01.2015 16:12 <DIR> .
Mi 28.01.2015 16:12 <DIR> ..
Di 20.01.2015 02:57 <DIR> 116
Sa 24.01.2015 02:51 <DIR> 121
Di 27.01.2015 02:51 <DIR> 122
Mi 28.01.2015 02:55 <DIR> 123
Fr 29.08.2014 03:25 <DIR> 2014-08-29_03-51-39
Fr 29.08.2014 02:51 <SYMLINKD> 4 [2014-08-29_03-51-39]
Fr 19.12.2014 02:57 2 lastFailedBuild
Mi 28.01.2015 02:55 3 lastStableBuild
Mi 28.01.2015 02:55 3 lastSuccessfulBuild
Mi 27.08.2014 07:19 2 lastUnstableBuild
Fr 19.12.2014 02:57 2 lastUnsuccessfulBuild
Mi 28.01.2015 16:12 0 legacyIds
6 Datei(en), 12 Bytes
8 Verzeichnis(se), 769.188.323.328 Bytes frei

The question is how you even got these build number symlinks to begin with—as far as I know they are not created when you run Jenkins on Windows using Java 6! (Even if the Windows system supports symlinks generally; IIRC XP does not, and newer systems may or may not.) The only way I can think to make them appear is to run 1.596- using Java 7, at least for a while, then run 1.597+ using Java 6.

OK, the 116+ builds are either new, or got correctly migrated, but then we also got

skipping deletion of directory 4

where this one was an old build with a SYMLINKD that did not get migrated.

I am almost 100% sure that there never has been another Java version than 6 (since 2010) on that machine.

BTW I could most probably switch to Java 7 or even 8 on that machine if it might solve all my problems. But then we (you) wouldn't have a problematic real time scenario any longer to test with Java 6!?

Integrated in jenkins_main_trunk #3957
JENKINS-26519 More diagnostics in case a numeric rename target already exists. (Revision 75c8c31f2fdc98081a0446dd3ee03da7243304a1)

Result = UNSTABLE
jesse glick : 75c8c31f2fdc98081a0446dd3ee03da7243304a1
Files :

• core/src/main/java/jenkins/model/RunIdMigrator.java

Attached 2014-11-26_03-40-28 build.xml

This is one of the files indicated in log "jenkins.err.log_on-startup.txt" as having no number.

Heiko Tappe Well I will check once I get a post-XP test system up. My reading of the code in older versions of Jenkins is that it would never create such a symlink when running on Java 6, but I may have missed something.

Steve Haworth Odd. The build is recorded as having been created by Jenkins 1.590, so before the switch, yet it has <timestamp> and no number, like the new format, but no <id>, like the old format. The migrator intentionally tries the folder rename before the build.xml rewrite, so it should not have been the result of an error midway through migration. So I have no explanation for how you got there and hope this is rare enough.

Jesse: Could the unexpected XML come from running the unmigration script?

Another excerpt with newest build:

Jan 29, 2015 8:50:35 AM WARNUNG jenkins.model.RunIdMigrator doMigrate

failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-10-16_10-32-01
java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\68 already exists
at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297)
at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274)
at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
at hudson.model.Job.onLoad(Job.java:194)
at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
at hudson.model.Project.onLoad(Project.java:98)
at hudson.model.Items.load(Items.java:281)
at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

skipping deletion of directory 5995

Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

skipping deletion of directory 5996

Jan 29, 2015 8:50:35 AM WARNUNG jenkins.model.RunIdMigrator doMigrate

failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-05_16-13-26
java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\69 already exists
at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297)
at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274)
at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
at hudson.model.Job.onLoad(Job.java:194)
at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
at hudson.model.Project.onLoad(Project.java:98)
at hudson.model.Items.load(Items.java:281)
at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

skipping deletion of directory 5997

Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

skipping deletion of directory 5998

Jan 29, 2015 8:50:35 AM INFO jenkins.model.RunIdMigrator migrate

Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.01-dev\builds

Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

skipping deletion of directory 124

Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

skipping deletion of directory 128

Build #3957

Log extract...

29-Jan-2015 10:40:35 jenkins.model.RunIdMigrator doMigrate
WARNING: failed to process C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\2014-11-10_17-23-06
java.io.IOException: C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\273 already exists
at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297)
at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274)
at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
at hudson.model.Job.onLoad(Job.java:194)
at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
at hudson.model.Project.onLoad(Project.java:98)
at hudson.model.Items.load(Items.java:281)
at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
29-Jan-2015 10:40:35 jenkins.model.RunIdMigrator doMigrate
WARNING: failed to process C:\Jenkins\jobs\GEMTEST_Page_84\builds\2014-12-23_14-42-31
java.io.IOException: C:\Jenkins\jobs\GEMTEST_Page_84\builds\1420 already exists
at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297)
at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274)
at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
at hudson.model.Job.onLoad(Job.java:194)
at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
at hudson.model.Project.onLoad(Project.java:98)
at hudson.model.Items.load(Items.java:281)
at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
29-Jan-2015 10:40:35 jenkins.model.RunIdMigrator doMigrate
WARNING: failed to process C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\2014-11-26_13-03-35
java.io.IOException: C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\288 already exists
at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297)
at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274)
at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
at hudson.model.Job.onLoad(Job.java:194)
at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
at hudson.model.Project.onLoad(Project.java:98)
at hudson.model.Items.load(Items.java:281)
at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Jenkins is running on a Virtual Machine with - Windows Server 2008 R2 Standard with Service Pack 1
Java version is - Java Standard Edition Version 8 Update 31

NOTE: Tried experiment... if I manually delete the shortcut directories in a job directory then migration is okay

Yes one workaround would be delete all numeric SYMLINKD, though I am not sure if there is an easy batch command to do that. Still intend to try to reproduce this and offer a proper fix.

Could not reproduce by starting Jenkins 1.596 on Windows 2012 as Administrator under JDK 7u21, creating a project, running a few builds, then stopping and starting a trunk build: builds got correctly migrated. So perhaps only arises under specific circumstances.

Also I checked in jrunscript under these circumstances that the code called by Util.resolveSymlink would work, as would the call to Util.deleteFile.

In the case of Heiko Tappe, running Java 6, resolveSymlink would be expected to fail, explaining the skipping deletion of directory … message, followed by the failure of the migrator. (Which as I mentioned before would not be expected to matter for most people because symlinks should not have been created to begin with; need to confirm this.) This I could fix by just trying deleteFile anyway—if it is really a symlink to a directory, that will work, but if it is really a directory, it should fail because the directory is not empty. It makes me a little nervous because isDirectory would also be true for a real directory created by 1.597+ (or a partly successful earlier migration), and relying on File.delete to fail in this case seems dangerous.

In the case of Steve Haworth, running Java 8, I am puzzled. I see no FINE-level messages here so I assume you did not turn on the custom logger, which might have some crucial information.

Ah, I was wrong, 1.596- on Windows Java 6 did attempt to create symlinks, using JNA, and this does produce the described error. So that is something I can and will fix.

Confusingly, for Windows on Java 6, 1, 2, etc. were created as true symlinks; the text files with build numbers were only used for permalinks like lastStableBuild, meaning my attempted fix in 056b446 was pointless and can probably just be reverted. (One of the unusual cases where “reopening” a bug is really correct.)

Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
core/src/main/java/jenkins/model/RunIdMigrator.java
core/src/test/java/jenkins/model/RunIdMigratorTest.java
http://jenkins-ci.org/commit/jenkins/388c4b5e6ba52037cae117c5c69f9f4156d41401
Log:
[FIXED JENKINS-26519] Build record migration failed on Windows using Java 6.
In this environment, Util.createSymlink and .isSymlink are implemented, but resolveSymlink is not.

Heiko Tappe I think your problem is now fixed in this build, toward 1.600. If you confirm I will try to backport it to the 1.599 release candidate branch so it gets out earlier.

Steve Haworth My latest fix might help in your case, but since you say you are running Java 8, you should not have been hitting it to begin with. Your symptoms are similar; the root cause may or may not be related. I would suggest trying once again with my newest build, assuming you still have the original build directories to reproduce against. First be sure to enable fine logging on this component.

Integrated in jenkins_main_trunk #3958
[FIXED JENKINS-26519] Build record migration failed on Windows using Java 6. (Revision 388c4b5e6ba52037cae117c5c69f9f4156d41401)

Result = SUCCESS
jesse glick : 388c4b5e6ba52037cae117c5c69f9f4156d41401
Files :

• changelog.html
• core/src/test/java/jenkins/model/RunIdMigratorTest.java
• core/src/main/java/jenkins/model/RunIdMigrator.java

Yeah! That's it. No more 45 year old builds
Thank you so much for your efforts. I very much appreciate it!

--Heiko

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/jenkins/model/RunIdMigrator.java
core/src/test/java/jenkins/model/RunIdMigratorTest.java
http://jenkins-ci.org/commit/jenkins/4a75e058bafd3cd98832a2a129c9f23c71d26f55
Log:
JENKINS-26519 Attempting to improve diagnostics in case rename fails.
(cherry picked from commit 0633beef0868c59a7e853731aee5f5c0745fa3fc)

Code changed in jenkins
User: Jesse Glick
Path:
core/src/main/java/jenkins/model/RunIdMigrator.java
http://jenkins-ci.org/commit/jenkins/22dc8add0e106d100a73c16cb628f1c5d1f94538
Log:
JENKINS-26519 More diagnostics in case a numeric rename target already exists.
(cherry picked from commit 75c8c31f2fdc98081a0446dd3ee03da7243304a1)

Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
core/src/main/java/jenkins/model/RunIdMigrator.java
core/src/test/java/jenkins/model/RunIdMigratorTest.java
http://jenkins-ci.org/commit/jenkins/7f82ab94a4f6247abaf9f65998360be5c3136bf8
Log:
[FIXED JENKINS-26519] Build record migration failed on Windows using Java 6.
In this environment, Util.createSymlink and .isSymlink are implemented, but resolveSymlink is not.
(cherry picked from commit 388c4b5e6ba52037cae117c5c69f9f4156d41401)

Conflicts:
changelog.html

Compare: https://github.com/jenkinsci/jenkins/compare/1debf06fe6d6...7f82ab94a4f6

Integrated in jenkins_main_trunk #3961
JENKINS-26519 Attempting to improve diagnostics in case rename fails. (Revision 4a75e058bafd3cd98832a2a129c9f23c71d26f55)
JENKINS-26519 More diagnostics in case a numeric rename target already exists. (Revision 22dc8add0e106d100a73c16cb628f1c5d1f94538)
[FIXED JENKINS-26519] Build record migration failed on Windows using Java 6. (Revision 7f82ab94a4f6247abaf9f65998360be5c3136bf8)

Result = SUCCESS
jesse glick : 4a75e058bafd3cd98832a2a129c9f23c71d26f55
Files :

• core/src/main/java/jenkins/model/RunIdMigrator.java
• core/src/test/java/jenkins/model/RunIdMigratorTest.java

jesse glick : 22dc8add0e106d100a73c16cb628f1c5d1f94538
Files :

• core/src/main/java/jenkins/model/RunIdMigrator.java

jesse glick : 7f82ab94a4f6247abaf9f65998360be5c3136bf8
Files :

• core/src/test/java/jenkins/model/RunIdMigratorTest.java
• core/src/main/java/jenkins/model/RunIdMigrator.java
• changelog.html

FYI: I ran into this same issue on CentOS 6.5. Old directories had symlinks like:
lrwxrwxrwx. 1 jenkins jenkins 19 Apr 22 09:06 401 -> 2015-04-22_09-06-23

New directories looked like:
drwxr-xr-x. 2 jenkins jenkins 4096 Mar 1 01:28 317

The UI showed the old builds as having run before I was born! In order to re-run the migration I did the following:
sudo find /var/lib/jenkins/ -name legacyIds -delete
sudo service jenkins restart

Once this was done things looked correct.