Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26519

Build records not migrated due to “failed to rename” on Windows

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Labels:
      None
    • Environment:
      1.597 on Windows
    • Similar Issues:

      Description

      After updating from 1.596 to 1.597 I seem to have wrong "last success" timestamps. See attached images.

        Attachments

        1. 2014-11-26_03-40-28 build.xml
          5 kB
        2. build.png
          build.png
          6 kB
        3. builds directory.jpg
          builds directory.jpg
          170 kB
        4. Error Log SH - 3953 .txt
          902 kB
        5. jenkins.err.log_on-startup.txt
          119 kB
        6. job.dir.txt
          0.8 kB
        7. job-builds.dir.txt
          0.8 kB
        8. lastSuccess.png
          lastSuccess.png
          6 kB
        9. tag-builds-dir.txt
          1 kB

          Issue Links

            Activity

            Hide
            danielbeck Daniel Beck added a comment -

            When you select a job, are the dates in the build history widget correct? Is the date correct when viewing a single build?

            Show
            danielbeck Daniel Beck added a comment - When you select a job, are the dates in the build history widget correct? Is the date correct when viewing a single build?
            Hide
            tdtappe Heiko Tappe added a comment -

            Old builds' dates are incorrect (01.01.1970), new builds (after having updated to 1.597) are correct. Same for viewing a single build.

            Show
            tdtappe Heiko Tappe added a comment - Old builds' dates are incorrect (01.01.1970), new builds (after having updated to 1.597) are correct. Same for viewing a single build.
            Hide
            danielbeck Daniel Beck added a comment -

            This seems pretty bad. Assigning to Jesse Glick speculatively.

            Show
            danielbeck Daniel Beck added a comment - This seems pretty bad. Assigning to Jesse Glick speculatively.
            Hide
            jglick Jesse Glick added a comment -

            Hmm, I definitely wrote (and tested) code to create <timestamp> in migrated build.xml records. Can you attach an example of the $JENKINS_HOME/jobs/*/builds/*/build.xml for one of the migrated builds showing a zero timestamp? Also if you happen to have your Jenkins log from the first startup after 1.597, attach that, in case there were any warnings. And attach the legacyIds file from the builds parent directory.

            Show
            jglick Jesse Glick added a comment - Hmm, I definitely wrote (and tested) code to create <timestamp> in migrated build.xml records. Can you attach an example of the $JENKINS_HOME/jobs/*/builds/*/build.xml for one of the migrated builds showing a zero timestamp? Also if you happen to have your Jenkins log from the first startup after 1.597, attach that, in case there were any warnings. And attach the legacyIds file from the builds parent directory.
            Hide
            danielbeck Daniel Beck added a comment -

            tdtappe: Did you configure a nondefault build record directory in the global configuration? I.e. are the builds for a job NOT in .../job/foo/builds ?

            Show
            danielbeck Daniel Beck added a comment - tdtappe: Did you configure a nondefault build record directory in the global configuration? I.e. are the builds for a job NOT in .../job/foo/builds ?
            Hide
            jglick Jesse Glick added a comment -

            Did you configure a nondefault build record directory in the global configuration?

            This should not matter, since the call to RunIdMigrator takes into account Job.getBuildDir, but of course this would be interesting information if so.

            (The reverse migration tool cannot handle nondefault build directories, unfortunately.)

            Show
            jglick Jesse Glick added a comment - Did you configure a nondefault build record directory in the global configuration? This should not matter, since the call to RunIdMigrator takes into account Job.getBuildDir , but of course this would be interesting information if so. (The reverse migration tool cannot handle nondefault build directories, unfortunately.)
            Hide
            tdtappe Heiko Tappe added a comment -

            No, I did not configure non-default build directories. All my projects seem to have their builds in the default build directory.

            Show
            tdtappe Heiko Tappe added a comment - No, I did not configure non-default build directories. All my projects seem to have their builds in the default build directory.
            Hide
            tdtappe Heiko Tappe added a comment -

            AFAICS all legacyIds files have size 0.

            Show
            tdtappe Heiko Tappe added a comment - AFAICS all legacyIds files have size 0.
            Hide
            tdtappe Heiko Tappe added a comment -

            Could it be a matter of whitespaces in the path?

            Show
            tdtappe Heiko Tappe added a comment - Could it be a matter of whitespaces in the path?
            Hide
            tdtappe Heiko Tappe added a comment -

            Sent a build.xml to Jesse privately.

            Show
            tdtappe Heiko Tappe added a comment - Sent a build.xml to Jesse privately.
            Hide
            tdtappe Heiko Tappe added a comment -

            This might be an important part of the jenkins log:

            INFO: Downloading jenkins.war
            ...
            ...
            INFO: Augmented all extensions
            ...
            ...
            WARNUNG: Build record migration (https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-24380+Migration) is one-way. If you need to downgrade Jenkins, run: java -classpath "D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\jenkins-core-1.597.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\commons-io-2.4.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\stapler-1.234.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\ant-1.8.4.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\commons-lang-2.6.jar" jenkins.model.RunIdMigrator "d:\NightlyBuilds\Jenkins"
            ...
            ...
            20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate
            INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 13.00-dev\builds
            20.01.2015 16:22:20 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: failed to rename 2013-09-04_01-36-36 to 275
            20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate
            INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.01.00-patched\builds
            20.01.2015 16:22:20 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: failed to rename 2015-01-16_01-03-35 to 175
            20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate
            INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 13.02-fibu\builds
            20.01.2015 16:22:20 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: failed to rename 2014-08-29_21-30-00 to 176
            20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate

            Show
            tdtappe Heiko Tappe added a comment - This might be an important part of the jenkins log: INFO: Downloading jenkins.war ... ... INFO: Augmented all extensions ... ... WARNUNG: Build record migration ( https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-24380+Migration ) is one-way. If you need to downgrade Jenkins, run: java -classpath "D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\jenkins-core-1.597.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\commons-io-2.4.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\stapler-1.234.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\ant-1.8.4.jar;D:\NightlyBuilds\Jenkins\war\WEB-INF\lib\commons-lang-2.6.jar" jenkins.model.RunIdMigrator "d:\NightlyBuilds\Jenkins" ... ... 20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 13.00-dev\builds 20.01.2015 16:22:20 jenkins.model.RunIdMigrator doMigrate WARNUNG: failed to rename 2013-09-04_01-36-36 to 275 20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.01.00-patched\builds 20.01.2015 16:22:20 jenkins.model.RunIdMigrator doMigrate WARNUNG: failed to rename 2015-01-16_01-03-35 to 175 20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 13.02-fibu\builds 20.01.2015 16:22:20 jenkins.model.RunIdMigrator doMigrate WARNUNG: failed to rename 2014-08-29_21-30-00 to 176 20.01.2015 16:22:20 jenkins.model.RunIdMigrator migrate
            Hide
            danielbeck Daniel Beck added a comment -

            If the rename is performed without first deleting the build number symlink, maybe this fails on Windows?

            Show
            danielbeck Daniel Beck added a comment - If the rename is performed without first deleting the build number symlink, maybe this fails on Windows?
            Hide
            jglick Jesse Glick added a comment -

            I glanced at one of the build.xml files and it was in the old format, which is clearly the problem. I do recall testing migration on Windows but there are a thousand and one things that can go wrong on a Windows filesystem and it looks like you hit one of them. Will see if I can find a cause.

            Show
            jglick Jesse Glick added a comment - I glanced at one of the build.xml files and it was in the old format, which is clearly the problem. I do recall testing migration on Windows but there are a thousand and one things that can go wrong on a Windows filesystem and it looks like you hit one of them. Will see if I can find a cause.
            Hide
            jglick Jesse Glick added a comment -

            The build number symlinks are deleted in one pass, then the build.xml modifications and directory renames happen in a second pass. There are no could not delete build number symlink warnings so it does not seem that this is the problem.

            Can you show me a long directory listing (IIRC dir suffices in a command shell) of one of the affected builds directories?

            Show
            jglick Jesse Glick added a comment - The build number symlinks are deleted in one pass, then the build.xml modifications and directory renames happen in a second pass. There are no could not delete build number symlink warnings so it does not seem that this is the problem. Can you show me a long directory listing (IIRC dir suffices in a command shell) of one of the affected builds directories?
            Hide
            jglick Jesse Glick added a comment -

            I suspect that the problem is that on some Windows systems, build number “symlinks” are really text files pointing to the target, and that these are not handled.

            Show
            jglick Jesse Glick added a comment - I suspect that the problem is that on some Windows systems, build number “symlinks” are really text files pointing to the target, and that these are not handled.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            changelog.html
            core/src/main/java/jenkins/model/RunIdMigrator.java
            core/src/test/java/jenkins/model/RunIdMigratorTest.java
            http://jenkins-ci.org/commit/jenkins/056b446480d9f24619c2c30ebdf9df6122a9b653
            Log:
            [FIXED JENKINS-26519] Accept Windows text file quasi-symlinks for build numbers.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/jenkins/model/RunIdMigrator.java core/src/test/java/jenkins/model/RunIdMigratorTest.java http://jenkins-ci.org/commit/jenkins/056b446480d9f24619c2c30ebdf9df6122a9b653 Log: [FIXED JENKINS-26519] Accept Windows text file quasi-symlinks for build numbers.
            Hide
            jglick Jesse Glick added a comment -

            In the interest of expediency I pushed what I think is a fix to the rc branch for 1.598 (which will I guess come out on Monday as usual), and am validating a merge to master. It would not be a bad idea to grab the jenkins.war from https://jenkins.ci.cloudbees.com/job/core/job/jenkins_rc_branch/317/ once it completes and try running it.

            In order to force a rerun of migration when using the patched version of Jenkins:

            1. Shut down Jenkins.
            2. Delete all %JENKINS_HOME%\jobs*\builds\legacyId files. (They are just caches and will be recreated as needed.)
            3. Start Jenkins again.
            4. Check the log file. You should see the Migrating build records messages again, but should not see the warnings.
            Show
            jglick Jesse Glick added a comment - In the interest of expediency I pushed what I think is a fix to the rc branch for 1.598 (which will I guess come out on Monday as usual), and am validating a merge to master . It would not be a bad idea to grab the jenkins.war from https://jenkins.ci.cloudbees.com/job/core/job/jenkins_rc_branch/317/ once it completes and try running it. In order to force a rerun of migration when using the patched version of Jenkins: Shut down Jenkins. Delete all %JENKINS_HOME%\jobs*\builds\legacyId files. (They are just caches and will be recreated as needed.) Start Jenkins again. Check the log file. You should see the Migrating build records messages again, but should not see the warnings.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/test/java/jenkins/model/RunIdMigratorTest.java
            http://jenkins-ci.org/commit/jenkins/c997ac0c865ca4866ffda0e9ae93fe818c311cba
            Log:
            JENKINS-26519 Confirming that it is safe to rerun migration by just deleting legacyId files.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/test/java/jenkins/model/RunIdMigratorTest.java http://jenkins-ci.org/commit/jenkins/c997ac0c865ca4866ffda0e9ae93fe818c311cba Log: JENKINS-26519 Confirming that it is safe to rerun migration by just deleting legacyId files.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/jenkins/model/RunIdMigrator.java
            http://jenkins-ci.org/commit/jenkins/fccc34987f03a2716f3744a998cfe7d9127abafc
            Log:
            JENKINS-26519 Unreproducible NPE from unmigrateJobsDir.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/RunIdMigrator.java http://jenkins-ci.org/commit/jenkins/fccc34987f03a2716f3744a998cfe7d9127abafc Log: JENKINS-26519 Unreproducible NPE from unmigrateJobsDir.
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3940
            JENKINS-26519 Confirming that it is safe to rerun migration by just deleting legacyId files. (Revision c997ac0c865ca4866ffda0e9ae93fe818c311cba)
            [FIXED JENKINS-26519] Accept Windows text file quasi-symlinks for build numbers. (Revision 056b446480d9f24619c2c30ebdf9df6122a9b653)

            Result = SUCCESS
            jesse glick : c997ac0c865ca4866ffda0e9ae93fe818c311cba
            Files :

            • core/src/test/java/jenkins/model/RunIdMigratorTest.java

            jesse glick : 056b446480d9f24619c2c30ebdf9df6122a9b653
            Files :

            • changelog.html
            • core/src/test/java/jenkins/model/RunIdMigratorTest.java
            • core/src/main/java/jenkins/model/RunIdMigrator.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3940 JENKINS-26519 Confirming that it is safe to rerun migration by just deleting legacyId files. (Revision c997ac0c865ca4866ffda0e9ae93fe818c311cba) [FIXED JENKINS-26519] Accept Windows text file quasi-symlinks for build numbers. (Revision 056b446480d9f24619c2c30ebdf9df6122a9b653) Result = SUCCESS jesse glick : c997ac0c865ca4866ffda0e9ae93fe818c311cba Files : core/src/test/java/jenkins/model/RunIdMigratorTest.java jesse glick : 056b446480d9f24619c2c30ebdf9df6122a9b653 Files : changelog.html core/src/test/java/jenkins/model/RunIdMigratorTest.java core/src/main/java/jenkins/model/RunIdMigrator.java
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3941
            JENKINS-26519 Unreproducible NPE from unmigrateJobsDir. (Revision fccc34987f03a2716f3744a998cfe7d9127abafc)

            Result = SUCCESS
            jesse glick : fccc34987f03a2716f3744a998cfe7d9127abafc
            Files :

            • core/src/main/java/jenkins/model/RunIdMigrator.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3941 JENKINS-26519 Unreproducible NPE from unmigrateJobsDir. (Revision fccc34987f03a2716f3744a998cfe7d9127abafc) Result = SUCCESS jesse glick : fccc34987f03a2716f3744a998cfe7d9127abafc Files : core/src/main/java/jenkins/model/RunIdMigrator.java
            Hide
            tdtappe Heiko Tappe added a comment -

            Switched to Jenkins ver. 1.598-SNAPSHOT (rc-01/21/2015 18:23 GMT-jenkins) after having deleted the legacyIds files. But still:

            ...
            ...
            22.01.2015 08:13:43 jenkins.model.RunIdMigrator migrate
            INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02-dev\builds
            22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: failed to rename 2014-08-30_00-58-41 to 79
            22.01.2015 08:13:43 jenkins.model.RunIdMigrator migrate
            INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 12.01.02-patched\builds
            22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: found no build.xml in 2012-12-07_00-13-31
            22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: failed to rename 2013-03-01_00-22-07 to 271
            22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: failed to rename 2015-01-20_01-45-40 to 82
            22.01.2015 08:13:43 jenkins.model.RunIdMigrator migrate
            ...
            ...

            Show
            tdtappe Heiko Tappe added a comment - Switched to Jenkins ver. 1.598-SNAPSHOT (rc-01/21/2015 18:23 GMT-jenkins) after having deleted the legacyIds files. But still: ... ... 22.01.2015 08:13:43 jenkins.model.RunIdMigrator migrate INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02-dev\builds 22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate WARNUNG: failed to rename 2014-08-30_00-58-41 to 79 22.01.2015 08:13:43 jenkins.model.RunIdMigrator migrate INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 12.01.02-patched\builds 22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate WARNUNG: found no build.xml in 2012-12-07_00-13-31 22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate WARNUNG: failed to rename 2013-03-01_00-22-07 to 271 22.01.2015 08:13:43 jenkins.model.RunIdMigrator doMigrate WARNUNG: failed to rename 2015-01-20_01-45-40 to 82 22.01.2015 08:13:43 jenkins.model.RunIdMigrator migrate ... ...
            Hide
            tdtappe Heiko Tappe added a comment -

            Listings of job and job/builds directories.

            Show
            tdtappe Heiko Tappe added a comment - Listings of job and job/builds directories.
            Hide
            martouf Steve Haworth added a comment - - edited

            Tried the new war (jenkins_main_trunk #3941) it didn't work for me. I have the same set-up Windows running under a service with many legacy builds.
            see err log file attached (jenkins.err.log_on-startup.txt)

            NOTE: What did work was adding
            <timestamp>1420730669680</timestamp>
            To the build.xml file. This resolved all my issues for this build.

            This value was missing the only value present was
            <startTime>1420730669680</startTime>

            Obviously doing this for all 10000+ builds will require a script, but that seems to resolve it.

            Show
            martouf Steve Haworth added a comment - - edited Tried the new war (jenkins_main_trunk #3941) it didn't work for me. I have the same set-up Windows running under a service with many legacy builds. see err log file attached (jenkins.err.log_on-startup.txt) NOTE: What did work was adding <timestamp>1420730669680</timestamp> To the build.xml file. This resolved all my issues for this build. This value was missing the only value present was <startTime>1420730669680</startTime> Obviously doing this for all 10000+ builds will require a script, but that seems to resolve it.
            Hide
            jglick Jesse Glick added a comment -

            Steve Haworth timestamp is never present in old builds. The migrator adds it. Your case looks to have at least two problems:

            1. failed to rename despite no prior error reported deleting the symlink. (The directory listing confirms that this is really a symlink, not a text file, so it is a distinct problem from the one I just fixed.) I would recommend stepping through RunIdMigrator in a debugger, starting with your original build directory (probably recoverable just be deleting all legacyId files).
            2. And then in a few cases could not find <number>, suggesting an unexpected or corrupt build.xml format.
            Show
            jglick Jesse Glick added a comment - Steve Haworth timestamp is never present in old builds. The migrator adds it. Your case looks to have at least two problems: failed to rename despite no prior error reported deleting the symlink. (The directory listing confirms that this is really a symlink, not a text file, so it is a distinct problem from the one I just fixed.) I would recommend stepping through RunIdMigrator in a debugger, starting with your original build directory (probably recoverable just be deleting all legacyId files). And then in a few cases could not find <number> , suggesting an unexpected or corrupt build.xml format.
            Hide
            jglick Jesse Glick added a comment -

            If you are willing to try up-to-the-minute dev builds, there are a few places where I think I could add better diagnostics.

            Show
            jglick Jesse Glick added a comment - If you are willing to try up-to-the-minute dev builds, there are a few places where I think I could add better diagnostics.
            Hide
            tdtappe Heiko Tappe added a comment -

            Sure!

            Show
            tdtappe Heiko Tappe added a comment - Sure!
            Hide
            martouf Steve Haworth added a comment -

            Jesse if the question was aimed at me "yes I am".

            Show
            martouf Steve Haworth added a comment - Jesse if the question was aimed at me "yes I am".
            Hide
            chorn Cameron Horn added a comment -

            I am also still experiencing this issue with 1.598.

            Show
            chorn Cameron Horn added a comment - I am also still experiencing this issue with 1.598.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/jenkins/model/RunIdMigrator.java
            core/src/test/java/jenkins/model/RunIdMigratorTest.java
            http://jenkins-ci.org/commit/jenkins/0633beef0868c59a7e853731aee5f5c0745fa3fc
            Log:
            JENKINS-26519 Attempting to improve diagnostics in case rename fails.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/RunIdMigrator.java core/src/test/java/jenkins/model/RunIdMigratorTest.java http://jenkins-ci.org/commit/jenkins/0633beef0868c59a7e853731aee5f5c0745fa3fc Log: JENKINS-26519 Attempting to improve diagnostics in case rename fails.
            Hide
            jglick Jesse Glick added a comment -

            To anyone still seeing issues, this build ought to have more informative error messages for the “failure to rename”.

            Steve Haworth your could not find <number> could only be diagnosed by seeing the named build.xml files.

            Show
            jglick Jesse Glick added a comment - To anyone still seeing issues, this build ought to have more informative error messages for the “failure to rename”. Steve Haworth your could not find <number> could only be diagnosed by seeing the named build.xml files.
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3953
            JENKINS-26519 Attempting to improve diagnostics in case rename fails. (Revision 0633beef0868c59a7e853731aee5f5c0745fa3fc)

            Result = SUCCESS
            jesse glick : 0633beef0868c59a7e853731aee5f5c0745fa3fc
            Files :

            • core/src/test/java/jenkins/model/RunIdMigratorTest.java
            • core/src/main/java/jenkins/model/RunIdMigrator.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3953 JENKINS-26519 Attempting to improve diagnostics in case rename fails. (Revision 0633beef0868c59a7e853731aee5f5c0745fa3fc) Result = SUCCESS jesse glick : 0633beef0868c59a7e853731aee5f5c0745fa3fc Files : core/src/test/java/jenkins/model/RunIdMigratorTest.java core/src/main/java/jenkins/model/RunIdMigrator.java
            Hide
            tdtappe Heiko Tappe added a comment -

            So here is part of the log with version #3953. Or what do you need? Do I have to enable some logging?

            28.01.2015 08:04:37 jenkins.model.RunIdMigrator migrate
            INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds
            28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-10-16_10-32-01
            java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\68 already exists
            at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285)
            at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262)
            at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
            at hudson.model.Job.onLoad(Job.java:194)
            at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
            at hudson.model.Project.onLoad(Project.java:98)
            at hudson.model.Items.load(Items.java:279)
            at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
            at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
            at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-05_16-13-26
            java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\69 already exists
            at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285)
            at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262)
            at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
            at hudson.model.Job.onLoad(Job.java:194)
            at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
            at hudson.model.Project.onLoad(Project.java:98)
            at hudson.model.Items.load(Items.java:279)
            at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
            at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
            at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-05_16-18-56
            java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\70 already exists
            at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285)
            at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262)
            at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
            at hudson.model.Job.onLoad(Job.java:194)
            at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
            at hudson.model.Project.onLoad(Project.java:98)
            at hudson.model.Items.load(Items.java:279)
            at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
            at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
            at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-19_17-16-21
            java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\71 already exists
            at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285)
            at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262)
            at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
            at hudson.model.Job.onLoad(Job.java:194)
            at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
            at hudson.model.Project.onLoad(Project.java:98)
            at hudson.model.Items.load(Items.java:279)
            at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
            at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
            at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate
            WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-19_17-20-07
            java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\72 already exists
            at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285)
            at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262)
            at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
            at hudson.model.Job.onLoad(Job.java:194)
            at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
            at hudson.model.Project.onLoad(Project.java:98)
            at hudson.model.Items.load(Items.java:279)
            at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
            at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
            at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            28.01.2015 08:04:37 jenkins.InitReactorRunner$1 onAttained
            INFO: Loaded all jobs

            Show
            tdtappe Heiko Tappe added a comment - So here is part of the log with version #3953. Or what do you need? Do I have to enable some logging? 28.01.2015 08:04:37 jenkins.model.RunIdMigrator migrate INFO: Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds 28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-10-16_10-32-01 java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\68 already exists at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285) at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262) at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166) at hudson.model.Job.onLoad(Job.java:194) at hudson.model.AbstractProject.onLoad(AbstractProject.java:318) at hudson.model.Project.onLoad(Project.java:98) at hudson.model.Items.load(Items.java:279) at jenkins.model.Jenkins$17.run(Jenkins.java:2673) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:903) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-05_16-13-26 java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\69 already exists at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285) at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262) at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166) at hudson.model.Job.onLoad(Job.java:194) at hudson.model.AbstractProject.onLoad(AbstractProject.java:318) at hudson.model.Project.onLoad(Project.java:98) at hudson.model.Items.load(Items.java:279) at jenkins.model.Jenkins$17.run(Jenkins.java:2673) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:903) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-05_16-18-56 java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\70 already exists at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285) at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262) at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166) at hudson.model.Job.onLoad(Job.java:194) at hudson.model.AbstractProject.onLoad(AbstractProject.java:318) at hudson.model.Project.onLoad(Project.java:98) at hudson.model.Items.load(Items.java:279) at jenkins.model.Jenkins$17.run(Jenkins.java:2673) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:903) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-19_17-16-21 java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\71 already exists at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285) at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262) at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166) at hudson.model.Job.onLoad(Job.java:194) at hudson.model.AbstractProject.onLoad(AbstractProject.java:318) at hudson.model.Project.onLoad(Project.java:98) at hudson.model.Items.load(Items.java:279) at jenkins.model.Jenkins$17.run(Jenkins.java:2673) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:903) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 28.01.2015 08:04:37 jenkins.model.RunIdMigrator doMigrate WARNUNG: failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-19_17-20-07 java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\72 already exists at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:285) at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:262) at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166) at hudson.model.Job.onLoad(Job.java:194) at hudson.model.AbstractProject.onLoad(AbstractProject.java:318) at hudson.model.Project.onLoad(Project.java:98) at hudson.model.Items.load(Items.java:279) at jenkins.model.Jenkins$17.run(Jenkins.java:2673) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:903) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 28.01.2015 08:04:37 jenkins.InitReactorRunner$1 onAttained INFO: Loaded all jobs
            Hide
            martouf Steve Haworth added a comment -

            Log attached - Error Log SH - 3953 .txt. Running with build #3953.

            Yes Understand the issue with "could not find <number>", a couple of disassociated builds with no build.xml, so I'll manually delete those.
            Main error persists though looks to be same as tdtappe's debug output

            Show
            martouf Steve Haworth added a comment - Log attached - Error Log SH - 3953 .txt. Running with build #3953. Yes Understand the issue with "could not find <number>", a couple of disassociated builds with no build.xml, so I'll manually delete those. Main error persists though looks to be same as tdtappe's debug output
            Hide
            martouf Steve Haworth added a comment -

            Attached a jpeg of the builds directory for a build (builds directory.jpg)
            NOTE: Build 1417 seems to have been migrated
            Builds 1418-1425 fail to migrate see log above.
            Builds 1426-1427 have been built with new format records

            Show
            martouf Steve Haworth added a comment - Attached a jpeg of the builds directory for a build (builds directory.jpg) NOTE: Build 1417 seems to have been migrated Builds 1418-1425 fail to migrate see log above. Builds 1426-1427 have been built with new format records
            Hide
            jglick Jesse Glick added a comment -

            Heiko Tappe so we have confirmation that the problem involves the target of the move already existing, yet the migrator apparently did not even attempt to delete it. Can you give me a directory listing of d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds please? (dir) And if for example 71 is a regular file, can you please attach it? I will attempt to strengthen the migrator here.

            Steve Haworth:

            Understand the issue with "could not find <number>", a couple of disassociated builds with no build.xml

            This warning ought to be printed only when build.xml does exist, but has unexpected content. Else you would see found no build.xml in … in this spot. So please attach the build.xml.

            Show
            jglick Jesse Glick added a comment - Heiko Tappe so we have confirmation that the problem involves the target of the move already existing, yet the migrator apparently did not even attempt to delete it. Can you give me a directory listing of d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds please? ( dir ) And if for example 71 is a regular file, can you please attach it? I will attempt to strengthen the migrator here. Steve Haworth : Understand the issue with "could not find <number>", a couple of disassociated builds with no build.xml This warning ought to be printed only when build.xml does exist, but has unexpected content. Else you would see found no build.xml in … in this spot. So please attach the build.xml .
            Hide
            jglick Jesse Glick added a comment -

            I am pushing another commit which tries harder to clean out existing numeric rename targets. Will post a download link when available. This is looking like some weirdness in how Windows handles symbolic links, which varies not only according to Windows version but also depending on filesystem and system configuration. Unfortunately Jenkins makes heavy use of the filesystem to store data, and lots of things behave unpredictably on Windows servers.

            You are encouraged to use Java 7 (or 8) rather than Java 6, as you will get better diagnostics: the file APIs available in Java 6- are poor.

            You should also go to /log/ (Log Manager) and create a custom log recording jenkins.model.RunIdMigrator at FINE. After the migrator runs at startup, logs are available in that URL. If they are truncated (only latest messages shown), you can install the Support Core plugin to capture fuller logs.

            Show
            jglick Jesse Glick added a comment - I am pushing another commit which tries harder to clean out existing numeric rename targets. Will post a download link when available. This is looking like some weirdness in how Windows handles symbolic links, which varies not only according to Windows version but also depending on filesystem and system configuration. Unfortunately Jenkins makes heavy use of the filesystem to store data, and lots of things behave unpredictably on Windows servers. You are encouraged to use Java 7 (or 8) rather than Java 6, as you will get better diagnostics: the file APIs available in Java 6- are poor. You should also go to /log/ ( Log Manager ) and create a custom log recording jenkins.model.RunIdMigrator at FINE . After the migrator runs at startup, logs are available in that URL. If they are truncated (only latest messages shown), you can install the Support Core plugin to capture fuller logs.
            Hide
            tdtappe Heiko Tappe added a comment -

            Directory listing of "KomalogWin - tag\builds"

            Show
            tdtappe Heiko Tappe added a comment - Directory listing of "KomalogWin - tag\builds"
            Hide
            jglick Jesse Glick added a comment -

            Are you running Java 6, or 7+?

            Show
            jglick Jesse Glick added a comment - Are you running Java 6, or 7+?
            Hide
            tdtappe Heiko Tappe added a comment -

            Java 6

            Show
            tdtappe Heiko Tappe added a comment - Java 6
            Hide
            jglick Jesse Glick added a comment -

            OK, guess I need to retest everything in Windows on Java 6 then.

            Show
            jglick Jesse Glick added a comment - OK, guess I need to retest everything in Windows on Java 6 then.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/jenkins/model/RunIdMigrator.java
            http://jenkins-ci.org/commit/jenkins/75c8c31f2fdc98081a0446dd3ee03da7243304a1
            Log:
            JENKINS-26519 More diagnostics in case a numeric rename target already exists.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/RunIdMigrator.java http://jenkins-ci.org/commit/jenkins/75c8c31f2fdc98081a0446dd3ee03da7243304a1 Log: JENKINS-26519 More diagnostics in case a numeric rename target already exists.
            Hide
            jglick Jesse Glick added a comment -

            Here is the latest round of fixes. In parallel I will see if I can reproduce the issue using Java 6.

            Show
            jglick Jesse Glick added a comment - Here is the latest round of fixes. In parallel I will see if I can reproduce the issue using Java 6.
            Hide
            tdtappe Heiko Tappe added a comment -

            Excerpt from custom logger with your newest build:

            Jan 28, 2015 4:12:36 PM INFO jenkins.model.RunIdMigrator migrate
            Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02.00-patched\builds
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping deletion of directory 116
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping deletion of directory 121
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping deletion of directory 122
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping deletion of directory 123
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping non-directory lastStableBuild
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping deletion of directory 4
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping new build dir 116
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping new build dir 121
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping new build dir 122
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping new build dir 123
            Jan 28, 2015 4:12:36 PM WARNUNG jenkins.model.RunIdMigrator doMigrate
            found no build.xml in 2014-08-29_03-51-39
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping new build dir 4
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping non-directory lastSuccessfulBuild
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping non-directory lastUnstableBuild
            Jan 28, 2015 4:12:36 PM INFO jenkins.model.RunIdMigrator migrate
            Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping deletion of directory 68
            Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator
            skipping deletion of directory 69
            Jan 28, 2015 4:12:36 PM INFO jenkins.model.RunIdMigrator migrate
            Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 12.02-dev\builds

            Show
            tdtappe Heiko Tappe added a comment - Excerpt from custom logger with your newest build: Jan 28, 2015 4:12:36 PM INFO jenkins.model.RunIdMigrator migrate Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02.00-patched\builds Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping deletion of directory 116 Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping deletion of directory 121 Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping deletion of directory 122 Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping deletion of directory 123 Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping non-directory lastStableBuild Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping deletion of directory 4 Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping new build dir 116 Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping new build dir 121 Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping new build dir 122 Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping new build dir 123 Jan 28, 2015 4:12:36 PM WARNUNG jenkins.model.RunIdMigrator doMigrate found no build.xml in 2014-08-29_03-51-39 Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping new build dir 4 Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping non-directory lastSuccessfulBuild Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping non-directory lastUnstableBuild Jan 28, 2015 4:12:36 PM INFO jenkins.model.RunIdMigrator migrate Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping deletion of directory 68 Jan 28, 2015 4:12:36 PM FEIN jenkins.model.RunIdMigrator skipping deletion of directory 69 Jan 28, 2015 4:12:36 PM INFO jenkins.model.RunIdMigrator migrate Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 12.02-dev\builds
            Hide
            jglick Jesse Glick added a comment -

            I am guessing that d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02.00-patched\builds\123 etc. are in fact SYMLINKD, not regular directories. The readSymlink call which would normally determine this cannot work on Java 6, and then apparently Windows claims that File.isDirectory based on the target. So I will see if I can reproduce and find a way to fix this case.

            Show
            jglick Jesse Glick added a comment - I am guessing that d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02.00-patched\builds\123 etc. are in fact SYMLINKD , not regular directories. The readSymlink call which would normally determine this cannot work on Java 6, and then apparently Windows claims that File.isDirectory based on the target. So I will see if I can reproduce and find a way to fix this case.
            Hide
            tdtappe Heiko Tappe added a comment -

            Actually not:

            Datentr„ger in Laufwerk D: ist DATEN
            Volumeseriennummer: 1225-FEC9

            Verzeichnis von D:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02.00-patched\builds

            Mi 28.01.2015 16:12 <DIR> .
            Mi 28.01.2015 16:12 <DIR> ..
            Di 20.01.2015 02:57 <DIR> 116
            Sa 24.01.2015 02:51 <DIR> 121
            Di 27.01.2015 02:51 <DIR> 122
            Mi 28.01.2015 02:55 <DIR> 123
            Fr 29.08.2014 03:25 <DIR> 2014-08-29_03-51-39
            Fr 29.08.2014 02:51 <SYMLINKD> 4 [2014-08-29_03-51-39]
            Fr 19.12.2014 02:57 2 lastFailedBuild
            Mi 28.01.2015 02:55 3 lastStableBuild
            Mi 28.01.2015 02:55 3 lastSuccessfulBuild
            Mi 27.08.2014 07:19 2 lastUnstableBuild
            Fr 19.12.2014 02:57 2 lastUnsuccessfulBuild
            Mi 28.01.2015 16:12 0 legacyIds
            6 Datei(en), 12 Bytes
            8 Verzeichnis(se), 769.188.323.328 Bytes frei

            Show
            tdtappe Heiko Tappe added a comment - Actually not: Datentr„ger in Laufwerk D: ist DATEN Volumeseriennummer: 1225-FEC9 Verzeichnis von D:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.02.00-patched\builds Mi 28.01.2015 16:12 <DIR> . Mi 28.01.2015 16:12 <DIR> .. Di 20.01.2015 02:57 <DIR> 116 Sa 24.01.2015 02:51 <DIR> 121 Di 27.01.2015 02:51 <DIR> 122 Mi 28.01.2015 02:55 <DIR> 123 Fr 29.08.2014 03:25 <DIR> 2014-08-29_03-51-39 Fr 29.08.2014 02:51 <SYMLINKD> 4 [2014-08-29_03-51-39] Fr 19.12.2014 02:57 2 lastFailedBuild Mi 28.01.2015 02:55 3 lastStableBuild Mi 28.01.2015 02:55 3 lastSuccessfulBuild Mi 27.08.2014 07:19 2 lastUnstableBuild Fr 19.12.2014 02:57 2 lastUnsuccessfulBuild Mi 28.01.2015 16:12 0 legacyIds 6 Datei(en), 12 Bytes 8 Verzeichnis(se), 769.188.323.328 Bytes frei
            Hide
            jglick Jesse Glick added a comment -

            The question is how you even got these build number symlinks to begin with—as far as I know they are not created when you run Jenkins on Windows using Java 6! (Even if the Windows system supports symlinks generally; IIRC XP does not, and newer systems may or may not.) The only way I can think to make them appear is to run 1.596- using Java 7, at least for a while, then run 1.597+ using Java 6.

            Show
            jglick Jesse Glick added a comment - The question is how you even got these build number symlinks to begin with—as far as I know they are not created when you run Jenkins on Windows using Java 6! (Even if the Windows system supports symlinks generally; IIRC XP does not, and newer systems may or may not.) The only way I can think to make them appear is to run 1.596- using Java 7, at least for a while, then run 1.597+ using Java 6.
            Hide
            jglick Jesse Glick added a comment -

            OK, the 116+ builds are either new, or got correctly migrated, but then we also got

            skipping deletion of directory 4

            where this one was an old build with a SYMLINKD that did not get migrated.

            Show
            jglick Jesse Glick added a comment - OK, the 116+ builds are either new, or got correctly migrated, but then we also got skipping deletion of directory 4 where this one was an old build with a SYMLINKD that did not get migrated.
            Hide
            tdtappe Heiko Tappe added a comment -

            I am almost 100% sure that there never has been another Java version than 6 (since 2010) on that machine.

            Show
            tdtappe Heiko Tappe added a comment - I am almost 100% sure that there never has been another Java version than 6 (since 2010) on that machine.
            Hide
            tdtappe Heiko Tappe added a comment -

            BTW I could most probably switch to Java 7 or even 8 on that machine if it might solve all my problems. But then we (you) wouldn't have a problematic real time scenario any longer to test with Java 6!?

            Show
            tdtappe Heiko Tappe added a comment - BTW I could most probably switch to Java 7 or even 8 on that machine if it might solve all my problems. But then we (you) wouldn't have a problematic real time scenario any longer to test with Java 6!?
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3957
            JENKINS-26519 More diagnostics in case a numeric rename target already exists. (Revision 75c8c31f2fdc98081a0446dd3ee03da7243304a1)

            Result = UNSTABLE
            jesse glick : 75c8c31f2fdc98081a0446dd3ee03da7243304a1
            Files :

            • core/src/main/java/jenkins/model/RunIdMigrator.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3957 JENKINS-26519 More diagnostics in case a numeric rename target already exists. (Revision 75c8c31f2fdc98081a0446dd3ee03da7243304a1) Result = UNSTABLE jesse glick : 75c8c31f2fdc98081a0446dd3ee03da7243304a1 Files : core/src/main/java/jenkins/model/RunIdMigrator.java
            Hide
            martouf Steve Haworth added a comment -

            Attached 2014-11-26_03-40-28 build.xml

            This is one of the files indicated in log "jenkins.err.log_on-startup.txt" as having no number.

            Show
            martouf Steve Haworth added a comment - Attached 2014-11-26_03-40-28 build.xml This is one of the files indicated in log "jenkins.err.log_on-startup.txt" as having no number.
            Hide
            jglick Jesse Glick added a comment -

            Heiko Tappe Well I will check once I get a post-XP test system up. My reading of the code in older versions of Jenkins is that it would never create such a symlink when running on Java 6, but I may have missed something.

            Steve Haworth Odd. The build is recorded as having been created by Jenkins 1.590, so before the switch, yet it has <timestamp> and no number, like the new format, but no <id>, like the old format. The migrator intentionally tries the folder rename before the build.xml rewrite, so it should not have been the result of an error midway through migration. So I have no explanation for how you got there and hope this is rare enough.

            Show
            jglick Jesse Glick added a comment - Heiko Tappe Well I will check once I get a post-XP test system up. My reading of the code in older versions of Jenkins is that it would never create such a symlink when running on Java 6, but I may have missed something. Steve Haworth Odd. The build is recorded as having been created by Jenkins 1.590, so before the switch, yet it has <timestamp> and no number , like the new format, but no <id> , like the old format. The migrator intentionally tries the folder rename before the build.xml rewrite, so it should not have been the result of an error midway through migration. So I have no explanation for how you got there and hope this is rare enough.
            Hide
            danielbeck Daniel Beck added a comment -

            Jesse: Could the unexpected XML come from running the unmigration script?

            Show
            danielbeck Daniel Beck added a comment - Jesse: Could the unexpected XML come from running the unmigration script?
            Hide
            tdtappe Heiko Tappe added a comment -

            Another excerpt with newest build:

            Jan 29, 2015 8:50:35 AM WARNUNG jenkins.model.RunIdMigrator doMigrate

            failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-10-16_10-32-01
            java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\68 already exists
            at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297)
            at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274)
            at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
            at hudson.model.Job.onLoad(Job.java:194)
            at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
            at hudson.model.Project.onLoad(Project.java:98)
            at hudson.model.Items.load(Items.java:281)
            at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
            at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
            at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

            Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

            skipping deletion of directory 5995

            Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

            skipping deletion of directory 5996

            Jan 29, 2015 8:50:35 AM WARNUNG jenkins.model.RunIdMigrator doMigrate

            failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-05_16-13-26
            java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\69 already exists
            at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297)
            at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274)
            at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
            at hudson.model.Job.onLoad(Job.java:194)
            at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
            at hudson.model.Project.onLoad(Project.java:98)
            at hudson.model.Items.load(Items.java:281)
            at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
            at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
            at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

            Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

            skipping deletion of directory 5997

            Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

            skipping deletion of directory 5998

            Jan 29, 2015 8:50:35 AM INFO jenkins.model.RunIdMigrator migrate

            Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.01-dev\builds

            Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

            skipping deletion of directory 124

            Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate

            skipping deletion of directory 128

            Show
            tdtappe Heiko Tappe added a comment - Another excerpt with newest build: Jan 29, 2015 8:50:35 AM WARNUNG jenkins.model.RunIdMigrator doMigrate failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-10-16_10-32-01 java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\68 already exists at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297) at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274) at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166) at hudson.model.Job.onLoad(Job.java:194) at hudson.model.AbstractProject.onLoad(AbstractProject.java:318) at hudson.model.Project.onLoad(Project.java:98) at hudson.model.Items.load(Items.java:281) at jenkins.model.Jenkins$17.run(Jenkins.java:2673) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:903) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate skipping deletion of directory 5995 Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate skipping deletion of directory 5996 Jan 29, 2015 8:50:35 AM WARNUNG jenkins.model.RunIdMigrator doMigrate failed to process d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\2014-11-05_16-13-26 java.io.IOException: d:\NightlyBuilds\Jenkins\jobs\KomalogWin - tag\builds\69 already exists at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297) at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274) at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166) at hudson.model.Job.onLoad(Job.java:194) at hudson.model.AbstractProject.onLoad(AbstractProject.java:318) at hudson.model.Project.onLoad(Project.java:98) at hudson.model.Items.load(Items.java:281) at jenkins.model.Jenkins$17.run(Jenkins.java:2673) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:903) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate skipping deletion of directory 5997 Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate skipping deletion of directory 5998 Jan 29, 2015 8:50:35 AM INFO jenkins.model.RunIdMigrator migrate Migrating build records in d:\NightlyBuilds\Jenkins\jobs\KomalogWin - 14.01-dev\builds Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate skipping deletion of directory 124 Jan 29, 2015 8:50:35 AM FEIN jenkins.model.RunIdMigrator doMigrate skipping deletion of directory 128
            Hide
            martouf Steve Haworth added a comment - - edited

            Build #3957

            Log extract...

            29-Jan-2015 10:40:35 jenkins.model.RunIdMigrator doMigrate
            WARNING: failed to process C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\2014-11-10_17-23-06
            java.io.IOException: C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\273 already exists
            at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297)
            at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274)
            at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
            at hudson.model.Job.onLoad(Job.java:194)
            at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
            at hudson.model.Project.onLoad(Project.java:98)
            at hudson.model.Items.load(Items.java:281)
            at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
            at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
            at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            29-Jan-2015 10:40:35 jenkins.model.RunIdMigrator doMigrate
            WARNING: failed to process C:\Jenkins\jobs\GEMTEST_Page_84\builds\2014-12-23_14-42-31
            java.io.IOException: C:\Jenkins\jobs\GEMTEST_Page_84\builds\1420 already exists
            at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297)
            at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274)
            at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
            at hudson.model.Job.onLoad(Job.java:194)
            at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
            at hudson.model.Project.onLoad(Project.java:98)
            at hudson.model.Items.load(Items.java:281)
            at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
            at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
            at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)
            29-Jan-2015 10:40:35 jenkins.model.RunIdMigrator doMigrate
            WARNING: failed to process C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\2014-11-26_13-03-35
            java.io.IOException: C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\288 already exists
            at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297)
            at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274)
            at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166)
            at hudson.model.Job.onLoad(Job.java:194)
            at hudson.model.AbstractProject.onLoad(AbstractProject.java:318)
            at hudson.model.Project.onLoad(Project.java:98)
            at hudson.model.Items.load(Items.java:281)
            at jenkins.model.Jenkins$17.run(Jenkins.java:2673)
            at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
            at jenkins.model.Jenkins$7.runTask(Jenkins.java:903)
            at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
            at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
            at java.lang.Thread.run(Unknown Source)

            Jenkins is running on a Virtual Machine with - Windows Server 2008 R2 Standard with Service Pack 1
            Java version is - Java Standard Edition Version 8 Update 31

            NOTE: Tried experiment... if I manually delete the shortcut directories in a job directory then migration is okay

            Show
            martouf Steve Haworth added a comment - - edited Build #3957 Log extract... 29-Jan-2015 10:40:35 jenkins.model.RunIdMigrator doMigrate WARNING: failed to process C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\2014-11-10_17-23-06 java.io.IOException: C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\273 already exists at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297) at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274) at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166) at hudson.model.Job.onLoad(Job.java:194) at hudson.model.AbstractProject.onLoad(AbstractProject.java:318) at hudson.model.Project.onLoad(Project.java:98) at hudson.model.Items.load(Items.java:281) at jenkins.model.Jenkins$17.run(Jenkins.java:2673) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:903) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 29-Jan-2015 10:40:35 jenkins.model.RunIdMigrator doMigrate WARNING: failed to process C:\Jenkins\jobs\GEMTEST_Page_84\builds\2014-12-23_14-42-31 java.io.IOException: C:\Jenkins\jobs\GEMTEST_Page_84\builds\1420 already exists at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297) at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274) at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166) at hudson.model.Job.onLoad(Job.java:194) at hudson.model.AbstractProject.onLoad(AbstractProject.java:318) at hudson.model.Project.onLoad(Project.java:98) at hudson.model.Items.load(Items.java:281) at jenkins.model.Jenkins$17.run(Jenkins.java:2673) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:903) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 29-Jan-2015 10:40:35 jenkins.model.RunIdMigrator doMigrate WARNING: failed to process C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\2014-11-26_13-03-35 java.io.IOException: C:\Jenkins\jobs\STX_USM_XFF-CAMARO_Package\builds\288 already exists at jenkins.model.RunIdMigrator.move(RunIdMigrator.java:297) at jenkins.model.RunIdMigrator.doMigrate(RunIdMigrator.java:274) at jenkins.model.RunIdMigrator.migrate(RunIdMigrator.java:166) at hudson.model.Job.onLoad(Job.java:194) at hudson.model.AbstractProject.onLoad(AbstractProject.java:318) at hudson.model.Project.onLoad(Project.java:98) at hudson.model.Items.load(Items.java:281) at jenkins.model.Jenkins$17.run(Jenkins.java:2673) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:903) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Jenkins is running on a Virtual Machine with - Windows Server 2008 R2 Standard with Service Pack 1 Java version is - Java Standard Edition Version 8 Update 31 NOTE: Tried experiment... if I manually delete the shortcut directories in a job directory then migration is okay
            Hide
            jglick Jesse Glick added a comment -

            Yes one workaround would be delete all numeric SYMLINKD, though I am not sure if there is an easy batch command to do that. Still intend to try to reproduce this and offer a proper fix.

            Show
            jglick Jesse Glick added a comment - Yes one workaround would be delete all numeric SYMLINKD , though I am not sure if there is an easy batch command to do that. Still intend to try to reproduce this and offer a proper fix.
            Hide
            jglick Jesse Glick added a comment -

            Could not reproduce by starting Jenkins 1.596 on Windows 2012 as Administrator under JDK 7u21, creating a project, running a few builds, then stopping and starting a trunk build: builds got correctly migrated. So perhaps only arises under specific circumstances.

            Show
            jglick Jesse Glick added a comment - Could not reproduce by starting Jenkins 1.596 on Windows 2012 as Administrator under JDK 7u21, creating a project, running a few builds, then stopping and starting a trunk build: builds got correctly migrated. So perhaps only arises under specific circumstances.
            Hide
            jglick Jesse Glick added a comment - - edited

            Also I checked in jrunscript under these circumstances that the code called by Util.resolveSymlink would work, as would the call to Util.deleteFile.

            In the case of Heiko Tappe, running Java 6, resolveSymlink would be expected to fail, explaining the skipping deletion of directory … message, followed by the failure of the migrator. (Which as I mentioned before would not be expected to matter for most people because symlinks should not have been created to begin with; need to confirm this.) This I could fix by just trying deleteFile anyway—if it is really a symlink to a directory, that will work, but if it is really a directory, it should fail because the directory is not empty. It makes me a little nervous because isDirectory would also be true for a real directory created by 1.597+ (or a partly successful earlier migration), and relying on File.delete to fail in this case seems dangerous.

            In the case of Steve Haworth, running Java 8, I am puzzled. I see no FINE-level messages here so I assume you did not turn on the custom logger, which might have some crucial information.

            Show
            jglick Jesse Glick added a comment - - edited Also I checked in jrunscript under these circumstances that the code called by Util.resolveSymlink would work, as would the call to Util.deleteFile . In the case of Heiko Tappe , running Java 6, resolveSymlink would be expected to fail, explaining the skipping deletion of directory … message, followed by the failure of the migrator. (Which as I mentioned before would not be expected to matter for most people because symlinks should not have been created to begin with; need to confirm this.) This I could fix by just trying deleteFile anyway—if it is really a symlink to a directory, that will work, but if it is really a directory, it should fail because the directory is not empty. It makes me a little nervous because isDirectory would also be true for a real directory created by 1.597+ (or a partly successful earlier migration), and relying on File.delete to fail in this case seems dangerous. In the case of Steve Haworth , running Java 8, I am puzzled. I see no FINE -level messages here so I assume you did not turn on the custom logger, which might have some crucial information.
            Hide
            jglick Jesse Glick added a comment -

            Ah, I was wrong, 1.596- on Windows Java 6 did attempt to create symlinks, using JNA, and this does produce the described error. So that is something I can and will fix.

            Show
            jglick Jesse Glick added a comment - Ah, I was wrong, 1.596- on Windows Java 6 did attempt to create symlinks, using JNA, and this does produce the described error. So that is something I can and will fix.
            Hide
            jglick Jesse Glick added a comment -

            Confusingly, for Windows on Java 6, 1, 2, etc. were created as true symlinks; the text files with build numbers were only used for permalinks like lastStableBuild, meaning my attempted fix in 056b446 was pointless and can probably just be reverted. (One of the unusual cases where “reopening” a bug is really correct.)

            Show
            jglick Jesse Glick added a comment - Confusingly, for Windows on Java 6, 1 , 2 , etc. were created as true symlinks; the text files with build numbers were only used for permalinks like lastStableBuild , meaning my attempted fix in 056b446 was pointless and can probably just be reverted. (One of the unusual cases where “reopening” a bug is really correct.)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            changelog.html
            core/src/main/java/jenkins/model/RunIdMigrator.java
            core/src/test/java/jenkins/model/RunIdMigratorTest.java
            http://jenkins-ci.org/commit/jenkins/388c4b5e6ba52037cae117c5c69f9f4156d41401
            Log:
            [FIXED JENKINS-26519] Build record migration failed on Windows using Java 6.
            In this environment, Util.createSymlink and .isSymlink are implemented, but resolveSymlink is not.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/jenkins/model/RunIdMigrator.java core/src/test/java/jenkins/model/RunIdMigratorTest.java http://jenkins-ci.org/commit/jenkins/388c4b5e6ba52037cae117c5c69f9f4156d41401 Log: [FIXED JENKINS-26519] Build record migration failed on Windows using Java 6. In this environment, Util.createSymlink and .isSymlink are implemented, but resolveSymlink is not.
            Hide
            jglick Jesse Glick added a comment -

            Heiko Tappe I think your problem is now fixed in this build, toward 1.600. If you confirm I will try to backport it to the 1.599 release candidate branch so it gets out earlier.

            Steve Haworth My latest fix might help in your case, but since you say you are running Java 8, you should not have been hitting it to begin with. Your symptoms are similar; the root cause may or may not be related. I would suggest trying once again with my newest build, assuming you still have the original build directories to reproduce against. First be sure to enable fine logging on this component.

            Show
            jglick Jesse Glick added a comment - Heiko Tappe I think your problem is now fixed in this build , toward 1.600. If you confirm I will try to backport it to the 1.599 release candidate branch so it gets out earlier. Steve Haworth My latest fix might help in your case, but since you say you are running Java 8, you should not have been hitting it to begin with. Your symptoms are similar; the root cause may or may not be related. I would suggest trying once again with my newest build, assuming you still have the original build directories to reproduce against. First be sure to enable fine logging on this component.
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3958
            [FIXED JENKINS-26519] Build record migration failed on Windows using Java 6. (Revision 388c4b5e6ba52037cae117c5c69f9f4156d41401)

            Result = SUCCESS
            jesse glick : 388c4b5e6ba52037cae117c5c69f9f4156d41401
            Files :

            • changelog.html
            • core/src/test/java/jenkins/model/RunIdMigratorTest.java
            • core/src/main/java/jenkins/model/RunIdMigrator.java
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3958 [FIXED JENKINS-26519] Build record migration failed on Windows using Java 6. (Revision 388c4b5e6ba52037cae117c5c69f9f4156d41401) Result = SUCCESS jesse glick : 388c4b5e6ba52037cae117c5c69f9f4156d41401 Files : changelog.html core/src/test/java/jenkins/model/RunIdMigratorTest.java core/src/main/java/jenkins/model/RunIdMigrator.java
            Hide
            tdtappe Heiko Tappe added a comment -

            Yeah! That's it. No more 45 year old builds
            Thank you so much for your efforts. I very much appreciate it!

            --Heiko

            Show
            tdtappe Heiko Tappe added a comment - Yeah! That's it. No more 45 year old builds Thank you so much for your efforts. I very much appreciate it! --Heiko
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/jenkins/model/RunIdMigrator.java
            core/src/test/java/jenkins/model/RunIdMigratorTest.java
            http://jenkins-ci.org/commit/jenkins/4a75e058bafd3cd98832a2a129c9f23c71d26f55
            Log:
            JENKINS-26519 Attempting to improve diagnostics in case rename fails.
            (cherry picked from commit 0633beef0868c59a7e853731aee5f5c0745fa3fc)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/RunIdMigrator.java core/src/test/java/jenkins/model/RunIdMigratorTest.java http://jenkins-ci.org/commit/jenkins/4a75e058bafd3cd98832a2a129c9f23c71d26f55 Log: JENKINS-26519 Attempting to improve diagnostics in case rename fails. (cherry picked from commit 0633beef0868c59a7e853731aee5f5c0745fa3fc)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            core/src/main/java/jenkins/model/RunIdMigrator.java
            http://jenkins-ci.org/commit/jenkins/22dc8add0e106d100a73c16cb628f1c5d1f94538
            Log:
            JENKINS-26519 More diagnostics in case a numeric rename target already exists.
            (cherry picked from commit 75c8c31f2fdc98081a0446dd3ee03da7243304a1)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/RunIdMigrator.java http://jenkins-ci.org/commit/jenkins/22dc8add0e106d100a73c16cb628f1c5d1f94538 Log: JENKINS-26519 More diagnostics in case a numeric rename target already exists. (cherry picked from commit 75c8c31f2fdc98081a0446dd3ee03da7243304a1)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            changelog.html
            core/src/main/java/jenkins/model/RunIdMigrator.java
            core/src/test/java/jenkins/model/RunIdMigratorTest.java
            http://jenkins-ci.org/commit/jenkins/7f82ab94a4f6247abaf9f65998360be5c3136bf8
            Log:
            [FIXED JENKINS-26519] Build record migration failed on Windows using Java 6.
            In this environment, Util.createSymlink and .isSymlink are implemented, but resolveSymlink is not.
            (cherry picked from commit 388c4b5e6ba52037cae117c5c69f9f4156d41401)

            Conflicts:
            changelog.html

            Compare: https://github.com/jenkinsci/jenkins/compare/1debf06fe6d6...7f82ab94a4f6

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/jenkins/model/RunIdMigrator.java core/src/test/java/jenkins/model/RunIdMigratorTest.java http://jenkins-ci.org/commit/jenkins/7f82ab94a4f6247abaf9f65998360be5c3136bf8 Log: [FIXED JENKINS-26519] Build record migration failed on Windows using Java 6. In this environment, Util.createSymlink and .isSymlink are implemented, but resolveSymlink is not. (cherry picked from commit 388c4b5e6ba52037cae117c5c69f9f4156d41401) Conflicts: changelog.html Compare: https://github.com/jenkinsci/jenkins/compare/1debf06fe6d6...7f82ab94a4f6
            Hide
            dogfood dogfood added a comment -

            Integrated in jenkins_main_trunk #3961
            JENKINS-26519 Attempting to improve diagnostics in case rename fails. (Revision 4a75e058bafd3cd98832a2a129c9f23c71d26f55)
            JENKINS-26519 More diagnostics in case a numeric rename target already exists. (Revision 22dc8add0e106d100a73c16cb628f1c5d1f94538)
            [FIXED JENKINS-26519] Build record migration failed on Windows using Java 6. (Revision 7f82ab94a4f6247abaf9f65998360be5c3136bf8)

            Result = SUCCESS
            jesse glick : 4a75e058bafd3cd98832a2a129c9f23c71d26f55
            Files :

            • core/src/main/java/jenkins/model/RunIdMigrator.java
            • core/src/test/java/jenkins/model/RunIdMigratorTest.java

            jesse glick : 22dc8add0e106d100a73c16cb628f1c5d1f94538
            Files :

            • core/src/main/java/jenkins/model/RunIdMigrator.java

            jesse glick : 7f82ab94a4f6247abaf9f65998360be5c3136bf8
            Files :

            • core/src/test/java/jenkins/model/RunIdMigratorTest.java
            • core/src/main/java/jenkins/model/RunIdMigrator.java
            • changelog.html
            Show
            dogfood dogfood added a comment - Integrated in jenkins_main_trunk #3961 JENKINS-26519 Attempting to improve diagnostics in case rename fails. (Revision 4a75e058bafd3cd98832a2a129c9f23c71d26f55) JENKINS-26519 More diagnostics in case a numeric rename target already exists. (Revision 22dc8add0e106d100a73c16cb628f1c5d1f94538) [FIXED JENKINS-26519] Build record migration failed on Windows using Java 6. (Revision 7f82ab94a4f6247abaf9f65998360be5c3136bf8) Result = SUCCESS jesse glick : 4a75e058bafd3cd98832a2a129c9f23c71d26f55 Files : core/src/main/java/jenkins/model/RunIdMigrator.java core/src/test/java/jenkins/model/RunIdMigratorTest.java jesse glick : 22dc8add0e106d100a73c16cb628f1c5d1f94538 Files : core/src/main/java/jenkins/model/RunIdMigrator.java jesse glick : 7f82ab94a4f6247abaf9f65998360be5c3136bf8 Files : core/src/test/java/jenkins/model/RunIdMigratorTest.java core/src/main/java/jenkins/model/RunIdMigrator.java changelog.html
            Hide
            forestjay Forest Handford added a comment -

            FYI: I ran into this same issue on CentOS 6.5. Old directories had symlinks like:
            lrwxrwxrwx. 1 jenkins jenkins 19 Apr 22 09:06 401 -> 2015-04-22_09-06-23

            New directories looked like:
            drwxr-xr-x. 2 jenkins jenkins 4096 Mar 1 01:28 317

            The UI showed the old builds as having run before I was born! In order to re-run the migration I did the following:
            sudo find /var/lib/jenkins/ -name legacyIds -delete
            sudo service jenkins restart

            Once this was done things looked correct.

            Show
            forestjay Forest Handford added a comment - FYI: I ran into this same issue on CentOS 6.5. Old directories had symlinks like: lrwxrwxrwx. 1 jenkins jenkins 19 Apr 22 09:06 401 -> 2015-04-22_09-06-23 New directories looked like: drwxr-xr-x. 2 jenkins jenkins 4096 Mar 1 01:28 317 The UI showed the old builds as having run before I was born! In order to re-run the migration I did the following: sudo find /var/lib/jenkins/ -name legacyIds -delete sudo service jenkins restart Once this was done things looked correct.

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              tdtappe Heiko Tappe
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: