Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29011

Jenkins 1.609.1 reverse migration script doesn't work when using a custom log folder

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
      None
    • Similar Issues:

      Description

      We recently upgraded our Jenkins LTS instance from v1.532.3 to the latest 1.609.1 version, after which we discovered some critical defects that required us to roll back to the prior version, 1.596.3. In so doing we encountered issues loading build logs and such due to the incompatible configuration changes applied by 1.609.1. We proceeded to run the downgrade script as described here to no avail. The script appeared to have completed without error but it didn't actually apply any changes to the configuration.

      Further ad-hoc testing reveals that one contributing factor is that we have set a custom path for the Jenkins logs on our master configuration (ie: ${JENKINS_HOME}\logs\builds\${ITEM_FULL_NAME}) and this seems to confuse the reverse migration script. To test my theory I moved a few of our build log folders out of our custom folder and into their default location under ${JENKINS_HOME}\jobs\ ${ITEM_FULL_NAME}\builds and then re-ran the downgrade script. All of the logs I had relocated were correctly downgraded to the old configuration, however none of the logs that were left under the custom log folder were touched.

      Needless to say moving the build logs back to their default locations is tedious and error prone at best, and the effort is exacerbated by larger build farms like ours where we have well over 1000 jobs which would need to be migrated.

      This, once again, puts us between a rock and a hard place. We are unable to continue using v1.609.1 due to failures to preserve backwards compatibility and if we downgrade to the previous version we are going to loose all of our build history, which is substantial - and of critical importance to our company - or we must incur significant expense to correct the issue ourselves.

        Attachments

          Issue Links

            Activity

            leedega Kevin Phillips created issue -
            leedega Kevin Phillips made changes -
            Field Original Value New Value
            Link This issue is related to JENKINS-24380 [ JENKINS-24380 ]
            leedega Kevin Phillips made changes -
            Summary Jenkins 1.609.1 reverse migration script doesn't always work after an upgrade Jenkins 1.609.1 reverse migration script doesn't work when using a custom log folder
            leedega Kevin Phillips made changes -
            Description We recently upgraded our Jenkins LTS instance from v1.532.3 to the latest 1.609.1 version, after which we discovered some critical defects that required us to roll back to the prior version, 1.596.3. In so doing we encountered issues loading build logs and such due to the incompatible configuration changes applied by 1.609.1. We proceeded to run the downgrade script as described [here|https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-24380+Migration] to no avail. The script appeared to have completed without error but it didn't actually apply any changes to the configuration.

            Further ad-hoc testing seems to indicate that the underlying problem has to do with the way newer versions of Jenkins store their build logs. For example, from what I can tell, if you install 1.609.1 on a clean system it will place build logs under the ./jobs subfolder along with the job they are associated with. However in older Jenkins versions the build logs were stored under the ./logs folder, disconnected from the job configurations. The interesting bit is that when you update from the legacy versions to the latest version the build logs don't appear to get moved to the new location under ./jobs. The master appears to have some way of indicating to itself that legacy build logs existed during the upgrade and that it should continue to maintain the logs in that location instead of the new location. Everything, that is, except for this reverse migration script.

            To check my theory I moved a few of our build log folders out of the ./logs folder and into their more modern location under ./jobs and then re-ran the downgrade script. All of the logs I had relocated were correctly downgraded to the old configuration, however none of the logs that were left under the old logs folder were touched.

            Needless to say moving the build logs to the new location is tedious and error prone at best, and the effort is exacerbated by larger build farms like ours where we have well over 1000 jobs which would need to be migrated.

            This, once again, puts us between a rock and a hard place. We are unable to continue using v1.609.1 due to failures to preserve backwards compatibility and if we downgrade to the previous version we are going to loose all of our build history, which is substantial - and of critical importance to our company - or we must incur significant expense to correct the issue ourselves.
            We recently upgraded our Jenkins LTS instance from v1.532.3 to the latest 1.609.1 version, after which we discovered some critical defects that required us to roll back to the prior version, 1.596.3. In so doing we encountered issues loading build logs and such due to the incompatible configuration changes applied by 1.609.1. We proceeded to run the downgrade script as described [here|https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-24380+Migration] to no avail. The script appeared to have completed without error but it didn't actually apply any changes to the configuration.

            Further ad-hoc testing reveals that one contributing factor is that we have set a custom path for the Jenkins logs on our master configuration (ie: ${JENKINS_HOME}\logs\builds\${ITEM_FULL_NAME}) and this seems to confuse the reverse migration script. To test my theory I moved a few of our build log folders out of our custom folder and into their default location under ${JENKINS_HOME}\jobs\ ${ITEM_FULL_NAME}\builds and then re-ran the downgrade script. All of the logs I had relocated were correctly downgraded to the old configuration, however none of the logs that were left under the custom log folder were touched.

            Needless to say moving the build logs back to their default locations is tedious and error prone at best, and the effort is exacerbated by larger build farms like ours where we have well over 1000 jobs which would need to be migrated.

            This, once again, puts us between a rock and a hard place. We are unable to continue using v1.609.1 due to failures to preserve backwards compatibility and if we downgrade to the previous version we are going to loose all of our build history, which is substantial - and of critical importance to our company - or we must incur significant expense to correct the issue ourselves.
            leedega Kevin Phillips made changes -
            Description We recently upgraded our Jenkins LTS instance from v1.532.3 to the latest 1.609.1 version, after which we discovered some critical defects that required us to roll back to the prior version, 1.596.3. In so doing we encountered issues loading build logs and such due to the incompatible configuration changes applied by 1.609.1. We proceeded to run the downgrade script as described [here|https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-24380+Migration] to no avail. The script appeared to have completed without error but it didn't actually apply any changes to the configuration.

            Further ad-hoc testing reveals that one contributing factor is that we have set a custom path for the Jenkins logs on our master configuration (ie: ${JENKINS_HOME}\logs\builds\${ITEM_FULL_NAME}) and this seems to confuse the reverse migration script. To test my theory I moved a few of our build log folders out of our custom folder and into their default location under ${JENKINS_HOME}\jobs\ ${ITEM_FULL_NAME}\builds and then re-ran the downgrade script. All of the logs I had relocated were correctly downgraded to the old configuration, however none of the logs that were left under the custom log folder were touched.

            Needless to say moving the build logs back to their default locations is tedious and error prone at best, and the effort is exacerbated by larger build farms like ours where we have well over 1000 jobs which would need to be migrated.

            This, once again, puts us between a rock and a hard place. We are unable to continue using v1.609.1 due to failures to preserve backwards compatibility and if we downgrade to the previous version we are going to loose all of our build history, which is substantial - and of critical importance to our company - or we must incur significant expense to correct the issue ourselves.
            We recently upgraded our Jenkins LTS instance from v1.532.3 to the latest 1.609.1 version, after which we discovered some critical defects that required us to roll back to the prior version, 1.596.3. In so doing we encountered issues loading build logs and such due to the incompatible configuration changes applied by 1.609.1. We proceeded to run the downgrade script as described [here|https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-24380+Migration] to no avail. The script appeared to have completed without error but it didn't actually apply any changes to the configuration.

            Further ad-hoc testing reveals that one contributing factor is that we have set a custom path for the Jenkins logs on our master configuration (ie: $\{JENKINS_HOME}\logs\builds\$\{ITEM_FULL_NAME}) and this seems to confuse the reverse migration script. To test my theory I moved a few of our build log folders out of our custom folder and into their default location under $\{JENKINS_HOME}\jobs\ $\{ITEM_FULL_NAME}\builds and then re-ran the downgrade script. All of the logs I had relocated were correctly downgraded to the old configuration, however none of the logs that were left under the custom log folder were touched.

            Needless to say moving the build logs back to their default locations is tedious and error prone at best, and the effort is exacerbated by larger build farms like ours where we have well over 1000 jobs which would need to be migrated.

            This, once again, puts us between a rock and a hard place. We are unable to continue using v1.609.1 due to failures to preserve backwards compatibility and if we downgrade to the previous version we are going to loose all of our build history, which is substantial - and of critical importance to our company - or we must incur significant expense to correct the issue ourselves.
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 163876 ] JNJira + In-Review [ 181418 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              leedega Kevin Phillips
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: