Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29011

Jenkins 1.609.1 reverse migration script doesn't work when using a custom log folder

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • core
    • None

      We recently upgraded our Jenkins LTS instance from v1.532.3 to the latest 1.609.1 version, after which we discovered some critical defects that required us to roll back to the prior version, 1.596.3. In so doing we encountered issues loading build logs and such due to the incompatible configuration changes applied by 1.609.1. We proceeded to run the downgrade script as described here to no avail. The script appeared to have completed without error but it didn't actually apply any changes to the configuration.

      Further ad-hoc testing reveals that one contributing factor is that we have set a custom path for the Jenkins logs on our master configuration (ie: ${JENKINS_HOME}\logs\builds\${ITEM_FULL_NAME}) and this seems to confuse the reverse migration script. To test my theory I moved a few of our build log folders out of our custom folder and into their default location under ${JENKINS_HOME}\jobs\ ${ITEM_FULL_NAME}\builds and then re-ran the downgrade script. All of the logs I had relocated were correctly downgraded to the old configuration, however none of the logs that were left under the custom log folder were touched.

      Needless to say moving the build logs back to their default locations is tedious and error prone at best, and the effort is exacerbated by larger build farms like ours where we have well over 1000 jobs which would need to be migrated.

      This, once again, puts us between a rock and a hard place. We are unable to continue using v1.609.1 due to failures to preserve backwards compatibility and if we downgrade to the previous version we are going to loose all of our build history, which is substantial - and of critical importance to our company - or we must incur significant expense to correct the issue ourselves.

          [JENKINS-29011] Jenkins 1.609.1 reverse migration script doesn't work when using a custom log folder

          Kevin Phillips created issue -
          Kevin Phillips made changes -
          Link New: This issue is related to JENKINS-24380 [ JENKINS-24380 ]

          Daniel Beck added a comment -

          'jobs' has always been the default location for build logs, 'logs' was never used for build logs. It can be customized in the global Jenkins config (first 'Advanced' button on the right) though.

          Only the default workspace location was moved in newly set up installs only (from '/job/foo/workspace' to 'workspace/foo').

          Daniel Beck added a comment - 'jobs' has always been the default location for build logs, 'logs' was never used for build logs. It can be customized in the global Jenkins config (first 'Advanced' button on the right) though. Only the default workspace location was moved in newly set up installs only (from '/job/foo/workspace' to 'workspace/foo').

          ah - my appologies. We must have customized that early on years ago when we first adopted Jenkins.

          That being the case, it would seem that this defect would apply to users who have customized the location of the logs folder then. The downgrading script appears to ignore those settings.

          Kevin Phillips added a comment - ah - my appologies. We must have customized that early on years ago when we first adopted Jenkins. That being the case, it would seem that this defect would apply to users who have customized the location of the logs folder then. The downgrading script appears to ignore those settings.
          Kevin Phillips made changes -
          Summary Original: Jenkins 1.609.1 reverse migration script doesn't always work after an upgrade New: Jenkins 1.609.1 reverse migration script doesn't work when using a custom log folder
          Kevin Phillips made changes -
          Description Original: We recently upgraded our Jenkins LTS instance from v1.532.3 to the latest 1.609.1 version, after which we discovered some critical defects that required us to roll back to the prior version, 1.596.3. In so doing we encountered issues loading build logs and such due to the incompatible configuration changes applied by 1.609.1. We proceeded to run the downgrade script as described [here|https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-24380+Migration] to no avail. The script appeared to have completed without error but it didn't actually apply any changes to the configuration.

          Further ad-hoc testing seems to indicate that the underlying problem has to do with the way newer versions of Jenkins store their build logs. For example, from what I can tell, if you install 1.609.1 on a clean system it will place build logs under the ./jobs subfolder along with the job they are associated with. However in older Jenkins versions the build logs were stored under the ./logs folder, disconnected from the job configurations. The interesting bit is that when you update from the legacy versions to the latest version the build logs don't appear to get moved to the new location under ./jobs. The master appears to have some way of indicating to itself that legacy build logs existed during the upgrade and that it should continue to maintain the logs in that location instead of the new location. Everything, that is, except for this reverse migration script.

          To check my theory I moved a few of our build log folders out of the ./logs folder and into their more modern location under ./jobs and then re-ran the downgrade script. All of the logs I had relocated were correctly downgraded to the old configuration, however none of the logs that were left under the old logs folder were touched.

          Needless to say moving the build logs to the new location is tedious and error prone at best, and the effort is exacerbated by larger build farms like ours where we have well over 1000 jobs which would need to be migrated.

          This, once again, puts us between a rock and a hard place. We are unable to continue using v1.609.1 due to failures to preserve backwards compatibility and if we downgrade to the previous version we are going to loose all of our build history, which is substantial - and of critical importance to our company - or we must incur significant expense to correct the issue ourselves.
          New: We recently upgraded our Jenkins LTS instance from v1.532.3 to the latest 1.609.1 version, after which we discovered some critical defects that required us to roll back to the prior version, 1.596.3. In so doing we encountered issues loading build logs and such due to the incompatible configuration changes applied by 1.609.1. We proceeded to run the downgrade script as described [here|https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-24380+Migration] to no avail. The script appeared to have completed without error but it didn't actually apply any changes to the configuration.

          Further ad-hoc testing reveals that one contributing factor is that we have set a custom path for the Jenkins logs on our master configuration (ie: ${JENKINS_HOME}\logs\builds\${ITEM_FULL_NAME}) and this seems to confuse the reverse migration script. To test my theory I moved a few of our build log folders out of our custom folder and into their default location under ${JENKINS_HOME}\jobs\ ${ITEM_FULL_NAME}\builds and then re-ran the downgrade script. All of the logs I had relocated were correctly downgraded to the old configuration, however none of the logs that were left under the custom log folder were touched.

          Needless to say moving the build logs back to their default locations is tedious and error prone at best, and the effort is exacerbated by larger build farms like ours where we have well over 1000 jobs which would need to be migrated.

          This, once again, puts us between a rock and a hard place. We are unable to continue using v1.609.1 due to failures to preserve backwards compatibility and if we downgrade to the previous version we are going to loose all of our build history, which is substantial - and of critical importance to our company - or we must incur significant expense to correct the issue ourselves.
          Kevin Phillips made changes -
          Description Original: We recently upgraded our Jenkins LTS instance from v1.532.3 to the latest 1.609.1 version, after which we discovered some critical defects that required us to roll back to the prior version, 1.596.3. In so doing we encountered issues loading build logs and such due to the incompatible configuration changes applied by 1.609.1. We proceeded to run the downgrade script as described [here|https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-24380+Migration] to no avail. The script appeared to have completed without error but it didn't actually apply any changes to the configuration.

          Further ad-hoc testing reveals that one contributing factor is that we have set a custom path for the Jenkins logs on our master configuration (ie: ${JENKINS_HOME}\logs\builds\${ITEM_FULL_NAME}) and this seems to confuse the reverse migration script. To test my theory I moved a few of our build log folders out of our custom folder and into their default location under ${JENKINS_HOME}\jobs\ ${ITEM_FULL_NAME}\builds and then re-ran the downgrade script. All of the logs I had relocated were correctly downgraded to the old configuration, however none of the logs that were left under the custom log folder were touched.

          Needless to say moving the build logs back to their default locations is tedious and error prone at best, and the effort is exacerbated by larger build farms like ours where we have well over 1000 jobs which would need to be migrated.

          This, once again, puts us between a rock and a hard place. We are unable to continue using v1.609.1 due to failures to preserve backwards compatibility and if we downgrade to the previous version we are going to loose all of our build history, which is substantial - and of critical importance to our company - or we must incur significant expense to correct the issue ourselves.
          New: We recently upgraded our Jenkins LTS instance from v1.532.3 to the latest 1.609.1 version, after which we discovered some critical defects that required us to roll back to the prior version, 1.596.3. In so doing we encountered issues loading build logs and such due to the incompatible configuration changes applied by 1.609.1. We proceeded to run the downgrade script as described [here|https://wiki.jenkins-ci.org/display/JENKINS/JENKINS-24380+Migration] to no avail. The script appeared to have completed without error but it didn't actually apply any changes to the configuration.

          Further ad-hoc testing reveals that one contributing factor is that we have set a custom path for the Jenkins logs on our master configuration (ie: $\{JENKINS_HOME}\logs\builds\$\{ITEM_FULL_NAME}) and this seems to confuse the reverse migration script. To test my theory I moved a few of our build log folders out of our custom folder and into their default location under $\{JENKINS_HOME}\jobs\ $\{ITEM_FULL_NAME}\builds and then re-ran the downgrade script. All of the logs I had relocated were correctly downgraded to the old configuration, however none of the logs that were left under the custom log folder were touched.

          Needless to say moving the build logs back to their default locations is tedious and error prone at best, and the effort is exacerbated by larger build farms like ours where we have well over 1000 jobs which would need to be migrated.

          This, once again, puts us between a rock and a hard place. We are unable to continue using v1.609.1 due to failures to preserve backwards compatibility and if we downgrade to the previous version we are going to loose all of our build history, which is substantial - and of critical importance to our company - or we must incur significant expense to correct the issue ourselves.

          I just re-worded the defect description to reflect this new understanding of the Jenkins folder structure.

          Kevin Phillips added a comment - I just re-worded the defect description to reflect this new understanding of the Jenkins folder structure.

          I probably should mention that I have managed to take those few sample projects I downgraded as part of my debugging work for this defect and extrapolate a pattern from the changes the downgrade script made to the folders and XML files, which allowed me to create a simple Python script that essentially walked the build logs sub-tree and corrected the legacy logs so they work correctly with the 1.596.3 edition.

          This has (thankfully) gotten us out of this particular difficult situation, however it probably would be good to still correct your reverse migration script so others don't hit this problem as well.

          Kevin Phillips added a comment - I probably should mention that I have managed to take those few sample projects I downgraded as part of my debugging work for this defect and extrapolate a pattern from the changes the downgrade script made to the folders and XML files, which allowed me to create a simple Python script that essentially walked the build logs sub-tree and corrected the legacy logs so they work correctly with the 1.596.3 edition. This has (thankfully) gotten us out of this particular difficult situation, however it probably would be good to still correct your reverse migration script so others don't hit this problem as well.
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 163876 ] New: JNJira + In-Review [ 181418 ]

            Unassigned Unassigned
            leedega Kevin Phillips
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: