Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57258

Lost builds historical data after updating to Jenkins v2.175

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Blocker Blocker
    • core, pipeline
    • None
    • Jenkins v2.175 / Unix master / Java 64bit 1.8.0_192

      I'm using Multibranch Pipeline to automatically build changes in my Enterprise GitHub Organization repositories.

      After updating Jenkins to 2.175 and restarting, ALL the successful builds of ALL the branches of ALL my GitHub org repositories (that's around 75 repositories and approximately 2,000 builds) have been retriggered and instantly failed. All the builds failed with the same error, see attached console log. Now, all the projects and branches and builds are marked as Failed, and some build information such as the build date appear to be lost (see screenshots).

      In the example log, we can see the build was successfully completed a week ago, but restarted and failed today after restarting Jenkins.  I can consistenly reproduce this issue by re-building a project and restarting Jenkins.

      The issue does not happen if I downgrade to Jenkins 2.174.

        1. branches.PNG
          branches.PNG
          68 kB
        2. build_after.xml
          16 kB
        3. build_before.xml
          25 kB
        4. build_history.PNG
          build_history.PNG
          17 kB
        5. failed_build_console_log.txt
          3 kB

          [JENKINS-57258] Lost builds historical data after updating to Jenkins v2.175

          I did prior to updating Jenkins, but restarted the instance in between. The only plugin updates were related to Blue Ocean.

          Nicolas Echegut added a comment - I did prior to updating Jenkins, but restarted the instance in between. The only plugin updates were related to Blue Ocean.

          Similar issue here, only not on all repos and all branches. I can't find a common pattern unfortunately.

          Jeremy Mathevet added a comment - Similar issue here, only not on all repos and all branches. I can't find a common pattern unfortunately.

          Baptiste Mathus added a comment - - edited

          That would (loosely?) hint about the serialization format change of the workflow-support:3.x line, but if you're saying you had updates all plugins before, that doesn't look like it.
          To be sure: can you confirm you were already running with all latest plugins, specifically workflow-support was 3.2?
          cc dnusbaum

          Baptiste Mathus added a comment - - edited That would (loosely?) hint about the serialization format change of the workflow-support:3.x line, but if you're saying you had updates all plugins before, that doesn't look like it. To be sure: can you confirm you were already running with all latest plugins, specifically workflow-support was 3.2? cc dnusbaum

          Oleg Nenashev added a comment -

          After going through https://github.com/jenkinsci/jenkins/compare/jenkins-2.174...jenkins-2.175 , I do not see anything strictly related to this issue. I could assume a weird side effect of https://github.com/jenkinsci/jenkins/pull/3980 , but there is no data to confirm it.

           

          batmat dnusbaum .dat files and format changes should be related to Running builds only, I am not sure these files are used for completed builds. Any chance we have issues with build status persistency in  build.xml 

           

          nico_echegut If you have build.xml files and System logs after the upgade to 2.175, it would help a lot

          Oleg Nenashev added a comment - After going through  https://github.com/jenkinsci/jenkins/compare/jenkins-2.174...jenkins-2.175  , I do not see anything strictly related to this issue. I could assume a weird side effect of  https://github.com/jenkinsci/jenkins/pull/3980  , but there is no data to confirm it.   batmat dnusbaum  .dat files and format changes should be related to Running builds only, I am not sure these files are used for completed builds. Any chance we have issues with build status persistency in  build.xml    nico_echegut If you have build.xml files and System logs after the upgade to 2.175, it would help a lot

          Devin Nusbaum added a comment -

          Any chance we have issues with build status persistency in build.xml

          We have had issues with this kind of thing in the past (JENKINS-50199), but I am not aware of any related issues in recent plugin versions, although it does remind me of the issues some people are seeing where Pipeline Jobs seem to hold onto flyweight executors after they have apparently completed (discussed in JENKINS-45571). nico_echegut If you have build.xml files from before the upgrade and can tell us whether you updated the Pipeline Job, Pipeline Groovy, or Pipeline Supporting APIs plugins and if so, what versions they were previously and what versions they are now, that would help.

          Devin Nusbaum added a comment - Any chance we have issues with build status persistency in build.xml We have had issues with this kind of thing in the past ( JENKINS-50199 ), but I am not aware of any related issues in recent plugin versions, although it does remind me of the issues some people are seeing where Pipeline Jobs seem to hold onto flyweight executors after they have apparently completed (discussed in JENKINS-45571 ). nico_echegut If you have build.xml files from before the upgrade and can tell us whether you updated the Pipeline Job, Pipeline Groovy, or Pipeline Supporting APIs plugins and if so, what versions they were previously and what versions they are now, that would help.

          I grabbed the build.xml from a build before and after restart. 

          I updated workflow-support from 3.2 to 3.3 right before upgrading Jenkins, but restarted Jenkins in-between. I was already on the latest version of workflow-job-plugin and workflow-cps-plugin.

          Nicolas Echegut added a comment - I grabbed the build.xml from a build before and after restart.  I updated workflow-support from 3.2 to 3.3 right before upgrading Jenkins, but restarted Jenkins in-between. I was already on the latest version of workflow-job-plugin and workflow-cps-plugin.

          Devin Nusbaum added a comment -

          Based on those files, the pre-update run and flow execution have a result, the run is marked as completed and the flow execution is marked as done, which all looks good. The only things that seem off is that the flow execution does not have persistedClean = true, but that by itself shouldn't be a problem, and that the CloverHtmlBuildAction and checkout information were nulled out as part of the update. This does remind me of JENKINS-57254, which was also reported against 2.175 recently, although apparently that was caused by the SCoverage plugin which doesn't appear in your build.xml files. Do you see any errors in your Jenkins system logs around the time of the upgrade?

          Devin Nusbaum added a comment - Based on those files, the pre-update run and flow execution have a result, the run is marked as completed and the flow execution is marked as done, which all looks good. The only things that seem off is that the flow execution does not have persistedClean = true , but that by itself shouldn't be a problem, and that the CloverHtmlBuildAction and checkout information were nulled out as part of the update. This does remind me of JENKINS-57254 , which was also reported against 2.175 recently, although apparently that was caused by the SCoverage plugin which doesn't appear in your build.xml files. Do you see any errors in your Jenkins system logs around the time of the upgrade?

          I don't see anything useful in the log and error files from the day of the upgrade.

          I'm not using the SCoverage plugin.

          Nicolas Echegut added a comment - I don't see anything useful in the log and error files from the day of the upgrade. I'm not using the SCoverage plugin.

          Daniel Beck added a comment -

          Does this problem still occur with Jenkins 2.176? I assume instances were restored from backup when this happened, so what happens when you skip 2.175?

          Daniel Beck added a comment - Does this problem still occur with Jenkins 2.176? I assume instances were restored from backup when this happened, so what happens when you skip 2.175?

          Unfortunately this Jenkins instance did not have any backup so nothing was recovered. I skipped 2.175 and 2.176 and upgraded to 2.177. The good news is the issue with builds automatically restarting and failing after a Jenkins restart is not occurring anymore. The bad news is we probably will never know what happened now  It may have been related to JENKINS-57254.

          Anyway, I will go ahead and close this Jira since the issue cannot be reproduced anymore. Thanks for looking into this.

          Nicolas Echegut added a comment - Unfortunately this Jenkins instance did not have any backup so nothing was recovered. I skipped 2.175 and 2.176 and upgraded to 2.177. The good news is the issue with builds automatically restarting and failing after a Jenkins restart is not occurring anymore. The bad news is we probably will never know what happened now   It may have been related to  JENKINS-57254 . Anyway, I will go ahead and close this Jira since the issue cannot be reproduced anymore. Thanks for looking into this.

            Unassigned Unassigned
            nico_echegut Nicolas Echegut
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: