Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62661

Views with status filters lost upgrading from 2.238 to 2.240

    • 2.258

      Upgraded using apt (running on ubuntu bionic).

      I was able to recover the config file from a support plugin archive, and ungarble it and try again and these items are consistently being eaten.

          [JENKINS-62661] Views with status filters lost upgrading from 2.238 to 2.240

          Josh Soref added a comment -

          I have been able to manually re-create these objects from the gui...

          Josh Soref added a comment - I have been able to manually re-create these objects from the gui...

          Daniel Beck added a comment -

          Were any warnings logged, or did the "old data monitor" trigger and complain about unloadable data? If so, what specifically?

          Daniel Beck added a comment - Were any warnings logged, or did the "old data monitor" trigger and complain about unloadable data? If so, what specifically?

          Josh Soref added a comment -

          I didn't see/couldn't find any.

           

          And I'm used to seeing them.

          Josh Soref added a comment - I didn't see/couldn't find any.   And I'm used to seeing them.

          I seem to have a similar issue, all my ListViews where statusFilter is in use are "missing" and unreadable data:

          ConversionException: Could not call hudson.model.ListView.readResolve() : null : Could not call hudson.model.ListView.readResolve() : null
          ---- Debugging information ----
          message             : Could not call hudson.model.ListView.readResolve() : null
          cause-exception     : com.thoughtworks.xstream.converters.reflection.ObjectAccessException
          cause-message       : Could not call hudson.model.ListView.readResolve() : null
          class               : hudson.model.ListView
          required-type       : hudson.model.ListView
          converter-type      : hudson.util.RobustReflectionConverter
          path                : /com.cloudbees.hudson.plugins.folder.Folder/folderViews/views/hudson.model.ListView
          line number         : 56
          -------------------------------
          

          It seems I can force the error with the JobDSL plugin using a view which tries to set a statusFilter:

          Caused by: java.io.IOException: Unable to read
          	at hudson.model.View.createViewFromXML(View.java:1400)
          	at javaposse.jobdsl.plugin.JenkinsJobManagement.createOrUpdateView(JenkinsJobManagement.java:183)
          	... 11 more
          Caused by: com.thoughtworks.xstream.converters.ConversionException: Could not call hudson.model.ListView.readResolve() : null : Could not call hudson.model.ListView.readResolve() : null
          ---- Debugging information ----
          message             : Could not call hudson.model.ListView.readResolve() : null
          cause-exception     : com.thoughtworks.xstream.converters.reflection.ObjectAccessException
          cause-message       : Could not call hudson.model.ListView.readResolve() : null
          class               : hudson.model.ListView
          required-type       : hudson.model.ListView
          converter-type      : hudson.util.RobustReflectionConverter
          path                : /hudson.model.ListView
          line number         : 43
          version             : not available
          -------------------------------
          	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79)
          	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
          	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
          	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
          	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
          	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
          	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
          	at hudson.util.XStream2.unmarshal(XStream2.java:161)
          	at hudson.util.XStream2.unmarshal(XStream2.java:132)
          	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
          	at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
          	at hudson.model.View.createViewFromXML(View.java:1395)
          	... 12 more
          Caused by: com.thoughtworks.xstream.converters.reflection.ObjectAccessException: Could not call hudson.model.ListView.readResolve() : null
          	at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:72)
          	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
          	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
          	... 23 more
          Caused by: java.lang.UnsupportedOperationException
          	at java.util.AbstractList.add(AbstractList.java:148)
          	at hudson.model.ListView.readResolve(ListView.java:153)
          	at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66)
          	... 25 more
          

          (This is with Jenkins 2.249.1)

          Tobias Gruetzmacher added a comment - I seem to have a similar issue, all my ListViews where statusFilter is in use are "missing" and unreadable data: ConversionException: Could not call hudson.model.ListView.readResolve() : null : Could not call hudson.model.ListView.readResolve() : null ---- Debugging information ---- message : Could not call hudson.model.ListView.readResolve() : null cause-exception : com.thoughtworks.xstream.converters.reflection.ObjectAccessException cause-message : Could not call hudson.model.ListView.readResolve() : null class : hudson.model.ListView required-type : hudson.model.ListView converter-type : hudson.util.RobustReflectionConverter path : /com.cloudbees.hudson.plugins.folder.Folder/folderViews/views/hudson.model.ListView line number : 56 ------------------------------- It seems I can force the error with the JobDSL plugin using a view which tries to set a statusFilter: Caused by: java.io.IOException: Unable to read at hudson.model.View.createViewFromXML(View.java:1400) at javaposse.jobdsl.plugin.JenkinsJobManagement.createOrUpdateView(JenkinsJobManagement.java:183) ... 11 more Caused by: com.thoughtworks.xstream.converters.ConversionException: Could not call hudson.model.ListView.readResolve() : null : Could not call hudson.model.ListView.readResolve() : null ---- Debugging information ---- message : Could not call hudson.model.ListView.readResolve() : null cause-exception : com.thoughtworks.xstream.converters.reflection.ObjectAccessException cause-message : Could not call hudson.model.ListView.readResolve() : null class : hudson.model.ListView required-type : hudson.model.ListView converter-type : hudson.util.RobustReflectionConverter path : /hudson.model.ListView line number : 43 version : not available ------------------------------- at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189) at hudson.util.XStream2.unmarshal(XStream2.java:161) at hudson.util.XStream2.unmarshal(XStream2.java:132) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173) at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053) at hudson.model.View.createViewFromXML(View.java:1395) ... 12 more Caused by: com.thoughtworks.xstream.converters.reflection.ObjectAccessException: Could not call hudson.model.ListView.readResolve() : null at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:72) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) ... 23 more Caused by: java.lang.UnsupportedOperationException at java.util.AbstractList.add(AbstractList.java:148) at hudson.model.ListView.readResolve(ListView.java:153) at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callReadResolve(SerializationMethodInvoker.java:66) ... 25 more (This is with Jenkins 2.249.1)

          Daniel Beck added a comment -

          tgr That looks promising. Do you have the XML that the plugin attempts to submit there?

          Daniel Beck added a comment - tgr That looks promising. Do you have the XML that the plugin attempts to submit there?

          The minimal JobDSL script provoking this error is:

          import javaposse.jobdsl.dsl.views.ListView
          
          listView('test') {
            statusFilter ListView.StatusFilter.ENABLED
          }
          

          which produces this XML:

          <hudson.model.ListView>
              <filterExecutors>false</filterExecutors>
              <filterQueue>false</filterQueue>
              <properties class='hudson.model.View$PropertyList'></properties>
              <jobNames class='tree-set'>
                  <comparator class='hudson.util.CaseInsensitiveComparator'></comparator>
              </jobNames>
              <jobFilters></jobFilters>
              <columns></columns>
              <statusFilter>true</statusFilter>
          </hudson.model.ListView>
          

          Ah: jobFilters is a DescribableList, which doesn't implement public void add(int index, E element) ...

          Tobias Gruetzmacher added a comment - The minimal JobDSL script provoking this error is: import javaposse.jobdsl.dsl.views.ListView listView( 'test' ) { statusFilter ListView.StatusFilter.ENABLED } which produces this XML: <hudson.model.ListView> <filterExecutors> false </filterExecutors> <filterQueue> false </filterQueue> <properties class= 'hudson.model.View$PropertyList' > </properties> <jobNames class= 'tree-set' > <comparator class= 'hudson.util.CaseInsensitiveComparator' > </comparator> </jobNames> <jobFilters> </jobFilters> <columns> </columns> <statusFilter> true </statusFilter> </hudson.model.ListView> Ah: jobFilters is a DescribableList, which doesn't implement public void add(int index, E element) ...

          Tim Jacomb added a comment -

          Tim Jacomb added a comment - There's some notes in here too: https://github.com/jenkins-infra/jenkins.io/pull/3697/files

          timja I'm not sure I get your comment. Do you mean maybe the users facing this issue did not take the upgrade guidelines into account?

          tgr danielbeck just to confirm: job-dsl is not the culprit but a way of triggering the issue, right?

          Evaristo Gutierrez added a comment - timja I'm not sure I get your comment. Do you mean maybe the users facing this issue did not take the upgrade guidelines into account? tgr danielbeck just to confirm: job-dsl is not the culprit but a way of triggering the issue, right?

          Tim Jacomb added a comment -

          egutierrez I meant that markewaite has been documenting the config change that users need to make at least for jcasc in relation to this change.

          Tim Jacomb added a comment - egutierrez I meant that markewaite has been documenting the config change that users need to make at least for jcasc in relation to this change.

          Mark Waite added a comment -

          https://github.com/jenkins-infra/jenkins.io/pull/3697 is the pull request that adds upgrade guide instructions for the move of statusFilter from an attribute of a view to a separately defined component of a view. It doesn't describe the case that jsoref reports but it might be considered to be the start of describing the case that tgr has reported. The description addresses configuration as code and how to make the transition. I assume a similar type of transition will be needed in JobDSL. If someone is willing to submit the JobDSL upgrade instructions, they could also be included in the upgrade guide.

          Mark Waite added a comment - https://github.com/jenkins-infra/jenkins.io/pull/3697 is the pull request that adds upgrade guide instructions for the move of statusFilter from an attribute of a view to a separately defined component of a view. It doesn't describe the case that jsoref reports but it might be considered to be the start of describing the case that tgr has reported. The description addresses configuration as code and how to make the transition. I assume a similar type of transition will be needed in JobDSL. If someone is willing to submit the JobDSL upgrade instructions, they could also be included in the upgrade guide.

          Daniel Beck added a comment - - edited

          This has nothing to do with Job DSL and nobody claimed it did.

          It's just that nobody bothered to do an update test here, and there's no autotest for it either.

          Steps to reproduce:

          1. Install 2.238
          2. Skip plugins
          3. create a job
          4. Go back to dashboard
          5. click "+" in the tab bar
          6. select List View
          7. Select "Enabled job only"
          8. Save
          9. Stop Jenkins
          10. Start 2.240

          The list view you just created will be gone.

          Daniel Beck added a comment - - edited This has nothing to do with Job DSL and nobody claimed it did. It's just that nobody bothered to do an update test here, and there's no autotest for it either. Steps to reproduce: Install 2.238 Skip plugins create a job Go back to dashboard click "+" in the tab bar select List View Select "Enabled job only" Save Stop Jenkins Start 2.240 The list view you just created will be gone.

          Josh Soref added a comment -

          That fits our problem. And it matches the earlier note about "status filter".

          Josh Soref added a comment - That fits our problem. And it matches the earlier note about "status filter".

          Josh Soref added a comment -

          The attachment clearly has:

          <statusFilter>true</statusFilter>

          Josh Soref added a comment - The attachment clearly has: <statusFilter>true</statusFilter>

          Marc Benstein added a comment -

          Same, all views using <statusFilter>true</statusFilter> are yanked from the config on Stable upgrade.

          Marc Benstein added a comment - Same, all views using <statusFilter>true</statusFilter> are yanked from the config on Stable upgrade.

          Mark Waite added a comment -

          We just recently added an entry to the Jenkins 2.249.1 upgrade guide to describe the statusFilter transition in Jenkins 2.249.1 LTS. I'm certainly open to more descriptions or better descriptions of techniques for the upgrade.

          Mark Waite added a comment - We just recently added an entry to the Jenkins 2.249.1 upgrade guide to describe the statusFilter transition in Jenkins 2.249.1 LTS. I'm certainly open to more descriptions or better descriptions of techniques for the upgrade.

          Josh Soref added a comment -

          markewaite: at the very least you have to mention the non declarative case. Crashing on startup isn't a big deal, you search for the error and fix things. Dataloss after upgrade is much worse.

          Personally, I'd suggest adding code on the LTS branch to perform migration at launch.

          Josh Soref added a comment - markewaite : at the very least you have to mention the non declarative case. Crashing on startup isn't a big deal, you search for the error and fix things. Dataloss after upgrade is much worse. Personally, I'd suggest adding code on the LTS branch to perform migration at launch.

          Mark Waite added a comment -

          Thanks jsoref. I hadn't performed the test myself to check the problem as described by Daniel. I wrote the upgrade guide based on my experience as a configuration as code user. My examples were unharmed by removing the statusFilter from the view definition (or at least I didn't detect the harm). I'll need to experiment more to create a better description of the upgrade impact. Daniel's description seems like the right technique - define a 2.235.5 view (JCasC or otherwise) that only uses a status filter and no other setting, then confirm that view is deleted on upgrade to 2.249.1.

          Mark Waite added a comment - Thanks jsoref . I hadn't performed the test myself to check the problem as described by Daniel. I wrote the upgrade guide based on my experience as a configuration as code user. My examples were unharmed by removing the statusFilter from the view definition (or at least I didn't detect the harm). I'll need to experiment more to create a better description of the upgrade impact. Daniel's description seems like the right technique - define a 2.235.5 view (JCasC or otherwise) that only uses a status filter and no other setting, then confirm that view is deleted on upgrade to 2.249.1.

          Daniel Beck added a comment -

          This is a bug in the (generally legitimate) data migration. It could be noted as a known issue, but primarily we need a fix towards 2.249.2.

          Daniel Beck added a comment - This is a bug in the (generally legitimate) data migration. It could be noted as a known issue, but primarily we need a fix towards 2.249.2.

          Peter Wiseman added a comment -

          When upgrading, I typically upgrade to the latest weekly after reviewing known issues.  I have multiple jenkins instances that typically only get upgraded once a year.  Upgrade notes aren't sufficient for my case, which I presume mirrors many other use cases.  Agree with danielbeck that a change introducing data loss is a bug.

          Peter Wiseman added a comment - When upgrading, I typically upgrade to the latest weekly after reviewing known issues.  I have multiple jenkins instances that typically only get upgraded once a year.  Upgrade notes aren't sufficient for my case, which I presume mirrors many other use cases.  Agree with danielbeck that a change introducing data loss is a bug.

          Mark Waite added a comment -

          Thanks pwiseman. I agree that the upgrade guide is not sufficient. We need a fix, preferably in Jenkins 2.249.2.

          Mark Waite added a comment - Thanks pwiseman . I agree that the upgrade guide is not sufficient. We need a fix, preferably in Jenkins 2.249.2.

          Peter Wiseman added a comment -

          The status has changed to Resolution and tagged as fixed in 2.258.  I've re-tested an upgrade from 2.238 to 2.258 and the problem is still present (the views go missing).  What was the RESOLUTION?

          Peter Wiseman added a comment - The status has changed to Resolution and tagged as fixed in 2.258.  I've re-tested an upgrade from 2.238 to 2.258 and the problem is still present (the views go missing).  What was the RESOLUTION?

          Mark Waite added a comment -

          That sounds like the fix provided in PR-4466 did not resolve the issue you are seeing. The fix was a code change in that pull request that was intended to handle the upgrade.

          Mark Waite added a comment - That sounds like the fix provided in PR-4466 did not resolve the issue you are seeing. The fix was a code change in that pull request that was intended to handle the upgrade.

          Mark Waite added a comment - - edited

          pwiseman I suspect more details will be required to duplicate the problem you're seeing. I was unable to duplicate the problem with 2.258, while I was able to duplicate the problem with 2.257. Steps I took:

          Jenkins 2.238

          1. Install Jenkins 2.238 with no plugins
          2. Define a job 'enabled-job' that is an enabled Freestyle job and run it several times
          3. Define a job 'disabled-job' this is a disabled Freestyle job
          4. Define a view 'Enabled' that uses the job status filter to show only enabled jobs, recursing in subfolders, matching the regular expression '.*'
          5. Define a view 'Disabled' that uses the job status filter to show only disabled jobs, recursing in subfolders, matching the regular expression '.*'
          6. Confirm that the view 'Enabled' shows only the enabled job and the view 'Disabled' shows only the disabled job
          7. Stop Jenkins 2.238
          8. Create a backup of the Jenkins home directory for reuse

          Jenkins 2.258

          1. Restore the Jenkins home directory backup from Jenkins 2.238
          2. Start Jenkins 2.258
          3. Confirm that the 'Disabled' and 'Enabled' views are visible in Jenkins 2.258

          Jenkins 2.257

          1. Restore the Jenkins home directory backup from Jenkins 2.238
          2. Start Jenkins 2.257
          3. Confirm that the 'Disabled' and 'Enabled' views are lost in Jenkins 2.257

          If someone needs a copy of the Jenkins home directory backup, I placed it temporarily at https://home.markwaite.net/~mwaite/jenkins-2.238.zip. The login user name is butler with the password butler.

          Mark Waite added a comment - - edited pwiseman I suspect more details will be required to duplicate the problem you're seeing. I was unable to duplicate the problem with 2.258, while I was able to duplicate the problem with 2.257. Steps I took: Jenkins 2.238 Install Jenkins 2.238 with no plugins Define a job 'enabled-job' that is an enabled Freestyle job and run it several times Define a job 'disabled-job' this is a disabled Freestyle job Define a view 'Enabled' that uses the job status filter to show only enabled jobs, recursing in subfolders, matching the regular expression '.*' Define a view 'Disabled' that uses the job status filter to show only disabled jobs, recursing in subfolders, matching the regular expression '.*' Confirm that the view 'Enabled' shows only the enabled job and the view 'Disabled' shows only the disabled job Stop Jenkins 2.238 Create a backup of the Jenkins home directory for reuse Jenkins 2.258 Restore the Jenkins home directory backup from Jenkins 2.238 Start Jenkins 2.258 Confirm that the 'Disabled' and 'Enabled' views are visible in Jenkins 2.258 Jenkins 2.257 Restore the Jenkins home directory backup from Jenkins 2.238 Start Jenkins 2.257 Confirm that the 'Disabled' and 'Enabled' views are lost in Jenkins 2.257 If someone needs a copy of the Jenkins home directory backup, I placed it temporarily at https://home.markwaite.net/~mwaite/jenkins-2.238.zip . The login user name is butler with the password butler .

          Peter Wiseman added a comment -

          markewaite Thanks for detailing the test (that helped), and sorry to have required you to re-test.  I found my problem.  I tested too quickly and failed to notice that the views do not appear on the main page unless there is at least one job.  I was looking at the Welcome screen, and thinking there were no views.  I failed to spot this pre-upgrade.  So all is good.  Many thanks for everyone's work on this.

          Peter Wiseman added a comment - markewaite Thanks for detailing the test (that helped), and sorry to have required you to re-test.  I found my problem.  I tested too quickly and failed to notice that the views do not appear on the main page unless there is at least one job.  I was looking at the Welcome screen, and thinking there were no views.  I failed to spot this pre-upgrade.  So all is good.  Many thanks for everyone's work on this.

            Unassigned Unassigned
            jsoref Josh Soref
            Votes:
            2 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: