Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62231

ec2 downgrade from 1.50 fails removing the configured cloud

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core, ec2-plugin
    • Jenkins 2.235, EC2 1.50 > EC2 1.49

      If you downgrade from a recent version of ec2-plugin with the security fixes: 1.50.2 or 1.49.2 or 1.46.3 to the previous installed version after you disabled the Administrative Monitor which show the message about weak strategies used (clicking on Dismiss these messages):

      the previous ec2-plugin version doesn't initialize correctly and the cloud configuration is lost. The nodes configurations are still there, but they cannot be used as the plugin didn't start.

      The log printed out:

      2020-05-10 11:12:03.789+0000 [id=33]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
      2020-05-10 11:12:04.301+0000 [id=30]    SEVERE  jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Amazon EC2 plugin v1.49.1 (ec2)
      java.lang.IllegalStateException: Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
              at hudson.ExtensionList.lookupSingleton(ExtensionList.java:451)
              at hudson.diagnosis.OldDataMonitor.get(OldDataMonitor.java:91)
              at hudson.diagnosis.OldDataMonitor.report(OldDataMonitor.java:223)
              at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:368)
              at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:267)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      Caused: com.thoughtworks.xstream.converters.ConversionException: Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0 : Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
      ---- Debugging information ----
      message             : Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
      cause-exception     : java.lang.IllegalStateException
      cause-message       : Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
      class               : hudson.plugins.ec2.PluginImpl
      required-type       : hudson.plugins.ec2.PluginImpl
      converter-type      : hudson.util.RobustReflectionConverter
      path                : /hudson.plugins.ec2.PluginImpl
      line number         : 4
      version             : not available
      -------------------------------
              at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79)
              at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
              at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
              at hudson.util.XStream2.unmarshal(XStream2.java:161)
              at hudson.util.XStream2.unmarshal(XStream2.java:132)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
              at hudson.XmlFile.unmarshal(XmlFile.java:180)
      Caused: java.io.IOException: Unable to read /home/rleon/develop/projects/ec2-plugin/jenkins-home-of-jenkins-2.235.war(8080)/ec2.xml
              at hudson.XmlFile.unmarshal(XmlFile.java:183)
              at hudson.XmlFile.unmarshal(XmlFile.java:163)
              at hudson.Plugin.load(Plugin.java:270)
              at hudson.plugins.ec2.PluginImpl.start(PluginImpl.java:46)
              at hudson.ClassicPluginStrategy.startPlugin(ClassicPluginStrategy.java:403)
              at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:392)
      Caused: java.io.IOException: Failed to initialize
              at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:395)
              at hudson.PluginManager$2$1$1.run(PluginManager.java:549)
              at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
              at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
              at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
              at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
              at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
              at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      2020-05-10 11:12:06.202+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
      2020-05-10 11:12:06.207+0000 [id=36]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
      2020-05-10 11:12:06.209+0000 [id=41]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
      2020-05-10 11:12:06.660+0000 [id=41]    WARNING jenkins.model.Nodes#load: could not load /home/rleon/develop/projects/ec2-plugin/jenkins-home-of-jenkins-2.235.war(8080)/nodes/EC2 (ec2) - linux (i-077e7393c9fb5fea5)-BACKUP
      com.thoughtworks.xstream.mapper.CannotResolveClassException: hudson.plugins.ec2.EC2OndemandSlave
              at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:79)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:379)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:43)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:48)
              at com.thoughtworks.xstream.core.util.HierarchicalStreams.readClassType(HierarchicalStreams.java:29)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:133)
              at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
              at hudson.util.XStream2.unmarshal(XStream2.java:161)
              at hudson.util.XStream2.unmarshal(XStream2.java:132)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
              at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
              at hudson.XmlFile.read(XmlFile.java:149)
      Caused: java.io.IOException: Unable to read /home/rleon/develop/projects/ec2-plugin/jenkins-home-of-jenkins-2.235.war(8080)/nodes/EC2 (ec2) - linux (i-077e7393c9fb5fea5)-BACKUP/config.xml
              at hudson.XmlFile.read(XmlFile.java:151)
              at jenkins.model.Nodes.load(Nodes.java:338)
              at jenkins.model.Jenkins$13.run(Jenkins.java:3254)
              at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
              at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
              at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
              at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
              at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
              at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      2020-05-10 11:12:06.661+0000 [id=41]    WARNING jenkins.model.Nodes#load: could not load /home/rleon/develop/projects/ec2-plugin/jenkins-home-of-jenkins-2.235.war(8080)/nodes/EC2 (ec2) - linux (i-077e7393c9fb5fea5)
      com.thoughtworks.xstream.mapper.CannotResolveClassException: hudson.plugins.ec2.EC2OndemandSlave
              at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:79)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:379)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:43)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:48)
              at com.thoughtworks.xstream.core.util.HierarchicalStreams.readClassType(HierarchicalStreams.java:29)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:133)
              at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
              at hudson.util.XStream2.unmarshal(XStream2.java:161)
              at hudson.util.XStream2.unmarshal(XStream2.java:132)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
              at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
              at hudson.XmlFile.read(XmlFile.java:149)
      Caused: java.io.IOException: Unable to read /home/rleon/develop/projects/ec2-plugin/jenkins-home-of-jenkins-2.235.war(8080)/nodes/EC2 (ec2) - linux (i-077e7393c9fb5fea5)/config.xml
              at hudson.XmlFile.read(XmlFile.java:151)
              at jenkins.model.Nodes.load(Nodes.java:338)
              at jenkins.model.Jenkins$13.run(Jenkins.java:3254)
              at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
              at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
              at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
              at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
              at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
              at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      2020-05-10 11:12:06.672+0000 [id=38]    INFO    jenkins.InitReactorRunner$1#onAttained: System config loaded
      
      

      If you remove the JENKINS_HOME/ec2.xml file created when you dismissed the specific messages, Jenkins will start correctly. But if you start it before doing this, your cloud configuration will go away.

      Steps to Reproduce

      • Start Jenkins 2.235 without any plugin installation
      • Install ec2-plugin 1.49.1 or any other version without the security fixes (! 1.50.2, 1.49.2, 1.46.3)
      • Configure some cloud and template
      • Update ec2-plugin to 1.50.2
      • Click on Dismiss these messages of the Administrative Monitor. The ec2.xml file is created.
      • Downgrade ec2-plugin to the previous one via pluginManager. Or any other way I believe (although not tested)
      • Restart Jenkins
      • Jenkins starts, the commented warning is printed in the log, the cloud configuration is not there.
      • If you now delete the ec2.xml file, it's too late.

          [JENKINS-62231] ec2 downgrade from 1.50 fails removing the configured cloud

          Ramon Leon added a comment -

          Ramon Leon added a comment - It seems to be a similar issue: https://issues.jenkins-ci.org/browse/JENKINS-59582

          Ramon Leon added a comment - - edited

          It looks like a bug on Jenkins core more likely than a bug on the EC2 plugin itself. danielbeck, oleg_nenashev any idea would be appreciated

          Ramon Leon added a comment - - edited It looks like a bug on Jenkins core more likely than a bug on the EC2 plugin itself. danielbeck , oleg_nenashev any idea would be appreciated

          Daniel Beck added a comment -

          How can this be reproduced? Specific, detailed and complete steps to do that please.

           

          Daniel Beck added a comment - How can this be reproduced? Specific, detailed and complete steps to do that please.  

          Ramon Leon added a comment -

          I may try to modify the start() method of PluginImpl on EC2 plugin to something like:

          @Override
          public void start() {
             try {
                     ...
                     load();
             } catch (Throwable t) {
                      if (t.getCause() instanceof ConversionException && t.getCause().getMessage().contains(OldDataMonitor.class.getName())) {
                            logger.log(Level.INFO, "There was an error loading the plugin, most likely not critical. See https://issues.jenkins-ci.org/browse/JENKINS-62231", t);
                      } else {
                            throw t;
                      }
              }
          }
          

          But it will help for future problems, not to the current problem.

          Ramon Leon added a comment - I may try to modify the start() method of PluginImpl on EC2 plugin to something like: @Override public void start() { try { ... load(); } catch (Throwable t) { if (t.getCause() instanceof ConversionException && t.getCause().getMessage().contains(OldDataMonitor. class. getName())) { logger.log(Level.INFO, "There was an error loading the plugin, most likely not critical. See https: //issues.jenkins-ci.org/browse/JENKINS-62231" , t); } else { throw t; } } } But it will help for future problems, not to the current problem.

          Ramon Leon added a comment -

          Currently, the only workaround is to remove the ec2.xml file before downgrading to a previous version.

          Ramon Leon added a comment - Currently, the only workaround is to remove the ec2.xml file before downgrading to a previous version.

          Ramon Leon added a comment -

          I've filed a PR just to warn users: https://github.com/jenkinsci/ec2-plugin/pull/462 but we should avoid any critical exception because of OldDataMonitor modifying the RobustReflectionConverter IMO.

          Ramon Leon added a comment - I've filed a PR just to warn users: https://github.com/jenkinsci/ec2-plugin/pull/462 but we should avoid any critical exception because of OldDataMonitor modifying the RobustReflectionConverter IMO.

          Ramon Leon added a comment -

          Filed a PR on Jenkins Core to avoid the OldDataMonitor triggers a plugin load failure: https://github.com/jenkinsci/jenkins/pull/4718

          Ramon Leon added a comment - Filed a PR on Jenkins Core to avoid the OldDataMonitor triggers a plugin load failure: https://github.com/jenkinsci/jenkins/pull/4718

          Ramon Leon added a comment -

          danielbeck Added the specific steps to the description. Thanks!

          Ramon Leon added a comment - danielbeck Added the specific steps to the description. Thanks!

          Ramon Leon added a comment -

          After I've moved all the logic from the start method to the postInitialized one, the loading is successful and I can see the Old Data Monitor information in Jenkins:

          But I think it's still needed to avoid a plugin total failure because of a report problem: https://github.com/jenkinsci/jenkins/pull/4718 There is a bunch of plugins (more than 100) doing so which may be on risk.

          Ramon Leon added a comment - After I've moved all the logic from the start method to the postInitialized one, the loading is successful and I can see the Old Data Monitor information in Jenkins: But I think it's still needed to avoid a plugin total failure because of a report problem: https://github.com/jenkinsci/jenkins/pull/4718 There is a bunch of plugins (more than 100) doing so which may be on risk.

          Ramon Leon added a comment -

          Filed a PR on EC2 to get rid of the problem for future versions: https://github.com/jenkinsci/ec2-plugin/pull/464

          Ramon Leon added a comment - Filed a PR on EC2 to get rid of the problem for future versions: https://github.com/jenkinsci/ec2-plugin/pull/464

            mramonleon Ramon Leon
            mramonleon Ramon Leon
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: