Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62231

ec2 downgrade from 1.50 fails removing the configured cloud

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core, ec2-plugin
    • Jenkins 2.235, EC2 1.50 > EC2 1.49

      If you downgrade from a recent version of ec2-plugin with the security fixes: 1.50.2 or 1.49.2 or 1.46.3 to the previous installed version after you disabled the Administrative Monitor which show the message about weak strategies used (clicking on Dismiss these messages):

      the previous ec2-plugin version doesn't initialize correctly and the cloud configuration is lost. The nodes configurations are still there, but they cannot be used as the plugin didn't start.

      The log printed out:

      2020-05-10 11:12:03.789+0000 [id=33]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
      2020-05-10 11:12:04.301+0000 [id=30]    SEVERE  jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Amazon EC2 plugin v1.49.1 (ec2)
      java.lang.IllegalStateException: Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
              at hudson.ExtensionList.lookupSingleton(ExtensionList.java:451)
              at hudson.diagnosis.OldDataMonitor.get(OldDataMonitor.java:91)
              at hudson.diagnosis.OldDataMonitor.report(OldDataMonitor.java:223)
              at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:368)
              at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:267)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
      Caused: com.thoughtworks.xstream.converters.ConversionException: Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0 : Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
      ---- Debugging information ----
      message             : Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
      cause-exception     : java.lang.IllegalStateException
      cause-message       : Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
      class               : hudson.plugins.ec2.PluginImpl
      required-type       : hudson.plugins.ec2.PluginImpl
      converter-type      : hudson.util.RobustReflectionConverter
      path                : /hudson.plugins.ec2.PluginImpl
      line number         : 4
      version             : not available
      -------------------------------
              at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79)
              at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
              at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
              at hudson.util.XStream2.unmarshal(XStream2.java:161)
              at hudson.util.XStream2.unmarshal(XStream2.java:132)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
              at hudson.XmlFile.unmarshal(XmlFile.java:180)
      Caused: java.io.IOException: Unable to read /home/rleon/develop/projects/ec2-plugin/jenkins-home-of-jenkins-2.235.war(8080)/ec2.xml
              at hudson.XmlFile.unmarshal(XmlFile.java:183)
              at hudson.XmlFile.unmarshal(XmlFile.java:163)
              at hudson.Plugin.load(Plugin.java:270)
              at hudson.plugins.ec2.PluginImpl.start(PluginImpl.java:46)
              at hudson.ClassicPluginStrategy.startPlugin(ClassicPluginStrategy.java:403)
              at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:392)
      Caused: java.io.IOException: Failed to initialize
              at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:395)
              at hudson.PluginManager$2$1$1.run(PluginManager.java:549)
              at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
              at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
              at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
              at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
              at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
              at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      2020-05-10 11:12:06.202+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
      2020-05-10 11:12:06.207+0000 [id=36]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
      2020-05-10 11:12:06.209+0000 [id=41]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
      2020-05-10 11:12:06.660+0000 [id=41]    WARNING jenkins.model.Nodes#load: could not load /home/rleon/develop/projects/ec2-plugin/jenkins-home-of-jenkins-2.235.war(8080)/nodes/EC2 (ec2) - linux (i-077e7393c9fb5fea5)-BACKUP
      com.thoughtworks.xstream.mapper.CannotResolveClassException: hudson.plugins.ec2.EC2OndemandSlave
              at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:79)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:379)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:43)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:48)
              at com.thoughtworks.xstream.core.util.HierarchicalStreams.readClassType(HierarchicalStreams.java:29)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:133)
              at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
              at hudson.util.XStream2.unmarshal(XStream2.java:161)
              at hudson.util.XStream2.unmarshal(XStream2.java:132)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
              at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
              at hudson.XmlFile.read(XmlFile.java:149)
      Caused: java.io.IOException: Unable to read /home/rleon/develop/projects/ec2-plugin/jenkins-home-of-jenkins-2.235.war(8080)/nodes/EC2 (ec2) - linux (i-077e7393c9fb5fea5)-BACKUP/config.xml
              at hudson.XmlFile.read(XmlFile.java:151)
              at jenkins.model.Nodes.load(Nodes.java:338)
              at jenkins.model.Jenkins$13.run(Jenkins.java:3254)
              at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
              at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
              at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
              at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
              at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
              at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      2020-05-10 11:12:06.661+0000 [id=41]    WARNING jenkins.model.Nodes#load: could not load /home/rleon/develop/projects/ec2-plugin/jenkins-home-of-jenkins-2.235.war(8080)/nodes/EC2 (ec2) - linux (i-077e7393c9fb5fea5)
      com.thoughtworks.xstream.mapper.CannotResolveClassException: hudson.plugins.ec2.EC2OndemandSlave
              at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:79)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:379)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:43)
              at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30)
              at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:48)
              at com.thoughtworks.xstream.core.util.HierarchicalStreams.readClassType(HierarchicalStreams.java:29)
              at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:133)
              at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
              at hudson.util.XStream2.unmarshal(XStream2.java:161)
              at hudson.util.XStream2.unmarshal(XStream2.java:132)
              at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
              at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
              at hudson.XmlFile.read(XmlFile.java:149)
      Caused: java.io.IOException: Unable to read /home/rleon/develop/projects/ec2-plugin/jenkins-home-of-jenkins-2.235.war(8080)/nodes/EC2 (ec2) - linux (i-077e7393c9fb5fea5)/config.xml
              at hudson.XmlFile.read(XmlFile.java:151)
              at jenkins.model.Nodes.load(Nodes.java:338)
              at jenkins.model.Jenkins$13.run(Jenkins.java:3254)
              at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
              at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
              at jenkins.model.Jenkins$5.runTask(Jenkins.java:1131)
              at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
              at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
              at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      2020-05-10 11:12:06.672+0000 [id=38]    INFO    jenkins.InitReactorRunner$1#onAttained: System config loaded
      
      

      If you remove the JENKINS_HOME/ec2.xml file created when you dismissed the specific messages, Jenkins will start correctly. But if you start it before doing this, your cloud configuration will go away.

      Steps to Reproduce

      • Start Jenkins 2.235 without any plugin installation
      • Install ec2-plugin 1.49.1 or any other version without the security fixes (! 1.50.2, 1.49.2, 1.46.3)
      • Configure some cloud and template
      • Update ec2-plugin to 1.50.2
      • Click on Dismiss these messages of the Administrative Monitor. The ec2.xml file is created.
      • Downgrade ec2-plugin to the previous one via pluginManager. Or any other way I believe (although not tested)
      • Restart Jenkins
      • Jenkins starts, the commented warning is printed in the log, the cloud configuration is not there.
      • If you now delete the ec2.xml file, it's too late.

          [JENKINS-62231] ec2 downgrade from 1.50 fails removing the configured cloud

          Ramon Leon created issue -
          Ramon Leon made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]

          Ramon Leon added a comment - - edited

          The error seems to be a kind of race-condition between the initialization of the ec2-plugin and the OldDataMonitor extension point.

          RobustReflectionConverter has to report a non-existing field, but it fails with an exception when it tries to do:

          OldDataMonitor.report((Saveable)result, (ArrayList<Throwable>)context.get("ReadError"));
          

          This call ends up on this other one:

           return ExtensionList.lookupSingleton(OldDataMonitor.class); 
          

          which triggers:

          java.lang.IllegalStateException: Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0
          

          Adding @Extension(ordinal=-1) to the EC2 plugin to try to load this one after the AdministrativeMonitors doesn't solve the problem.

          Ramon Leon added a comment - - edited The error seems to be a kind of race-condition between the initialization of the ec2-plugin and the OldDataMonitor extension point. RobustReflectionConverter has to report a non-existing field, but it fails with an exception when it tries to do: OldDataMonitor.report((Saveable)result, (ArrayList<Throwable>)context.get( "ReadError" )); This call ends up on this other one: return ExtensionList.lookupSingleton(OldDataMonitor.class); which triggers: java.lang.IllegalStateException: Expected 1 instance of hudson.diagnosis.OldDataMonitor but got 0 Adding @Extension(ordinal=-1) to the EC2 plugin to try to load this one after the AdministrativeMonitors doesn't solve the problem.
          Ramon Leon made changes -
          Link New: This issue is related to JENKINS-59582 [ JENKINS-59582 ]

          Ramon Leon added a comment -

          Ramon Leon added a comment - It seems to be a similar issue: https://issues.jenkins-ci.org/browse/JENKINS-59582

          Ramon Leon added a comment - - edited

          It looks like a bug on Jenkins core more likely than a bug on the EC2 plugin itself. danielbeck, oleg_nenashev any idea would be appreciated

          Ramon Leon added a comment - - edited It looks like a bug on Jenkins core more likely than a bug on the EC2 plugin itself. danielbeck , oleg_nenashev any idea would be appreciated

          Daniel Beck added a comment -

          How can this be reproduced? Specific, detailed and complete steps to do that please.

           

          Daniel Beck added a comment - How can this be reproduced? Specific, detailed and complete steps to do that please.  

          Ramon Leon added a comment -

          I may try to modify the start() method of PluginImpl on EC2 plugin to something like:

          @Override
          public void start() {
             try {
                     ...
                     load();
             } catch (Throwable t) {
                      if (t.getCause() instanceof ConversionException && t.getCause().getMessage().contains(OldDataMonitor.class.getName())) {
                            logger.log(Level.INFO, "There was an error loading the plugin, most likely not critical. See https://issues.jenkins-ci.org/browse/JENKINS-62231", t);
                      } else {
                            throw t;
                      }
              }
          }
          

          But it will help for future problems, not to the current problem.

          Ramon Leon added a comment - I may try to modify the start() method of PluginImpl on EC2 plugin to something like: @Override public void start() { try { ... load(); } catch (Throwable t) { if (t.getCause() instanceof ConversionException && t.getCause().getMessage().contains(OldDataMonitor. class. getName())) { logger.log(Level.INFO, "There was an error loading the plugin, most likely not critical. See https: //issues.jenkins-ci.org/browse/JENKINS-62231" , t); } else { throw t; } } } But it will help for future problems, not to the current problem.

          Ramon Leon added a comment -

          Currently, the only workaround is to remove the ec2.xml file before downgrading to a previous version.

          Ramon Leon added a comment - Currently, the only workaround is to remove the ec2.xml file before downgrading to a previous version.
          Ramon Leon made changes -
          Link New: This issue relates to JENKINS-62195 [ JENKINS-62195 ]

            mramonleon Ramon Leon
            mramonleon Ramon Leon
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: