Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64738

ec2-plugin error when updating config for live EC2 nodes

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Minor
    • Resolution: Fixed
    • ec2-plugin
    • None
    • Jenkins 2.276
      ec2-plugin 1.56
    • 1.60

    Description

      We ran into the following issue after updating to ec2-plugin version 1.56:

      We are no longer able to update the configuration of a live/running EC2 Node. For example, when trying to change "Idle termination time":

      We usually need to set this to "0" or empty when there's issues brining up new EC2 Nodes that need investigating and we want existing nodes to not be destroyed in the meantime.

      We can still change the same configuration for all new/future EC2 Nodes via the "Configure Clouds" link though.

      The issue goes away when reverting ec2-plugin back to version 1.55.

      Here's the (scrubbed) stacktrace we see in the Jenkins system logs for the error:

      Jan 26, 2021 12:50:49 PM WARNING hudson.init.impl.InstallUncaughtExceptionHandler handleException
      Caught unhandled exception with ID 519812a5-a9f0-42da-8494-27c75f4af56a
      org.kohsuke.stapler.NoStaplerConstructorException: There's no @DataBoundConstructor on any constructor of class hudson.plugins.ec2.AMITypeData
      	at org.kohsuke.stapler.ClassDescriptor.loadConstructorParamNames(ClassDescriptor.java:265)
      	at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:775)
      	at org.kohsuke.stapler.RequestImpl.access$200(RequestImpl.java:84)
      	at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:679)
      Caused: java.lang.IllegalArgumentException: Failed to instantiate class hudson.plugins.ec2.AMITypeData from {"rootCommandPrefix":"","slaveCommandPrefix":"","slaveCommandSuffix":"","sshPort":"22"}
      	at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:682)
      	at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:479)
      	at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:788)
      Caused: java.lang.IllegalArgumentException: Failed to convert the amiType parameter of the constructor public hudson.plugins.ec2.EC2OndemandSlave(java.lang.String,java.lang.String,java.lang.String,java.lang.String,int,java.lang.String,hudson.model.Node$Mode,java.lang.String,java.lang.String,java.util.List,java.lang.String,java.lang.String,boolean,java.lang.String,java.lang.String,java.lang.String,java.util.List,java.lang.String,boolean,int,hudson.plugins.ec2.AMITypeData,hudson.plugins.ec2.ConnectionStrategy,int) throws hudson.model.Descriptor$FormException,java.io.IOException
      	at org.kohsuke.stapler.RequestImpl.instantiate(RequestImpl.java:790)
      	at org.kohsuke.stapler.RequestImpl.access$200(RequestImpl.java:84)
      	at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:679)
      Caused: java.lang.IllegalArgumentException: Failed to instantiate class hudson.plugins.ec2.EC2OndemandSlave from {"name":"SCRUBBED (i-SCRUBBED)","spotInstanceRequestId":"","cloudName":"SCRUBBED","templateDescription":"SCRUBBED","instanceId":"i-SCRUBBED","ec2Type":"On Demand","":["10 min","0"],"publicDNS":"","privateDNS":"SCRUBBED","nodeDescription":"","numExecutors":"10","labelString":"SCRUBBED","mode":"NORMAL","initScript":"#!/usr/bin/env bash\nset -xve\necho \"Started Jenkins Init script from jenkins-master at [$(date)]\" >> /tmp/init_script.log\n\ncp ~SCRUBBED/jenkins/worker/init_script.sh /tmp/\nchmod +x /tmp/init_script.sh\n/tmp/init_script.sh\necho \"Finished Jenkins Init script from jenkins-master at [$(date)]\" >> /tmp/init_script.log\n","remoteFS":"/var/lib/jenkins","remoteAdmin":"jenkins","stopOnTerminate":false,"idleTerminationMinutes":"6","tags":[{"name":"os","value":"SCRUBBED"},{"name":"Deployment","value":"SCRUBBED"},{"name":"SCRUBBED","value":"SCRUBBED"},{"name":"Description","value":"SCRUBBED"},{"name":"Environment","value":"production"},{"name":"Product","value":"SCRUBBED"},{"name":"name","value":"SCRUBBED"},{"name":"os_version","value":"SCRUBBED"},{"name":"Name","value":"SCRUBBED"},{"name":"Owner","value":"SCRUBBED"},{"name":"jenkins","value":"worker"},{"name":"jenkins_server_url","value":"SCRUBBED"},{"name":"Service","value":"SCRUBBED"},{"name":"Team","value":"SCRUBBED"},{"name":"super_type","value":"jenkins"},{"name":"type","value":"jenkins-worker"},{"name":"jenkins_slave_type","value":"demand_SCRUBBED"}],"usePrivateDnsName":false,"useDedicatedTenancy":false,"amiType":{"rootCommandPrefix":"","slaveCommandPrefix":"","slaveCommandSuffix":"","sshPort":"22"},"maxTotalUses":"-1","nodeProperties":{"stapler-class-bag":"true"},"Jenkins-Crumb":"a9a18e0b0a36539891af0bf858b4e4fd6e76b24235215b59d62fc533eef79ae1"}
      	at org.kohsuke.stapler.RequestImpl$TypePair.convertJSON(RequestImpl.java:682)
      	at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:479)
      	at org.kohsuke.stapler.RequestImpl.bindJSON(RequestImpl.java:475)
      	at hudson.model.Descriptor.newInstance(Descriptor.java:598)
      Caused: java.lang.Error: Failed to instantiate class hudson.plugins.ec2.EC2OndemandSlave from {"name":"SCRUBBED (i-SCRUBBED)","spotInstanceRequestId":"","cloudName":"SCRUBBED","templateDescription":"SCRUBBED","instanceId":"i-SCRUBBED","ec2Type":"On Demand","":["10 min","0"],"publicDNS":"","privateDNS":"SCRUBBED","nodeDescription":"","numExecutors":"10","labelString":"SCRUBBED","mode":"NORMAL","initScript":"#!/usr/bin/env bash\nset -xve\necho \"Started Jenkins Init script from jenkins-master at [$(date)]\" >> /tmp/init_script.log\n\ncp ~SCRUBBED/jenkins/worker/init_script.sh /tmp/\nchmod +x /tmp/init_script.sh\n/tmp/init_script.sh\necho \"Finished Jenkins Init script from jenkins-master at [$(date)]\" >> /tmp/init_script.log\n","remoteFS":"/var/lib/jenkins","remoteAdmin":"jenkins","stopOnTerminate":false,"idleTerminationMinutes":"6","tags":[{"name":"os","value":"SCRUBBED"},{"name":"Deployment","value":"SCRUBBED"},{"name":"SCRUBBED","value":"SCRUBBED"},{"name":"Description","value":"SCRUBBED"},{"name":"Environment","value":"production"},{"name":"Product","value":"SCRUBBED"},{"name":"name","value":"SCRUBBED"},{"name":"os_version","value":"SCRUBBED"},{"name":"Name","value":"SCRUBBED"},{"name":"Owner","value":"SCRUBBED"},{"name":"jenkins","value":"worker"},{"name":"jenkins_server_url","value":"SCRUBBED"},{"name":"Service","value":"SCRUBBED"},{"name":"Team","value":"SCRUBBED"},{"name":"super_type","value":"jenkins"},{"name":"type","value":"jenkins-worker"},{"name":"jenkins_slave_type","value":"demand_SCRUBBED"}],"usePrivateDnsName":false,"useDedicatedTenancy":false,"amiType":{"rootCommandPrefix":"","slaveCommandPrefix":"","slaveCommandSuffix":"","sshPort":"22"},"maxTotalUses":"-1","nodeProperties":{"stapler-class-bag":"true"},"Jenkins-Crumb":"a9a18e0b0a36539891af0bf858b4e4fd6e76b24235215b59d62fc533eef79ae1"}
      	at hudson.model.Descriptor.newInstance(Descriptor.java:606)
      	at hudson.model.Node.reconfigure(Node.java:547)
      	at hudson.plugins.ec2.EC2AbstractSlave.reconfigure(EC2AbstractSlave.java:463)
      	at hudson.plugins.ec2.EC2OndemandSlave.reconfigure(EC2OndemandSlave.java:126)
      	at hudson.model.Computer.doConfigSubmit(Computer.java:1513)
      	at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
      	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
      	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
      	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
      	at org.kohsuke.stapler.SelectionInterceptedFunction$Adapter.invoke(SelectionInterceptedFunction.java:36)
      	at org.kohsuke.stapler.verb.HttpVerbInterceptor.invoke(HttpVerbInterceptor.java:48)
      	at org.kohsuke.stapler.SelectionInterceptedFunction.bindAndInvoke(SelectionInterceptedFunction.java:26)
      	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
      	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:536)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
      Caused: javax.servlet.ServletException
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:816)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
      	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:457)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
      	at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:220)
      	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
      	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
      	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
      	at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
      	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
      	at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:111)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
      	at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
      	at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
      	at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)
      	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
      	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)
      	at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
      	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
      	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)
      	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
      	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
      	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)
      	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)
      	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)
      	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      	at org.eclipse.jetty.server.Server.handle(Server.java:516)
      	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
      	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
      	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
      	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
      	at java.base/java.lang.Thread.run(Thread.java:834)
      

      Attachments

        Issue Links

          Activity

            rdesmond Ryan Desmond added a comment -

            Caught by the same thing.  I usually put the time to live to 60 (instead of 5) when I need the logs, but otherwise I see the same error.

            rdesmond Ryan Desmond added a comment - Caught by the same thing.  I usually put the time to live to 60 (instead of 5) when I need the logs, but otherwise I see the same error.

            We are still seeing this issue with Jenkins 2.288, and plugin version 1.56, so having to roll back to 1.55.  

            Also worth mentioning we are running Java 11 on our master servers, but the workers are running java 13.

            ruffiano Justin Rodante added a comment - We are still seeing this issue with Jenkins 2.288, and plugin version 1.56, so having to roll back to 1.55.   Also worth mentioning we are running Java 11 on our master servers, but the workers are running java 13.
            markewaite Mark Waite added a comment -

            ruffiano the Jenkins project does no testing of Java 13. The Jenkins project also recommends that the controller JVM and the agent JVM should be running the same JVM version for best results. If your agents are running Java 13, I would expect all sorts of surprises and issues from those agents.

            markewaite Mark Waite added a comment - ruffiano the Jenkins project does no testing of Java 13. The Jenkins project also recommends that the controller JVM and the agent JVM should be running the same JVM version for best results. If your agents are running Java 13, I would expect all sorts of surprises and issues from those agents.

            Facing the same issue with version 1.57. It looks like versions before 1.56 included an extra field in the amiType object needed to correctly deserialize the update request. (since AMITypeData is just an abstract class)

            v1.55:

            "amiType": {"stapler-class": "hudson.plugins.ec2.UnixData", "rootCommandPrefix": "", "slaveCommandPrefix": "", "slaveCommandSuffix": "", "sshPort": "22"}
            

            v1.56+:

            "amiType":{"rootCommandPrefix":"","slaveCommandPrefix":"","slaveCommandSuffix":"","sshPort":"22"}
            
            enricodev Enrico Pelizzon added a comment - Facing the same issue with version 1.57. It looks like versions before 1.56 included an extra field in the amiType object needed to correctly deserialize the update request. (since AMITypeData is just an abstract class) v1.55: "amiType": {"stapler-class": "hudson.plugins.ec2.UnixData", "rootCommandPrefix": "", "slaveCommandPrefix": "", "slaveCommandSuffix": "", "sshPort": "22"} v1.56+: "amiType":{"rootCommandPrefix":"","slaveCommandPrefix":"","slaveCommandSuffix":"","sshPort":"22"}
            enricodev Enrico Pelizzon added a comment - PR submitted: https://github.com/jenkinsci/ec2-plugin/pull/608

            People

              Unassigned Unassigned
              zg_fernandoe Fernando
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: