Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69228

vSphere Cloud plugin losing connection to slaves

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • vsphere-cloud-plugin
    • None
    • Jenkins version 2.346.1
      vsphere-cloud version 2.27
      hashicorp-vault-plugin version 3.8.0

      Hello,

      We have following problem. Our vSphere plugin is losing connection to auto-provisioned slaved in vSphere so it can't terminate idle slaves and they are keep running in vSphere. Only restart of Jenkins controller helps to fix the issue until next time.

      Test connection is working fine. But `Check template` works only on a first Template in the list. Next one is failing:

      Stack trace:

      Aug 03, 2022 9:17:59 AM SEVERE com.vmware.vim25.ws.WSClient invokeException caught while invoking method: Login
com.vmware.vim25.InvalidLogin: Cannot complete login due to an incorrect user name or password.
	at jdk.internal.reflect.GeneratedConstructorAccessor1635.newInstance(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
	at java.base/java.lang.Class.newInstance(Class.java:584)
	at com.vmware.vim25.ws.XmlGenDom.fromXml(XmlGenDom.java:253)
	at com.vmware.vim25.ws.XmlGenDom.parseSoapFault(XmlGenDom.java:151)
	at com.vmware.vim25.ws.XmlGenDom.fromXML(XmlGenDom.java:109)
	at com.vmware.vim25.ws.SoapClient.unMarshall(SoapClient.java:253)
	at com.vmware.vim25.ws.WSClient.invoke(WSClient.java:96)
	at com.vmware.vim25.ws.VimStub.login(VimStub.java:1732)
	at com.vmware.vim25.mo.SessionManager.login(SessionManager.java:153)
	at com.vmware.vim25.mo.ServiceInstance.login(ServiceInstance.java:238)
	at com.vmware.vim25.mo.ServiceInstance.constructServiceInstance(ServiceInstance.java:131)
	at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:79)
	at com.vmware.vim25.mo.ServiceInstance.<init>(ServiceInstance.java:69)
	at org.jenkinsci.plugins.vsphere.tools.VSphere.<init>(VSphere.java:75)
	at org.jenkinsci.plugins.vsphere.tools.VSphere.connect(VSphere.java:98)
	at org.jenkinsci.plugins.vSphereCloud.vSphereInstance(vSphereCloud.java:302)
	at org.jenkinsci.plugins.vSphereCloudSlaveComputer.getVMInformation(vSphereCloudSlaveComputer.java:100)
	at org.jenkinsci.plugins.vSphereCloudSlaveComputer.getVmInformationError(vSphereCloudSlaveComputer.java:77)
	at jdk.internal.reflect.GeneratedMethodAccessor3894.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
	at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
	at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
	at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
	at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
	at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
	at org.apache.commons.jexl.parser.ASTNENode.value(ASTNENode.java:55)
	at org.apache.commons.jexl.parser.ASTExpression.value(ASTExpression.java:54)
	at org.apache.commons.jexl.parser.ASTExpressionExpression.value(ASTExpressionExpression.java:56)
	at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
	at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:76)
	at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61)
	at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsBoolean(ExpressionSupport.java:71)
	at org.apache.commons.jelly.tags.core.WhenTag.doTag(WhenTag.java:44)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
	at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
	at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:99)
	at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:172)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:101)
	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:101)
	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:101)
	at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
	at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122)
	at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
	at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122)
	at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
	at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:99)
	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:66)
	at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:55)
	at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56)
	at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43)
	at org.kohsuke.stapler.Facet.handleIndexRequest(Facet.java:284)
	at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:105)
	at org.kohsuke.stapler.IndexViewDispatcher.dispatch(IndexViewDispatcher.java:32)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
	at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
	at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:224)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:112)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at com.cloudbees.jenkins.support.slowrequest.SlowRequestFilter.doFilter(SlowRequestFilter.java:37)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at com.atlassian.bitbucket.jenkins.internal.applink.oauth.serviceprovider.auth.OAuth1aRequestFilter.doFilter(OAuth1aRequestFilter.java:91)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at org.jenkinsci.plugins.modernstatus.ModernStatusFilter.doFilter(ModernStatusFilter.java:50)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:160)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
	at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:102)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:219)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:213)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:571)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)

      Debug XML error:

      Aug 03, 2022 9:17:59 AM FINEST com.vmware.vim25.ws.XmlGenDom fromXML
XML Document: 
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Body>
<soapenv:Fault>
    <faultcode>ServerFaultCode</faultcode>
    <faultstring>Cannot complete login due to an incorrect user name or password.</faultstring>
    <detail>
        <InvalidLoginFault xmlns="urn:vim25" xsi:type="InvalidLogin">
            <faultMessage>
                <key>vim.fault.InvalidLoginWithReason</key>
                <arg>
                    <key>1</key>
                    <value xsi:type="xsd:string">vim.fault.InvalidLoginReasonRegisterFailed</value>
                </arg>
                <message>Login failed because the session could not be re-registered.</message>
            </faultMessage>
        </InvalidLoginFault>
    </detail>
</soapenv:Fault>
</soapenv:Body>
</soapenv:Envelope>

       

        1. image-2022-08-03-15-26-08-641.png
          image-2022-08-03-15-26-08-641.png
          835 kB

            Unassigned Unassigned
            mkushakov Misha Kushakov
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: