-
Bug
-
Resolution: Duplicate
-
Blocker
-
None
-
Jenkins 2.138.2
Azure VM Plugin: 0.7.5
We are experiencing this problem quite extensively only on our autoscaled agents.
The agent get disconnected during the execution of a job and gets deleted in azure as well. This gives us no time to login to VM and see the agent logs.
The job continues to hold the agent even though it is "disconnected" in jenkins and actual VM is already deleted.
The Jenkins Agent logs shows:
Connection was broken
java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:140) at hudson.remoting.Command.readFrom(Command.java:126) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
and we looked at logs on jenkins server and found this:
2019-03-12T13:04:28.614306215Z INFO: AzureVMCloud: createProvisionedAgent: Deployment winconfig7-0312130351356 not yet finished (Running): Microsoft.Compute/virtualMachines:winconfi9e19b0 - waited 30 seconds
2019-03-12T13:05:01.195797620Z INFO: AzureVMCloud: createProvisionedAgent: Deployment winconfig7-0312130351356 not yet finished (Running): Microsoft.Compute/virtualMachines:winconfi9e19b0 - waited 60 seconds
2019-03-12T13:05:34.204246893Z INFO: AzureVMCloud: createProvisionedAgent: Deployment winconfig7-0312130351356 not yet finished (Running): Microsoft.Compute/virtualMachines:winconfi9e19b0 - waited 90 seconds
2019-03-12T13:06:07.630586283Z INFO: AzureVMCloud: createProvisionedAgent: Deployment winconfig7-0312130351356 not yet finished (Running): Microsoft.Compute/virtualMachines:winconfi9e19b0 - waited 120 seconds
2019-03-12T13:06:40.605311559Z INFO: AzureVMCloud: createProvisionedAgent: VM available: winconfi9e19b0
2019-03-12T13:06:40.682752835Z found agent winconfi9e19b0
2019-03-12T13:06:41.085375927Z nodeNamewinconfi9e19b0
2019-03-12T13:06:41.085392927Z INFO: Azure Cloud: provision: Adding agent winconfi9e19b0 to Jenkins nodes
2019-03-12T13:06:41.085557426Z INFO: AzureVMAgent: createComputer: start for agent winconfi9e19b0
2019-03-12T13:06:41.086645120Z INFO: AzureVMCloudRetensionStrategy: start: azureComputer name winconfi9e19b0
2019-03-12T13:06:41.124592912Z INFO: AzureVMAgentSSHLauncher: launch: launch method called for agent winconfi9e19b0
2019-03-12T13:07:31.163904528Z INFO: AzureVMAgentCleanUpTask: cleanVMs: node winconfi9e19b0 blocked to cleanup
2019-03-12T13:21:34.609221023Z Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to winconfi9e19b0
2019-03-12T14:16:07.211594879Z Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to winconfi9e19b0
2019-03-12T14:33:49.513820368Z Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to winconfi9e19b0
2019-03-12T14:45:50.112174433Z Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to winconfi9e19b0
2019-03-12T15:04:38.977113250Z Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to winconfi9e19b0
2019-03-12T15:11:23.332151176Z SEVERE: I/O error in channel winconfi9e19b0
2019-03-12T15:23:24.569513881Z WARNING: Issue with creating launcher for agent winconfi9e19b0. The agent has not been fully initialized yetProbably there is a race condition with Agent reconnection or disconnection, check other log entries
2019-03-12T15:27:31.917195333Z INFO: AzureVMManagementServiceDelegate: virtualMachineExists: check for winconfi9e19b0
2019-03-12T15:27:32.116311424Z INFO: AzureVMManagementServiceDelegate: virtualMachineExists: winconfi9e19b0 doesnt exist
2019-03-12T15:27:32.116318724Z INFO: AzureVMAgentCleanUpTask: cleanVMs: node winconfi9e19b0 doesnt exist, removing
2019-03-12T15:28:21.183757963Z INFO: cleanLeakedResources: deleting winconfi9e19b0NIC from resource group jenkins-agents
Is there a suggestion for us, where can we look at to know what exactly is causing this ?
Everything was working fine just a week ago and we did not even updated anything.