Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56535

Caused: java.io.IOException: Unexpected termination of the channel for Azure Agents

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Duplicate
    • Component/s: _unsorted
    • Labels:
      None
    • Environment:
      Jenkins 2.138.2
      Azure VM Plugin: 0.7.5
    • Similar Issues:

      Description

      We are experiencing this problem quite extensively only on our autoscaled agents. 

      The agent get disconnected during the execution of a job and gets deleted in azure as well. This gives us no time to login to VM and see the agent logs.

      The job continues to hold the agent even though it is "disconnected" in jenkins and actual VM is already deleted.

      The Jenkins Agent logs shows: 

      Connection was broken

      java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2681) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49) at hudson.remoting.Command.readFrom(Command.java:140) at hudson.remoting.Command.readFrom(Command.java:126) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:36) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)

       

       

      and we looked at logs on jenkins server and found this: 

       

       

      2019-03-12T13:04:28.614306215Z INFO: AzureVMCloud: createProvisionedAgent: Deployment winconfig7-0312130351356 not yet finished (Running): Microsoft.Compute/virtualMachines:winconfi9e19b0 - waited 30 seconds
      2019-03-12T13:05:01.195797620Z INFO: AzureVMCloud: createProvisionedAgent: Deployment winconfig7-0312130351356 not yet finished (Running): Microsoft.Compute/virtualMachines:winconfi9e19b0 - waited 60 seconds
      2019-03-12T13:05:34.204246893Z INFO: AzureVMCloud: createProvisionedAgent: Deployment winconfig7-0312130351356 not yet finished (Running): Microsoft.Compute/virtualMachines:winconfi9e19b0 - waited 90 seconds
      2019-03-12T13:06:07.630586283Z INFO: AzureVMCloud: createProvisionedAgent: Deployment winconfig7-0312130351356 not yet finished (Running): Microsoft.Compute/virtualMachines:winconfi9e19b0 - waited 120 seconds
      2019-03-12T13:06:40.605311559Z INFO: AzureVMCloud: createProvisionedAgent: VM available: winconfi9e19b0
      2019-03-12T13:06:40.682752835Z  found agent winconfi9e19b0
      2019-03-12T13:06:41.085375927Z nodeNamewinconfi9e19b0
      2019-03-12T13:06:41.085392927Z INFO: Azure Cloud: provision: Adding agent winconfi9e19b0 to Jenkins nodes
      2019-03-12T13:06:41.085557426Z INFO: AzureVMAgent: createComputer: start for agent winconfi9e19b0
      2019-03-12T13:06:41.086645120Z INFO: AzureVMCloudRetensionStrategy: start: azureComputer name winconfi9e19b0
      2019-03-12T13:06:41.124592912Z INFO: AzureVMAgentSSHLauncher: launch: launch method called for agent winconfi9e19b0
      2019-03-12T13:07:31.163904528Z INFO: AzureVMAgentCleanUpTask: cleanVMs: node winconfi9e19b0 blocked to cleanup
      2019-03-12T13:21:34.609221023Z  Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to winconfi9e19b0
      2019-03-12T14:16:07.211594879Z  Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to winconfi9e19b0
      2019-03-12T14:33:49.513820368Z  Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to winconfi9e19b0
      2019-03-12T14:45:50.112174433Z  Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to winconfi9e19b0
      2019-03-12T15:04:38.977113250Z  Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to winconfi9e19b0
      2019-03-12T15:11:23.332151176Z SEVERE: I/O error in channel winconfi9e19b0
      2019-03-12T15:23:24.569513881Z WARNING: Issue with creating launcher for agent winconfi9e19b0. The agent has not been fully initialized yetProbably there is a race condition with Agent reconnection or disconnection, check other log entries
      2019-03-12T15:27:31.917195333Z INFO: AzureVMManagementServiceDelegate: virtualMachineExists: check for winconfi9e19b0
      2019-03-12T15:27:32.116311424Z INFO: AzureVMManagementServiceDelegate: virtualMachineExists: winconfi9e19b0 doesnt exist
      2019-03-12T15:27:32.116318724Z INFO: AzureVMAgentCleanUpTask: cleanVMs: node winconfi9e19b0 doesnt exist, removing
      2019-03-12T15:28:21.183757963Z INFO: cleanLeakedResources: deleting winconfi9e19b0NIC from resource group jenkins-agents

      Is there a suggestion for us, where can we look at to know what exactly is causing this ? 

      Everything was working fine just a week ago and we did not even updated anything.

        Attachments

          Activity

          Hide
          jieshe Jie Shen added a comment -

          Thanks for reporting the issue here. I think this issue may be relevant with https://github.com/jenkinsci/jenkins/blob/1c9eb43283e7321ee4d3a0e1e9995453493ff04a/core/src/main/java/hudson/model/Slave.java#L496-L500 . The agent does not set up connection with master successfully. Do you choose SSH to connect to slave? The code is not enough robust for now. I think we should set timeout and retry strategy for setting up the connection.

          Show
          jieshe Jie Shen added a comment - Thanks for reporting the issue here. I think this issue may be relevant with https://github.com/jenkinsci/jenkins/blob/1c9eb43283e7321ee4d3a0e1e9995453493ff04a/core/src/main/java/hudson/model/Slave.java#L496-L500  . The agent does not set up connection with master successfully. Do you choose SSH to connect to slave? The code is not enough robust for now. I think we should set timeout and retry strategy for setting up the connection.
          Hide
          faizan Muhammad Faizan ul haq added a comment -

          It was working more stable up until two weeks ago. and yes, we are using ssh. 

          It seems like the agent was up and serving the jobs for two hours. the it got allocated to yet another job and during the execution of the job, the VM got disappeared (the deletion seems to be ordrered by the plugin) and job wiill fail with errors like: 

           
          ERROR: Issue with creating launcher for agent testtempba65f0. The agent has not been fully initialized yet
          Channel "unknown": Remote call on testtempba65f0 failed. The channel is closing down or has closed down
           
          We do not get a chance to look into agent, what has gone wrong. This is big time stopper for us right now.

          Show
          faizan Muhammad Faizan ul haq added a comment - It was working more stable up until two weeks ago. and yes, we are using ssh.  It seems like the agent was up and serving the jobs for two hours. the it got allocated to yet another job and during the execution of the job, the VM got disappeared (the deletion seems to be ordrered by the plugin) and job wiill fail with errors like:    ERROR: Issue with creating launcher for agent testtempba65f0. The agent has not been fully initialized yet Channel "unknown": Remote call on testtempba65f0 failed. The channel is closing down or has closed down   We do not get a chance to look into agent, what has gone wrong. This is big time stopper for us right now.
          Hide
          cdavies Chance Davies added a comment -

          Just like Muhammad, we have only begun to see these errors only after the last update. We now have many slaves failing to run jobs which have been very reliable in the past. 

          Show
          cdavies Chance Davies added a comment - Just like Muhammad, we have only begun to see these errors only after the last update. We now have many slaves failing to run jobs which have been very reliable in the past. 
          Hide
          jieshe Jie Shen added a comment -

          Chance Davies Which version were you use before you updated. The latest version of this plugin only fixes error messages in the configuration page and add another way to provide images for agents. This kind of connection errors are always annoying but hard to debug. I plan to put a high priority for vm-agent-plugin's stability in the my next work plan.

          Show
          jieshe Jie Shen added a comment - Chance Davies Which version were you use before you updated. The latest version of this plugin only fixes error messages in the configuration page and add another way to provide images for agents. This kind of connection errors are always annoying but hard to debug. I plan to put a high priority for vm-agent-plugin's stability in the my next work plan.
          Hide
          faizan Muhammad Faizan ul haq added a comment -

          We fought a little and found the issue.

          We spun up a test-jenkins from a backup of our production jenkins server. Both server had same ID and somehow after aprox two hours of an agent creation, the agents created by one server were being removed by other server. Jie Shen can probably explain this. 

          Show
          faizan Muhammad Faizan ul haq added a comment - We fought a little and found the issue. We spun up a test-jenkins from a backup of our production jenkins server. Both server had same ID and somehow after aprox two hours of an agent creation, the agents created by one server were being removed by other server. Jie Shen can probably explain this. 
          Hide
          timja Tim Jacomb added a comment -

          All issues have been transferred to GitHub.

          See https://github.com/jenkinsci/azure-vm-agents-plugin/issues

          Search the issue title to find it.

          (This is a bulk comment and can't link to the specific issue)

          Show
          timja Tim Jacomb added a comment - All issues have been transferred to GitHub. See https://github.com/jenkinsci/azure-vm-agents-plugin/issues Search the issue title to find it. (This is a bulk comment and can't link to the specific issue)

            People

            Assignee:
            jieshe Jie Shen
            Reporter:
            faizan Muhammad Faizan ul haq
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: