Tim Jacomb added a comment - 2020-11-17 07:57 larsskj Could you try out this pre-release build please? https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/azure-commons/1.0.6-rc220.c094849fd287/azure-commons-1.0.6-rc220.c094849fd287.hpi https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/azure-vm-agents/1.5.2-rc745.d4adbb0f0776/azure-vm-agents-1.5.2-rc745.d4adbb0f0776.hpi (needs both plugins updated)

Lars Skjærlund added a comment - 2020-11-25 12:25 - edited timja Thank you very much for looking into it. Apologize for the late response - I was under the assumption that I would be notified when the ticket was updated? Anyway: I've tested your code above and it works for me. But only partly, unfortunately. The UI problems has gone and I can configure the plugin properly. I can also create a Linux machine using the stock image and it works. But when I try to provision a Windows node based on a custom image, the plugin fails. Here's an extract from the log: AzureVMAgentSSHLauncher: connectToSsh: Got remote connection Nov 25, 2020 11:20:25 AM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launchAzureVMAgentSSHLauncher: launch: checking for java runtime Nov 25, 2020 11:20:25 AM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommandAzureVMAgentSSHLauncher: executeRemoteCommand: starting java -fullversion Nov 25, 2020 11:20:28 AM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommandAzureVMAgentSSHLauncher: executeRemoteCommand: executed successfully Nov 25, 2020 11:20:28 AM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launchAzureVMAgentSSHLauncher: launch: java runtime present, copying remoting.jar to remote Nov 25, 2020 11:20:28 AM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher copyFileToRemoteAzureVMAgentSSHLauncher: copyFileToRemote: Initiating file transfer to remoting.jar Nov 25, 2020 11:20:39 AM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher copyFileToRemoteAzureVMAgentSSHLauncher: copyFileToRemote: copied file Successfully to remoting.jar Nov 25, 2020 11:20:39 AM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launchAzureVMAgentSSHLauncher: launch: launching agent: java -jar remoting.jar Nov 25, 2020 11:20:39 AM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launchAzureVMAgentSSHLauncher: launch: Connected successfully Nov 25, 2020 11:23:12 AM INFO hudson.remoting.SynchronousCommandTransport$ReaderThread runI/O error in channel azure-vmf05cb0 java.io.IOException: Unexpected EOF at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:100) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61) Nov 25, 2020 11:23:12 AM SEVERE com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launchAzureVMAgentSSHLauncher: launch: got exception java.io.IOException: Unexpected EOF at hudson.remoting.ChunkedInputStream.readUntilBreak(ChunkedInputStream.java:100) at hudson.remoting.ChunkedCommandTransport.readBlock(ChunkedCommandTransport.java:39) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61) Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to azure-vmf05cb0 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800) at hudson.remoting.Request.call(Request.java:199) at hudson.remoting.Channel.call(Channel.java:1000) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:678) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:426) at com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher.launch(AzureVMAgentSSHLauncher.java:250) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Caused: hudson.remoting.RequestAbortedException at hudson.remoting.Request.abort(Request.java:345) at hudson.remoting.Channel.terminate(Channel.java:1085) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:89) Nov 25, 2020 11:23:12 AM INFO hudson.remoting.Request$2 runFailed to send back a reply to the request hudson.remoting.Request$2@5f5926a7: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@3b6eb92c:azure-vmf05cb0" : channel is already closed Nov 25, 2020 11:23:13 AM WARNING com.microsoft.jenkins.azurecommons.telemetry.AppInsightsClient sendEventFail to send trace to App Insights due to:Failed to send telemetry event AzureJenkinsPlugin.azure-vm-agents.VMAgent.SSHLaunchFailed, status code is 400, details: Invalid instrumentation key Nov 25, 2020 11:23:13 AM SEVERE com.microsoft.azure.vmagent.AzureVMCloud$2 callFailure to in post-provisioning for 'azure-vmf05cb0' java.io.IOException: Agent failed to connect, even though the launcher didn't report it. See the log output for details. at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:322) Caused: java.util.concurrent.ExecutionException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:866) at com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:818) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Nov 25, 2020 11:23:13 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExistsAzureVMManagementServiceDelegate: virtualMachineExists: check for azure-vmf05cb0 Nov 25, 2020 11:23:13 AM INFO com.microsoft. rest .interceptors.LoggingInterceptor log<-- HTTP FAILED: java.io.IOException: unexpected end of stream on Connection{management.azure.com:443, proxy=DIRECT hostAddress=management.azure.com/13.69.114.0:443 cipherSuite=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 protocol=http/1.1} Nov 25, 2020 11:23:13 AM INFO okhttp3.internal.platform.Platform logALPN callback dropped: HTTP/2 is disabled. Is alpn-boot on the boot class path? Nov 25, 2020 11:23:13 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExistsAzureVMManagementServiceDelegate: virtualMachineExists: azure-vmf05cb0 exists Nov 25, 2020 11:23:13 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate terminateVirtualMachineAzureVMManagementServiceDelegate: terminateVirtualMachine: Removing virtual machine azure-vmf05cb0 Nov 25, 2020 11:25:14 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate terminateVirtualMachineAzureVMManagementServiceDelegate: terminateVirtualMachine: Removing managed disk with id: /subscriptions/7f48c7ad-e1fe-4227-9ba3-7ab0b28b9108/resourceGroups/Jenkins/providers/Microsoft.Compute/disks/azure-vmf05cb0_disk1_978dad1f6d004057b040f25dba032223 Nov 25, 2020 11:25:17 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExistsAzureVMManagementServiceDelegate: virtualMachineExists: check for azure-vmf05cb0 Nov 25, 2020 11:25:17 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExistsAzureVMManagementServiceDelegate: virtualMachineExists: azure-vmf05cb0 doesnt exist Nov 25, 2020 11:25:44 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate terminateVirtualMachineClean operation starting for azure-vmf05cb0 NIC and IP Nov 25, 2020 11:25:44 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate removeIPNameRemove NIC azure-vmf05cb0NIC Nov 25, 2020 11:25:44 AM WARNING com.microsoft.jenkins.azurecommons.telemetry.AppInsightsClient sendEventFail to send trace to App Insights due to:Failed to send telemetry event AzureJenkinsPlugin.azure-vm-agents.VMAgent.ProvisionFailed, status code is 400, details: Invalid instrumentation key Nov 25, 2020 11:25:45 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate removeIPNameRemove IP azure-vmf05cb0IPName Nov 25, 2020 11:25:49 AM WARNING hudson.slaves.NodeProvisioner lambda$update$6Unexpected exception encountered while provisioning agent azure-vm-win java.io.IOException: Agent failed to connect, even though the launcher didn't report it. See the log output for details. at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:322) Caused: java.util.concurrent.ExecutionException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:866) Caused: com.microsoft.azure.vmagent.exceptions.AzureCloudException at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:54) at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:33) at com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:895) at com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:818) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748) Nov 25, 2020 11:26:09 AM INFO com.microsoft.azure.vmagent.AzureVMCloud provisionAzureVMCloud: provision: start for label azure-vm-win workLoad 1 I'll keep an eye on the thread now if I can be of any further help. Best regards, Lars

Tim Jacomb added a comment - 2020-11-25 13:47 notifications is a known issue: https://groups.google.com/g/jenkins-infra/c/LR2VnEEiWak/m/cqqEMaFlAQAJ

Lars Skjærlund added a comment - 2020-11-26 08:06

Tim Jacomb added a comment - 2020-12-26 09:39 Hi larsskj I had time time today to try this out, I haven't got a custom image built but I tried with the built in one and it worked ok. Does it definitely consistently fail? Sometimes the agents do fail to start because of cloudiness.