Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54496

Node monitoring should not log exceptions for disconnected nodes

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Minor Minor
    • core, remoting
    • None
    • Jenkins 2.220

      In an environment in which Jenkins nodes are being created and removed a lot, e.g. when using the docker-plugin to create single-use containers as slave nodes, the node monitoring functionality can fill the logs full of spurious exceptions, e.g.

      Oct 13, 2018 10:18:14 PM io.jenkins.docker.DockerTransientNode$1 println
      INFO: Disconnected computer for node 'docker-hst02-00030qwaqtu3g'.
      Oct 13, 2018 10:18:14 PM io.jenkins.docker.DockerTransientNode$1 println
      INFO: Removed Node for node 'docker-hst02-00030qwaqtu3g'.
      Oct 13, 2018 10:18:14 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
      WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Clock Difference
      java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
      	at hudson.remoting.Channel$2.adapt(Channel.java:992)
      	at hudson.remoting.Channel$2.adapt(Channel.java:986)
      	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
      	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
      	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
      Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
      	at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
      	at hudson.remoting.Channel$2.adapt(Channel.java:990)
      	... 5 more
      Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
      	at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
      	at hudson.remoting.ExportTable.get(ExportTable.java:397)
      	at hudson.remoting.Channel.getExportedObject(Channel.java:780)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
      	... 14 more
      
      Oct 13, 2018 10:18:14 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
      WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Free Temp Space
      java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
      	at hudson.remoting.Channel$2.adapt(Channel.java:992)
      	at hudson.remoting.Channel$2.adapt(Channel.java:986)
      	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
      	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
      	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
      Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
      	at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
      	at hudson.remoting.Channel$2.adapt(Channel.java:990)
      	... 5 more
      Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
      	at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
      	at hudson.remoting.ExportTable.get(ExportTable.java:397)
      	at hudson.remoting.Channel.getExportedObject(Channel.java:780)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
      	... 14 more
      
      Oct 13, 2018 10:18:15 PM io.jenkins.docker.DockerTransientNode$1 println
      INFO: Stopped container 'f42781f352e6b9a08e66654e9d385999a4a210bb0956f3dc9f8125e169e245fe' for node 'docker-hst02-00030qwaqtu3g'.
      
      Oct 13, 2018 10:18:15 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
      WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Free Disk Space
      java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
      	at hudson.remoting.Channel$2.adapt(Channel.java:992)
      	at hudson.remoting.Channel$2.adapt(Channel.java:986)
      	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
      	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
      	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
      Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
      	at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
      	at hudson.remoting.Channel$2.adapt(Channel.java:990)
      	... 5 more
      Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
      	at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
      	at hudson.remoting.ExportTable.get(ExportTable.java:397)
      	at hudson.remoting.Channel.getExportedObject(Channel.java:780)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
      	... 14 more
      
      Oct 13, 2018 10:18:15 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
      WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Free Swap Space
      java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
      	at hudson.remoting.Channel$2.adapt(Channel.java:992)
      	at hudson.remoting.Channel$2.adapt(Channel.java:986)
      	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
      	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
      	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
      Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
      	at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
      	at hudson.remoting.Channel$2.adapt(Channel.java:990)
      	... 5 more
      Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
      	at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
      	at hudson.remoting.ExportTable.get(ExportTable.java:397)
      	at hudson.remoting.Channel.getExportedObject(Channel.java:780)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
      	... 14 more
      
      Oct 13, 2018 10:18:15 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
      WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Response Time
      java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
      	at hudson.remoting.Channel$2.adapt(Channel.java:992)
      	at hudson.remoting.Channel$2.adapt(Channel.java:986)
      	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
      	at hudson.node_monitors.ResponseTimeMonitor$1.monitor(ResponseTimeMonitor.java:57)
      	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
      Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
      	at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
      	at hudson.remoting.Channel$2.adapt(Channel.java:990)
      	... 5 more
      Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
      	at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
      	at hudson.remoting.ExportTable.get(ExportTable.java:397)
      	at hudson.remoting.Channel.getExportedObject(Channel.java:780)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
      	... 14 more
      

      This is ugly and obscures "real errors" within the log. This causes real-world problems when trying to debug actual slave disconnection issues, as the log is full of "normal" exceptions that obscure any abnormal exceptions.

      Recommendation:
      Node monitoring code should only report exceptions to the main jenkins.err.log file if the slave node still exists by the time the exception is caught ; there's no point logging huge exception traces for a slave node which has already been removed.
      If the node doesn't exist then either suppress the logging completely or log it at a lower severity so it doesn't fill the main logfile.

          [JENKINS-54496] Node monitoring should not log exceptions for disconnected nodes

          pjdarton created issue -
          pjdarton made changes -
          Description Original: In an environment in which Jenkins nodes are being created and removed a lot, e.g. when using the docker-plugin to create single-use containers as slave nodes, the node monitoring functionality can fill the logs full of spurious exceptions, e.g.
          {code:none}Oct 13, 2018 10:18:14 PM io.jenkins.docker.DockerTransientNode$1 println
          INFO: Disconnected computer for node 'docker-hst02-00030qwaqtu3g'.
          Oct 13, 2018 10:18:14 PM io.jenkins.docker.DockerTransientNode$1 println
          INFO: Removed Node for node 'docker-hst02-00030qwaqtu3g'.
          Oct 13, 2018 10:18:14 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
          WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Clock Difference
          java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.Channel$2.adapt(Channel.java:992)
          at hudson.remoting.Channel$2.adapt(Channel.java:986)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
          Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
          at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
          at hudson.remoting.Channel$2.adapt(Channel.java:990)
          ... 5 more
          Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
          at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
          at hudson.remoting.ExportTable.get(ExportTable.java:397)
          at hudson.remoting.Channel.getExportedObject(Channel.java:780)
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
          ... 14 more

          Oct 13, 2018 10:18:14 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
          WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Free Temp Space
          java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.Channel$2.adapt(Channel.java:992)
          at hudson.remoting.Channel$2.adapt(Channel.java:986)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
          Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
          at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
          at hudson.remoting.Channel$2.adapt(Channel.java:990)
          ... 5 more
          Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
          at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
          at hudson.remoting.ExportTable.get(ExportTable.java:397)
          at hudson.remoting.Channel.getExportedObject(Channel.java:780)
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
          ... 14 more

          Oct 13, 2018 10:18:15 PM io.jenkins.docker.DockerTransientNode$1 println
          INFO: Stopped container 'f42781f352e6b9a08e66654e9d385999a4a210bb0956f3dc9f8125e169e245fe' for node 'docker-hst02-00030qwaqtu3g'.

          Oct 13, 2018 10:18:15 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
          WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Free Disk Space
          java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.Channel$2.adapt(Channel.java:992)
          at hudson.remoting.Channel$2.adapt(Channel.java:986)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
          Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
          at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
          at hudson.remoting.Channel$2.adapt(Channel.java:990)
          ... 5 more
          Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
          at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
          at hudson.remoting.ExportTable.get(ExportTable.java:397)
          at hudson.remoting.Channel.getExportedObject(Channel.java:780)
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
          ... 14 more

          Oct 13, 2018 10:18:15 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
          WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Free Swap Space
          java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.Channel$2.adapt(Channel.java:992)
          at hudson.remoting.Channel$2.adapt(Channel.java:986)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
          Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
          at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
          at hudson.remoting.Channel$2.adapt(Channel.java:990)
          ... 5 more
          Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
          at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
          at hudson.remoting.ExportTable.get(ExportTable.java:397)
          at hudson.remoting.Channel.getExportedObject(Channel.java:780)
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
          ... 14 more

          Oct 13, 2018 10:18:15 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
          WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Response Time
          java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.Channel$2.adapt(Channel.java:992)
          at hudson.remoting.Channel$2.adapt(Channel.java:986)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
          at hudson.node_monitors.ResponseTimeMonitor$1.monitor(ResponseTimeMonitor.java:57)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
          Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
          at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
          at hudson.remoting.Channel$2.adapt(Channel.java:990)
          ... 5 more
          Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
          at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
          at hudson.remoting.ExportTable.get(ExportTable.java:397)
          at hudson.remoting.Channel.getExportedObject(Channel.java:780)
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
          ... 14 more
          {code}

          This is ugly and obscures "real errors" within the log. This causes real-world problems when trying to debug _actual_ slave disconnection issues, as the log is full of "normal" exceptions that obscure any abnormal exceptions.

          Recommendation:
          Node monitoring code should only report exceptions to the main `jenkins.err.log` file if the slave node still exists by the time the exception is caught ; there's no point logging huge exception traces for a slave node which has already been removed.
          If the node doesn't exist then either suppress the logging completely or log it at a lower severity so it doesn't fill the main logfile.
          New: In an environment in which Jenkins nodes are being created and removed a lot, e.g. when using the docker-plugin to create single-use containers as slave nodes, the node monitoring functionality can fill the logs full of spurious exceptions, e.g.
          {code:none}Oct 13, 2018 10:18:14 PM io.jenkins.docker.DockerTransientNode$1 println
          INFO: Disconnected computer for node 'docker-hst02-00030qwaqtu3g'.
          Oct 13, 2018 10:18:14 PM io.jenkins.docker.DockerTransientNode$1 println
          INFO: Removed Node for node 'docker-hst02-00030qwaqtu3g'.
          Oct 13, 2018 10:18:14 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
          WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Clock Difference
          java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.Channel$2.adapt(Channel.java:992)
          at hudson.remoting.Channel$2.adapt(Channel.java:986)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
          Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
          at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
          at hudson.remoting.Channel$2.adapt(Channel.java:990)
          ... 5 more
          Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
          at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
          at hudson.remoting.ExportTable.get(ExportTable.java:397)
          at hudson.remoting.Channel.getExportedObject(Channel.java:780)
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
          ... 14 more

          Oct 13, 2018 10:18:14 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
          WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Free Temp Space
          java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.Channel$2.adapt(Channel.java:992)
          at hudson.remoting.Channel$2.adapt(Channel.java:986)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
          Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
          at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
          at hudson.remoting.Channel$2.adapt(Channel.java:990)
          ... 5 more
          Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
          at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
          at hudson.remoting.ExportTable.get(ExportTable.java:397)
          at hudson.remoting.Channel.getExportedObject(Channel.java:780)
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
          ... 14 more

          Oct 13, 2018 10:18:15 PM io.jenkins.docker.DockerTransientNode$1 println
          INFO: Stopped container 'f42781f352e6b9a08e66654e9d385999a4a210bb0956f3dc9f8125e169e245fe' for node 'docker-hst02-00030qwaqtu3g'.

          Oct 13, 2018 10:18:15 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
          WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Free Disk Space
          java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.Channel$2.adapt(Channel.java:992)
          at hudson.remoting.Channel$2.adapt(Channel.java:986)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
          Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
          at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
          at hudson.remoting.Channel$2.adapt(Channel.java:990)
          ... 5 more
          Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
          at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
          at hudson.remoting.ExportTable.get(ExportTable.java:397)
          at hudson.remoting.Channel.getExportedObject(Channel.java:780)
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
          ... 14 more

          Oct 13, 2018 10:18:15 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
          WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Free Swap Space
          java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.Channel$2.adapt(Channel.java:992)
          at hudson.remoting.Channel$2.adapt(Channel.java:986)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:76)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
          Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
          at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
          at hudson.remoting.Channel$2.adapt(Channel.java:990)
          ... 5 more
          Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
          at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
          at hudson.remoting.ExportTable.get(ExportTable.java:397)
          at hudson.remoting.Channel.getExportedObject(Channel.java:780)
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
          ... 14 more

          Oct 13, 2018 10:18:15 PM hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor monitorDetailed
          WARNING: Failed to monitor docker-hst02-00030qwaqtu3g for Response Time
          java.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.Channel$2.adapt(Channel.java:992)
          at hudson.remoting.Channel$2.adapt(Channel.java:986)
          at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
          at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitorDetailed(AbstractAsyncNodeMonitorDescriptor.java:114)
          at hudson.node_monitors.ResponseTimeMonitor$1.monitor(ResponseTimeMonitor.java:57)
          at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
          Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(2) in the channel exported table
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:107)
          at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:128)
          at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826)
          at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713)
          at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000)
          at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535)
          at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          at hudson.remoting.UserRequest$NormalResponse.retrieve(UserRequest.java:326)
          at hudson.remoting.Channel$2.adapt(Channel.java:990)
          ... 5 more
          Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=7
          at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
          at hudson.remoting.ExportTable.get(ExportTable.java:397)
          at hudson.remoting.Channel.getExportedObject(Channel.java:780)
          at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:105)
          ... 14 more
          {code}

          This is ugly and obscures "real errors" within the log. This causes real-world problems when trying to debug _actual_ slave disconnection issues, as the log is full of "normal" exceptions that obscure any abnormal exceptions.

          Recommendation:
          Node monitoring code should only report exceptions to the main {{jenkins.err.log}} file if the slave node still exists by the time the exception is caught ; there's no point logging huge exception traces for a slave node which has already been removed.
          If the node doesn't exist then either suppress the logging completely or log it at a lower severity so it doesn't fill the main logfile.
          Oleg Nenashev made changes -
          Released As New: Jenkins 2.220
          Resolution New: Fixed [ 1 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]

            jthompson Jeff Thompson
            pjdarton pjdarton
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: