Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48810

How to disable hudson.remoting.RemoteInvocationHandler$Unexporter reportStats?

    • Icon: Task Task
    • Resolution: Done
    • Icon: Minor Minor
    • remoting
    • None
    • Ubuntu 14.04.5
      Openjdk 8u141
      Jenkins core: 2.60.3
      ssh-slave plugin: 1.16

      I recently upgraded Jenkins core from 2.46.2 to 2.60.3 along with some other Jenkins plugins.

      After the upgrade,
      I noticed under the slave node's log,
      the slave node would call "hudson.remoting.RemoteInvocationHandler$Unexporter reportStats" every minute.

      [01/03/18 12:56:17] [SSH] Checking java version of java
      [01/03/18 12:56:17] [SSH] java -version returned 1.8.0_141.
      [01/03/18 12:56:17] [SSH] Starting sftp client.
      [01/03/18 12:56:17] [SSH] Copying latest slave.jar...
      [01/03/18 12:56:18] [SSH] Copied 719,269 bytes.
      Expanded the channel window size to 4MB
      [01/03/18 12:56:18] [SSH] Starting slave process: cd "/home/devops/jenkins_slave_robot" && java  -jar slave.jar
      <===[JENKINS REMOTING CAPACITY]===>channel started
      Slave.jar version: 3.7
      This is a Unix agent
      Evacuated stdout
      Agent successfully connected and online
      Jan 03, 2018 1:44:28 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
      INFO: rate(1min) = 108.5±158.0/sec; rate(5min) = 112.7±187.8/sec; rate(15min) = 100.9±206.8/sec; rate(total) = 76.1±201.5/sec; N = 573
      Jan 03, 2018 1:45:28 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
      INFO: rate(1min) = 99.1±158.2/sec; rate(5min) = 108.7±182.8/sec; rate(15min) = 100.3±203.9/sec; rate(total) = 76.4±200.6/sec; N = 585
      Jan 03, 2018 1:46:28 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
      INFO: rate(1min) = 98.4±150.5/sec; rate(5min) = 107.1±177.2/sec; rate(15min) = 100.2±200.8/sec; rate(total) = 76.8±199.8/sec; N = 597
      Jan 03, 2018 1:47:28 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
      INFO: rate(1min) = 104.5±156.7/sec; rate(5min) = 108.3±175.0/sec; rate(15min) = 101.2±198.8/sec; rate(total) = 77.6±199.2/sec; N = 609
      Jan 03, 2018 1:49:28 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
      INFO: rate(1min) = 106.3±150.0/sec; rate(5min) = 104.2±166.1/sec; rate(15min) = 100.3±192.9/sec; rate(total) = 78.2±197.5/sec; N = 633
      

      Is there anyway I could disable this "hudson.remoting.RemoteInvocationHandler$Unexporter reportStats"?

      or do you know which Jenins core version I need to downgrade back to get rid of this feature?

      or do you know this "hudson.remoting.RemoteInvocationHandler$Unexporter reportStats" is called/triggered by what plugins that I can downgrade or uninstall?

          [JENKINS-48810] How to disable hudson.remoting.RemoteInvocationHandler$Unexporter reportStats?

          Oleg Nenashev added a comment -

          The message comes from here: https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/RemoteInvocationHandler.java#L697 . You have a pretty high request rate on the agent, so that it reports the issue. This diagnostics has been introduced in remoting 2.58, which is pretty old now. So this message should be happening in all Jenkins 2 versions, it's not directly related to your upgrade.

          In order to disable the logging, you need to set lower logging level (e.g. WARNING) in RemoteInvocationHandler or Jenkins globally.

          Generally I think that such diagnostics is valid for the INFO level, but it would be possible to add a system property flag to disable that. It would be a feature request, not a bug

          Oleg Nenashev added a comment - The message comes from here: https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/RemoteInvocationHandler.java#L697 . You have a pretty high request rate on the agent, so that it reports the issue. This diagnostics has been introduced in remoting 2.58, which is pretty old now. So this message should be happening in all Jenkins 2 versions, it's not directly related to your upgrade. In order to disable the logging, you need to set lower logging level (e.g. WARNING) in RemoteInvocationHandler or Jenkins globally. Generally I think that such diagnostics is valid for the INFO level, but it would be possible to add a system property flag to disable that. It would be a feature request, not a bug

          Rick Liu added a comment - - edited

          Thank you oleg_nenashev very mcuh~

          I know it's not a bug,
          but I don't know what's the best type I should choose~
          Let me change to task type for now.

          Right now I'm trying to troubleshoot https://issues.jenkins-ci.org/browse/JENKINS-25858
          I have also followed the steps https://wiki.jenkins.io/display/JENKINS/Remoting+issue

          But from my Jenkins server logs or slave node logs,
          I don't see "ping failed" messages:

          Ping failed. Terminating

          Hence, I was suspecting something to do with these diagnostic lines.
          as I didn't see these print out before in Jenkins 2.46.2.

          Could you tell me what the high request rate mean?
          Maybe it's the request rate too high caused

          java.io.IOException: Unexpected termination of the channel

          Rick Liu added a comment - - edited Thank you oleg_nenashev very mcuh~ I know it's not a bug, but I don't know what's the best type I should choose~ Let me change to task type for now. Right now I'm trying to troubleshoot https://issues.jenkins-ci.org/browse/JENKINS-25858 I have also followed the steps https://wiki.jenkins.io/display/JENKINS/Remoting+issue But from my Jenkins server logs or slave node logs, I don't see "ping failed" messages: Ping failed. Terminating Hence, I was suspecting something to do with these diagnostic lines. as I didn't see these print out before in Jenkins 2.46.2. Could you tell me what the high request rate mean? Maybe it's the request rate too high caused java.io.IOException: Unexpected termination of the channel

          Daniel Beck added a comment -

          Generally I think that such diagnostics is valid for the INFO level, but it would be possible to add a system property flag to disable that. It would be a feature request, not a bug

          So far there's not even a use case for not having stats reported, so let's not.

          Daniel Beck added a comment - Generally I think that such diagnostics is valid for the INFO level, but it would be possible to add a system property flag to disable that. It would be a feature request, not a bug So far there's not even a use case for not having stats reported, so let's not.

          Oleg Nenashev added a comment -

          > Could you tell me what the high request rate mean?

          generally it means that there are many requests being invoked on the agent side. It does not immediately mean a functional issue, some plugin like Maven Integration plugin just invoke lots of the requests by design.

          High request rate may cause Remoting outage when there are hanging requests on the agent side (e.g. interlock or so).

          JENKINS-25858 is a too generic issue to handle, it has to be splitted. E.g. your stacktrace there can be handled separately for sure. Will follow-up

          Oleg Nenashev added a comment - > Could you tell me what the high request rate mean? generally it means that there are many requests being invoked on the agent side. It does not immediately mean a functional issue, some plugin like Maven Integration plugin just invoke lots of the requests by design. High request rate may cause Remoting outage when there are hanging requests on the agent side (e.g. interlock or so). JENKINS-25858 is a too generic issue to handle, it has to be splitted. E.g. your stacktrace there can be handled separately for sure. Will follow-up

          Rick Liu added a comment -

          Thank you very much Oleg~
          I'm very appriciaged your help and explanation!
          I have created a separate bug: JENKINS-48850

          Back to high request rate question~
          I'm not using Maven but using pipeline.
          I'd like to understand request more so that maybe I could tune up my request rate.

          Does each "sh" shell step makes up one request?
          or the whole pipeline file makes up one request?

          When running "Execute Shell" build step in Freestyle project,
          the whole "Execute Shell" build step is copied as shell script file to slave node to run.
          I'm guessing this makes up one request only?

          Rick Liu added a comment - Thank you very much Oleg~ I'm very appriciaged your help and explanation! I have created a separate bug: JENKINS-48850 Back to high request rate question~ I'm not using Maven but using pipeline. I'd like to understand request more so that maybe I could tune up my request rate. Does each "sh" shell step makes up one request? or the whole pipeline file makes up one request? When running "Execute Shell" build step in Freestyle project, the whole "Execute Shell" build step is copied as shell script file to slave node to run. I'm guessing this makes up one request only?

          Oleg Nenashev added a comment -

          > Does each "sh" shell step makes up one request?

          No. It may do lots of calls depending on the plugins you have (LaunchDecorators and so). STDERR/STDOUT is being also sent back to master over remoting, and it may end up in hundreds of calls.

          > or the whole pipeline file makes up one request?

          Definitely no. Pipelines are being executed on Master

          > I'm guessing this makes up one request only?

          No as explained above. I could do some math to calculate numbers of calls, but I would rather wait till JENKINS-27035 from jglick gets landed. It offers better diagnostics than the currently available tools.

          Oleg Nenashev added a comment - > Does each "sh" shell step makes up one request? No. It may do lots of calls depending on the plugins you have (LaunchDecorators and so). STDERR/STDOUT is being also sent back to master over remoting, and it may end up in hundreds of calls. > or the whole pipeline file makes up one request? Definitely no. Pipelines are being executed on Master > I'm guessing this makes up one request only? No as explained above. I could do some math to calculate numbers of calls, but I would rather wait till JENKINS-27035 from jglick gets landed. It offers better diagnostics than the currently available tools.

          Rick Liu added a comment -

          thank you~
          no action is needed~

          Rick Liu added a comment - thank you~ no action is needed~

            Unassigned Unassigned
            totoroliu Rick Liu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: