Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48810

How to disable hudson.remoting.RemoteInvocationHandler$Unexporter reportStats?

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Resolved (View Workflow)
    • Priority: Minor
    • Resolution: Done
    • Component/s: remoting
    • Labels:
      None
    • Environment:
      Ubuntu 14.04.5
      Openjdk 8u141
      Jenkins core: 2.60.3
      ssh-slave plugin: 1.16
    • Similar Issues:

      Description

      I recently upgraded Jenkins core from 2.46.2 to 2.60.3 along with some other Jenkins plugins.

      After the upgrade,
      I noticed under the slave node's log,
      the slave node would call "hudson.remoting.RemoteInvocationHandler$Unexporter reportStats" every minute.

      [01/03/18 12:56:17] [SSH] Checking java version of java
      [01/03/18 12:56:17] [SSH] java -version returned 1.8.0_141.
      [01/03/18 12:56:17] [SSH] Starting sftp client.
      [01/03/18 12:56:17] [SSH] Copying latest slave.jar...
      [01/03/18 12:56:18] [SSH] Copied 719,269 bytes.
      Expanded the channel window size to 4MB
      [01/03/18 12:56:18] [SSH] Starting slave process: cd "/home/devops/jenkins_slave_robot" && java  -jar slave.jar
      <===[JENKINS REMOTING CAPACITY]===>channel started
      Slave.jar version: 3.7
      This is a Unix agent
      Evacuated stdout
      Agent successfully connected and online
      Jan 03, 2018 1:44:28 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
      INFO: rate(1min) = 108.5±158.0/sec; rate(5min) = 112.7±187.8/sec; rate(15min) = 100.9±206.8/sec; rate(total) = 76.1±201.5/sec; N = 573
      Jan 03, 2018 1:45:28 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
      INFO: rate(1min) = 99.1±158.2/sec; rate(5min) = 108.7±182.8/sec; rate(15min) = 100.3±203.9/sec; rate(total) = 76.4±200.6/sec; N = 585
      Jan 03, 2018 1:46:28 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
      INFO: rate(1min) = 98.4±150.5/sec; rate(5min) = 107.1±177.2/sec; rate(15min) = 100.2±200.8/sec; rate(total) = 76.8±199.8/sec; N = 597
      Jan 03, 2018 1:47:28 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
      INFO: rate(1min) = 104.5±156.7/sec; rate(5min) = 108.3±175.0/sec; rate(15min) = 101.2±198.8/sec; rate(total) = 77.6±199.2/sec; N = 609
      Jan 03, 2018 1:49:28 PM hudson.remoting.RemoteInvocationHandler$Unexporter reportStats
      INFO: rate(1min) = 106.3±150.0/sec; rate(5min) = 104.2±166.1/sec; rate(15min) = 100.3±192.9/sec; rate(total) = 78.2±197.5/sec; N = 633
      

      Is there anyway I could disable this "hudson.remoting.RemoteInvocationHandler$Unexporter reportStats"?

      or do you know which Jenins core version I need to downgrade back to get rid of this feature?

      or do you know this "hudson.remoting.RemoteInvocationHandler$Unexporter reportStats" is called/triggered by what plugins that I can downgrade or uninstall?

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            The message comes from here: https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/RemoteInvocationHandler.java#L697 . You have a pretty high request rate on the agent, so that it reports the issue. This diagnostics has been introduced in remoting 2.58, which is pretty old now. So this message should be happening in all Jenkins 2 versions, it's not directly related to your upgrade.

            In order to disable the logging, you need to set lower logging level (e.g. WARNING) in RemoteInvocationHandler or Jenkins globally.

            Generally I think that such diagnostics is valid for the INFO level, but it would be possible to add a system property flag to disable that. It would be a feature request, not a bug

            Show
            oleg_nenashev Oleg Nenashev added a comment - The message comes from here: https://github.com/jenkinsci/remoting/blob/master/src/main/java/hudson/remoting/RemoteInvocationHandler.java#L697 . You have a pretty high request rate on the agent, so that it reports the issue. This diagnostics has been introduced in remoting 2.58, which is pretty old now. So this message should be happening in all Jenkins 2 versions, it's not directly related to your upgrade. In order to disable the logging, you need to set lower logging level (e.g. WARNING) in RemoteInvocationHandler or Jenkins globally. Generally I think that such diagnostics is valid for the INFO level, but it would be possible to add a system property flag to disable that. It would be a feature request, not a bug
            Hide
            totoroliu Rick Liu added a comment - - edited

            Thank you Oleg Nenashev very mcuh~

            I know it's not a bug,
            but I don't know what's the best type I should choose~
            Let me change to task type for now.

            Right now I'm trying to troubleshoot https://issues.jenkins-ci.org/browse/JENKINS-25858
            I have also followed the steps https://wiki.jenkins.io/display/JENKINS/Remoting+issue

            But from my Jenkins server logs or slave node logs,
            I don't see "ping failed" messages:

            Ping failed. Terminating

            Hence, I was suspecting something to do with these diagnostic lines.
            as I didn't see these print out before in Jenkins 2.46.2.

            Could you tell me what the high request rate mean?
            Maybe it's the request rate too high caused

            java.io.IOException: Unexpected termination of the channel

            Show
            totoroliu Rick Liu added a comment - - edited Thank you Oleg Nenashev very mcuh~ I know it's not a bug, but I don't know what's the best type I should choose~ Let me change to task type for now. Right now I'm trying to troubleshoot https://issues.jenkins-ci.org/browse/JENKINS-25858 I have also followed the steps https://wiki.jenkins.io/display/JENKINS/Remoting+issue But from my Jenkins server logs or slave node logs, I don't see "ping failed" messages: Ping failed. Terminating Hence, I was suspecting something to do with these diagnostic lines. as I didn't see these print out before in Jenkins 2.46.2. Could you tell me what the high request rate mean? Maybe it's the request rate too high caused java.io.IOException: Unexpected termination of the channel
            Hide
            danielbeck Daniel Beck added a comment -

            Generally I think that such diagnostics is valid for the INFO level, but it would be possible to add a system property flag to disable that. It would be a feature request, not a bug

            So far there's not even a use case for not having stats reported, so let's not.

            Show
            danielbeck Daniel Beck added a comment - Generally I think that such diagnostics is valid for the INFO level, but it would be possible to add a system property flag to disable that. It would be a feature request, not a bug So far there's not even a use case for not having stats reported, so let's not.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            > Could you tell me what the high request rate mean?

            generally it means that there are many requests being invoked on the agent side. It does not immediately mean a functional issue, some plugin like Maven Integration plugin just invoke lots of the requests by design.

            High request rate may cause Remoting outage when there are hanging requests on the agent side (e.g. interlock or so).

            JENKINS-25858 is a too generic issue to handle, it has to be splitted. E.g. your stacktrace there can be handled separately for sure. Will follow-up

            Show
            oleg_nenashev Oleg Nenashev added a comment - > Could you tell me what the high request rate mean? generally it means that there are many requests being invoked on the agent side. It does not immediately mean a functional issue, some plugin like Maven Integration plugin just invoke lots of the requests by design. High request rate may cause Remoting outage when there are hanging requests on the agent side (e.g. interlock or so). JENKINS-25858 is a too generic issue to handle, it has to be splitted. E.g. your stacktrace there can be handled separately for sure. Will follow-up
            Hide
            totoroliu Rick Liu added a comment -

            Thank you very much Oleg~
            I'm very appriciaged your help and explanation!
            I have created a separate bug: JENKINS-48850

            Back to high request rate question~
            I'm not using Maven but using pipeline.
            I'd like to understand request more so that maybe I could tune up my request rate.

            Does each "sh" shell step makes up one request?
            or the whole pipeline file makes up one request?

            When running "Execute Shell" build step in Freestyle project,
            the whole "Execute Shell" build step is copied as shell script file to slave node to run.
            I'm guessing this makes up one request only?

            Show
            totoroliu Rick Liu added a comment - Thank you very much Oleg~ I'm very appriciaged your help and explanation! I have created a separate bug: JENKINS-48850 Back to high request rate question~ I'm not using Maven but using pipeline. I'd like to understand request more so that maybe I could tune up my request rate. Does each "sh" shell step makes up one request? or the whole pipeline file makes up one request? When running "Execute Shell" build step in Freestyle project, the whole "Execute Shell" build step is copied as shell script file to slave node to run. I'm guessing this makes up one request only?
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            > Does each "sh" shell step makes up one request?

            No. It may do lots of calls depending on the plugins you have (LaunchDecorators and so). STDERR/STDOUT is being also sent back to master over remoting, and it may end up in hundreds of calls.

            > or the whole pipeline file makes up one request?

            Definitely no. Pipelines are being executed on Master

            > I'm guessing this makes up one request only?

            No as explained above. I could do some math to calculate numbers of calls, but I would rather wait till JENKINS-27035 from Jesse Glick gets landed. It offers better diagnostics than the currently available tools.

            Show
            oleg_nenashev Oleg Nenashev added a comment - > Does each "sh" shell step makes up one request? No. It may do lots of calls depending on the plugins you have (LaunchDecorators and so). STDERR/STDOUT is being also sent back to master over remoting, and it may end up in hundreds of calls. > or the whole pipeline file makes up one request? Definitely no. Pipelines are being executed on Master > I'm guessing this makes up one request only? No as explained above. I could do some math to calculate numbers of calls, but I would rather wait till JENKINS-27035 from Jesse Glick gets landed. It offers better diagnostics than the currently available tools.
            Hide
            totoroliu Rick Liu added a comment -

            thank you~
            no action is needed~

            Show
            totoroliu Rick Liu added a comment - thank you~ no action is needed~

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              totoroliu Rick Liu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: