Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49027

Improve diagnostics of rejected objects in Remoting

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Minor Minor
    • remoting

      Currently it is hard to understand which field has been rejected from serialization in Remoting. You get only stacktraces like this one:

      [android-lint] Collecting Android Lint files...
      ERROR: Build step failed with exception
      java.lang.SecurityException: Rejected: com.google.common.collect.AbstractMultimap
      	at hudson.remoting.ClassFilter.check(ClassFilter.java:75)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:129)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:277)
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:310)
      	at hudson.remoting.Channel.call(Channel.java:909)
      	at hudson.FilePath.act(FilePath.java:998)
      

      I propose to...

      • If FINE/INFO logger is enabled for the class filter, run deep object analysis and log info about structure and rejected classes
      • Since there is no way to hook on Java deserialization, use Reflection for such operation
      • Nice2have: Print information about rejected classes into a separate file if workdir is enabled

          [JENKINS-49027] Improve diagnostics of rejected objects in Remoting

          Jesse Glick added a comment -

          I have a functional test which reproduced the above stack trace, so I may spend a little time experimenting with a fix for this.

          Jesse Glick added a comment - I have a functional test which reproduced the above stack trace, so I may spend a little time experimenting with a fix for this.

          Jesse Glick added a comment -

          Unnecessary to patch XStream, since that already gives a meaningful stack trace with lines such as

          java.lang.RuntimeException: Failed to serialize com.amazonaws.AmazonServiceException#httpHeaders for class com.amazonaws.services.s3.model.AmazonS3Exception
          

          (Observable by commenting out the agent creation in S3UploadStepIntegrationTest for JENKINS-49025, so that the S3 calls happen on the master without Remoting; then the error appears when org.jenkinsci.plugins.workflow.actions.ErrorAction attempts to save the stack trace. That is not fatal.)

          Jesse Glick added a comment - Unnecessary to patch XStream, since that already gives a meaningful stack trace with lines such as java.lang.RuntimeException: Failed to serialize com.amazonaws.AmazonServiceException#httpHeaders for class com.amazonaws.services.s3.model.AmazonS3Exception (Observable by commenting out the agent creation in S3UploadStepIntegrationTest for JENKINS-49025 , so that the S3 calls happen on the master without Remoting; then the error appears when org.jenkinsci.plugins.workflow.actions.ErrorAction attempts to save the stack trace. That is not fatal.)

          Jesse Glick added a comment -

          …though I filed workflow-api PR 64 to behave better in that respect too.

          Jesse Glick added a comment - …though I filed  workflow-api PR 64 to behave better in that respect too.

          Jesse Glick added a comment -

          To your suggestions:

          run deep object analysis and log info about structure

          I cannot think of any easy options here. JBoss Marshalling has better diagnostics, but uses an incompatible stream format, so it is useless here. Long ago I wrote a tool to dump Java serialization data (purely structural—without doing any actual classloading), which could be revived and brought up to Java 8, but implementing the complete Object Serialization Stream Protocol is quite tedious.

          (Also to use such a tool you would need to retain a copy of the InputStream being passed to ObjectInputStream for purposes of retroactive diagnosis, which adds some overhead.)

          use Reflection for such operation

          I spent some time playing with MultiClassLoaderSerializer.Input but found no way to obtain better diagnostics, even with reflection. There is simply no useful context available outside stack-allocated variables when resolveClass is called, and no other overridable methods which would yield anything of interest.

          Print information about rejected classes into a separate file

          Seems pointless. We already log all rejected classes. That is the most easily available information already.

          if workdir is enabled

          Remoting workdir is inapplicable here, since we are talking about the master.

          Jesse Glick added a comment - To your suggestions: run deep object analysis and log info about structure I cannot think of any easy options here. JBoss Marshalling has better diagnostics, but uses an incompatible stream format, so it is useless here. Long ago I wrote a tool to dump Java serialization data (purely structural—without doing any actual classloading), which could be revived and brought up to Java 8, but implementing the complete Object Serialization Stream Protocol is quite tedious. (Also to use such a tool you would need to retain a copy of the InputStream being passed to ObjectInputStream for purposes of retroactive diagnosis, which adds some overhead.) use Reflection for such operation I spent some time playing with MultiClassLoaderSerializer.Input but found no way to obtain better diagnostics, even with reflection. There is simply no useful context available outside stack-allocated variables when resolveClass is called, and no other overridable methods which would yield anything of interest. Print information about rejected classes into a separate file Seems pointless. We already log all rejected classes. That is the most easily available information already. if workdir is enabled Remoting workdir is inapplicable here, since we are talking about the master.

          Jesse Glick added a comment -

          I was reluctant to considered this assigned/in review given that my PR does not purport to implement the real improvements listed in the description, but OK.

          Jesse Glick added a comment - I was reluctant to considered this assigned/in review given that my PR does not purport to implement the real improvements listed in the description, but OK.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          src/main/java/hudson/remoting/Channel.java
          src/main/java/hudson/remoting/ClassFilter.java
          src/test/java/hudson/remoting/ChannelFilterTest.java
          http://jenkins-ci.org/commit/remoting/9e6472fbffaf7fc84dd84395725f2db404260def
          Log:
          JENKINS-49027 Better report JEP-200 violations in Remoting (#247)

          • Updated test to look for nested causes.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/remoting/Channel.java src/main/java/hudson/remoting/ClassFilter.java src/test/java/hudson/remoting/ChannelFilterTest.java http://jenkins-ci.org/commit/remoting/9e6472fbffaf7fc84dd84395725f2db404260def Log: JENKINS-49027 Better report JEP-200 violations in Remoting (#247) JENKINS-49027 Better report JEP-200 violations in Remoting. Updated test to look for nested causes.

          Oleg Nenashev added a comment - - edited

          It has been released in Remoting 3.17 and Jenkins 2.106

          Oleg Nenashev added a comment - - edited It has been released in Remoting 3.17 and Jenkins 2.106

            Unassigned Unassigned
            oleg_nenashev Oleg Nenashev
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: