Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-41852

Remoting object pinning can result in an object that is released immediately

      The current logic for reference counting and pinning contains a potential bug that could give rise to objects being unexported.

              void pin() {
                  if (referenceCount<Integer.MAX_VALUE/2)
                      referenceCount += Integer.MAX_VALUE/2;
              }
      

      So if we have the sequence:

      • Export table entry created -> 0x00000000
      • Object pinned -> 0x3FFFFFFF
      • ...
      • Object released -> 0x3FFFFFFE
      • Object pinned -> 0x7FFFFFFD
      • Object addRef -> 0x7FFFFFFE
      • Object addRef -> 0x80000000
      • Object deallocated as refcount < 0

      The number one symptom of this bug is:

      2017-02-07 11:11:21.087-0500 [id=47743]	WARNING	h.n.AbstractAsyncNodeMonitorDescriptor#monitor: Failed to monitor ___REDACTED___ for Clock Differencejava.util.concurrent.ExecutionException: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(-1) in the channel exported table
      	at hudson.remoting.Channel$2.adapt(Channel.java:829)
      	at hudson.remoting.Channel$2.adapt(Channel.java:824)
      	at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      	at hudson.node_monitors.AbstractAsyncNodeMonitorDescriptor.monitor(AbstractAsyncNodeMonitorDescriptor.java:96)
      	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:305)
      Caused by: java.io.IOException: Cannot locate RemoteClassLoader.ClassLoaderProxy(-1) in the channel exported table
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:102)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.resolveClass(MultiClassLoaderSerializer.java:123)
      	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1612)
      	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
      	at hudson.remoting.UserResponse.retrieve(UserRequest.java:250)
      	at hudson.remoting.Channel$2.adapt(Channel.java:827)
      	... 4 more
      Caused by: java.util.concurrent.ExecutionException: Invalid object ID 2 iota=26
      	at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:408)
      	at hudson.remoting.ExportTable.get(ExportTable.java:333)
      	at hudson.remoting.Channel.getExportedObject(Channel.java:633)
      	at hudson.remoting.MultiClassLoaderSerializer$Input.readClassLoader(MultiClassLoaderSerializer.java:100)
      	... 13 more
      Caused by: java.lang.Exception: Object appears to be deallocated at lease before Tue Feb 07 11:08:06 EST 2017
      	at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:404)
      	... 16 more
      

      Specifically that this is Invalid object ID 2

      object ID 1 is always the Channel exported as an IChannel

      object ID 2 is always the RemoteClassLoader$ClassLoaderProxy exported as an hudson.remoting.RemoteClassLoader$IClassLoader

      There is no circumstances I can see where object ID 2 should ever be safe to remove. In fact there is code that explicitly pins the classloader

      Strangely when I look at the export tables from agents, however I see things like:

      #2 (ref.571) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.1073744046) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.456) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.516) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.517) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.513) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.1028) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.403) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.313) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.494) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.1073742439) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.447) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.316) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.314) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.316) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.1073742160) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.324) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.321) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.326) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.324) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.318) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.323) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      #2 (ref.320) : object=hudson.remoting.RemoteClassLoader$ClassLoaderProxy@1a1feb69[WebAppClassLoader=Jenkins v2.19.4@1a1feb69] type=hudson.remoting.RemoteClassLoader$ClassLoaderProxy interfaces=[hudson.remoting.RemoteClassLoader$IClassLoader]
      

      which is interesting, because we have some of these pinned (i.e. values like 1073744046, 1073742439 and 1073742160) and some unpinned...

      This is rather indicative of something strange going on

          [JENKINS-41852] Remoting object pinning can result in an object that is released immediately

          Code changed in jenkins
          User: Stephen Connolly
          Path:
          src/main/java/hudson/remoting/ExportTable.java
          http://jenkins-ci.org/commit/remoting/849e01553c4a938add869699068c2fa1e0a67ea6
          Log:
          [FIXED JENKINS-41852] Tweak pinning logic

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/hudson/remoting/ExportTable.java http://jenkins-ci.org/commit/remoting/849e01553c4a938add869699068c2fa1e0a67ea6 Log: [FIXED JENKINS-41852] Tweak pinning logic

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/hudson/remoting/ExportTable.java
          http://jenkins-ci.org/commit/remoting/eb2aa0b2c98d1f5c486d6824b4a3f4c3f61a317f
          Log:
          Merge pull request #148 from stephenc/jenkins-41852-2.x

          [FIXED JENKINS-41852] Tweak pinning logic

          Compare: https://github.com/jenkinsci/remoting/compare/059b89357397...eb2aa0b2c98d

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/remoting/ExportTable.java http://jenkins-ci.org/commit/remoting/eb2aa0b2c98d1f5c486d6824b4a3f4c3f61a317f Log: Merge pull request #148 from stephenc/jenkins-41852-2.x [FIXED JENKINS-41852] Tweak pinning logic Compare: https://github.com/jenkinsci/remoting/compare/059b89357397...eb2aa0b2c98d

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          docs/configuration.md
          src/main/java/hudson/remoting/Base64.java
          src/main/java/hudson/remoting/Channel.java
          src/main/java/hudson/remoting/ErrorPropagatingOutputStream.java
          src/main/java/hudson/remoting/ExportTable.java
          src/main/java/hudson/remoting/LocalChannel.java
          src/main/java/hudson/remoting/RemoteInvocationHandler.java
          src/main/java/hudson/remoting/VirtualChannel.java
          http://jenkins-ci.org/commit/remoting/27bb51dd14db44952348539d15ee004c90c248b2
          Log:
          Document and annotate object Export/Unexport logic

          During last releases we have fixed several issues related to the object export/unexport operations (JENKINS-23271, JENKINS-41852). I decided that it makes sense to improve the method documentation just to make Javadocs more explicit.

          The change also adds missing documentation of hudson.remoting.ExportTable.unexportLogSize

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: docs/configuration.md src/main/java/hudson/remoting/Base64.java src/main/java/hudson/remoting/Channel.java src/main/java/hudson/remoting/ErrorPropagatingOutputStream.java src/main/java/hudson/remoting/ExportTable.java src/main/java/hudson/remoting/LocalChannel.java src/main/java/hudson/remoting/RemoteInvocationHandler.java src/main/java/hudson/remoting/VirtualChannel.java http://jenkins-ci.org/commit/remoting/27bb51dd14db44952348539d15ee004c90c248b2 Log: Document and annotate object Export/Unexport logic During last releases we have fixed several issues related to the object export/unexport operations ( JENKINS-23271 , JENKINS-41852 ). I decided that it makes sense to improve the method documentation just to make Javadocs more explicit. The change also adds missing documentation of hudson.remoting.ExportTable.unexportLogSize

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/815da8aa732baa699481828dda67dd5835ba4992
          Log:
          Update remoting to 3.5

          https://github.com/jenkinsci/remoting/edit/master/CHANGELOG.md

          Fixed issues:

          Improvements:

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/815da8aa732baa699481828dda67dd5835ba4992 Log: Update remoting to 3.5 https://github.com/jenkinsci/remoting/edit/master/CHANGELOG.md Fixed issues: JENKINS-40710 ( https://issues.jenkins-ci.org/browse/JENKINS-40710 ) - Match headers case-insensitively in `JnlpAgentEndpointResolver` in order to be compliant with HTTP2 lower-case headers. ( PR #139 ( https://github.com/jenkinsci/remoting/pull/139 ), PR #140 ( https://github.com/jenkinsci/remoting/pull/140 )) JENKINS-41513 ( https://issues.jenkins-ci.org/browse/JENKINS-41513 ) - Prevent `NullPointerException` in `JnlpAgentEndpointResolver` when receiving a header with `null` name. ( PR #140 ( https://github.com/jenkinsci/remoting/pull/140 )) JENKINS-41852 ( https://issues.jenkins-ci.org/browse/JENKINS-41852 ) - Fix exported object pinning logic to prevent release due to the integer overflow. ( PR #148 ( https://github.com/jenkinsci/remoting/pull/148 )) Improvements: JENKINS-41730 ( https://issues.jenkins-ci.org/browse/JENKINS-41730 ) - Add the new `org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.ignoreJenkinsAgentProtocolsHeader` property, which allows specifying a custom list of supported protocols instead of the one returned by the Jenkins master. ( PR #146 ( https://github.com/jenkinsci/remoting/pull/146 )) Print the Filesystem Jar Cache directory location in the error message when this cache directory is not writable. ( PR #143 ( https://github.com/jenkinsci/remoting/pull/143 )) Replace `MimicException` with the older `ProxyException` when serializing non-serializable exceptions thrown by the remote code. ( PR #141 ( https://github.com/jenkinsci/remoting/pull/141 )) Use OID of the `ClassLoaderProxy` in error message when the proxy cannot be located in the export table. ( PR #147 ( https://github.com/jenkinsci/remoting/pull/147 ))

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/1763e6400f56da0300a44c69b9c10bcf9d2cafe9
          Log:
          Merge pull request #2751 from oleg-nenashev/remoting/3.5

          [JENKINS-40710, JENKINS-41513, JENKINS-41852, JENKINS-41730] - Update remoting to 3.5

          Compare: https://github.com/jenkinsci/jenkins/compare/def0f0d11266...1763e6400f56

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/1763e6400f56da0300a44c69b9c10bcf9d2cafe9 Log: Merge pull request #2751 from oleg-nenashev/remoting/3.5 [JENKINS-40710, JENKINS-41513, JENKINS-41852, JENKINS-41730] - Update remoting to 3.5 Compare: https://github.com/jenkinsci/jenkins/compare/def0f0d11266...1763e6400f56

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          pom.xml
          http://jenkins-ci.org/commit/jenkins/08c5da35d3fafd5cf90143181e07f0d22f4f5d60
          Log:
          Update remoting to 3.5

          https://github.com/jenkinsci/remoting/edit/master/CHANGELOG.md

          Fixed issues:

          Improvements:

          (cherry picked from commit 815da8aa732baa699481828dda67dd5835ba4992)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: pom.xml http://jenkins-ci.org/commit/jenkins/08c5da35d3fafd5cf90143181e07f0d22f4f5d60 Log: Update remoting to 3.5 https://github.com/jenkinsci/remoting/edit/master/CHANGELOG.md Fixed issues: JENKINS-40710 ( https://issues.jenkins-ci.org/browse/JENKINS-40710 ) - Match headers case-insensitively in `JnlpAgentEndpointResolver` in order to be compliant with HTTP2 lower-case headers. ( PR #139 ( https://github.com/jenkinsci/remoting/pull/139 ), PR #140 ( https://github.com/jenkinsci/remoting/pull/140 )) JENKINS-41513 ( https://issues.jenkins-ci.org/browse/JENKINS-41513 ) - Prevent `NullPointerException` in `JnlpAgentEndpointResolver` when receiving a header with `null` name. ( PR #140 ( https://github.com/jenkinsci/remoting/pull/140 )) JENKINS-41852 ( https://issues.jenkins-ci.org/browse/JENKINS-41852 ) - Fix exported object pinning logic to prevent release due to the integer overflow. ( PR #148 ( https://github.com/jenkinsci/remoting/pull/148 )) Improvements: JENKINS-41730 ( https://issues.jenkins-ci.org/browse/JENKINS-41730 ) - Add the new `org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver.ignoreJenkinsAgentProtocolsHeader` property, which allows specifying a custom list of supported protocols instead of the one returned by the Jenkins master. ( PR #146 ( https://github.com/jenkinsci/remoting/pull/146 )) Print the Filesystem Jar Cache directory location in the error message when this cache directory is not writable. ( PR #143 ( https://github.com/jenkinsci/remoting/pull/143 )) Replace `MimicException` with the older `ProxyException` when serializing non-serializable exceptions thrown by the remote code. ( PR #141 ( https://github.com/jenkinsci/remoting/pull/141 )) Use OID of the `ClassLoaderProxy` in error message when the proxy cannot be located in the export table. ( PR #147 ( https://github.com/jenkinsci/remoting/pull/147 )) (cherry picked from commit 815da8aa732baa699481828dda67dd5835ba4992)

            stephenconnolly Stephen Connolly
            stephenconnolly Stephen Connolly
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: