Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22853

SEVERE: Trying to unexport an object that's already unexported

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • core

      Just upgraded to 1.561, and I'm happy to see JENKINS-20769 fixed, but now I'm seeing these instead:

      SEVERE: Trying to unexport an object that's already unexported
      java.lang.IllegalStateException: Invalid object ID 6 iota=36
      	at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:277)
      	at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:300)
      	at hudson.remoting.Channel.unexport(Channel.java:600)
      	at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:38)
      	at hudson.remoting.Channel$2.handle(Channel.java:475)
      	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
      Caused by: java.lang.Exception: Object was recently deallocated
      #6 (ref.0) : hudson.remoting.FastPipedOutputStream
        Created at Fri May 02 09:50:35 PDT 2014
      	at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:86)
      	at hudson.remoting.ExportTable.export(ExportTable.java:239)
      	at hudson.remoting.Channel.export(Channel.java:592)
      	at hudson.remoting.Pipe.writeObject(Pipe.java:158)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:988)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1495)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
      	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
      	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
      	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
      	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
      	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
      	at hudson.remoting.UserRequest._serialize(UserRequest.java:155)
      	at hudson.remoting.UserRequest.serialize(UserRequest.java:164)
      	at hudson.remoting.UserRequest.<init>(UserRequest.java:62)
      	at hudson.remoting.Channel.callAsync(Channel.java:766)
      	at hudson.FilePath.read(FilePath.java:1640)
      	at hudson.FilePath.readToString(FilePath.java:1662)
      	at hudson.tools.JDKInstaller.performInstallation(JDKInstaller.java:126)
      	at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:61)
      	at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:107)
      	at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:204)
      	at hudson.model.JDK.forNode(JDK.java:126)
      	at org.jenkinsci.plugins.envinject.service.EnvInjectVariableGetter.getBuildVariables(EnvInjectVariableGetter.java:79)
      	at org.jenkinsci.plugins.envinject.EnvInjectListener.setUpEnvironmentWithoutJobPropertyObject(EnvInjectListener.java:231)
      	at org.jenkinsci.plugins.envinject.EnvInjectListener.setUpEnvironment(EnvInjectListener.java:46)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.createLauncher(AbstractBuild.java:570)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:476)
      	at hudson.model.Run.execute(Run.java:1709)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:231)
        Released at Fri May 02 09:50:35 PDT 2014
      	at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115)
      	at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303)
      	at hudson.remoting.Channel.unexport(Channel.java:600)
      	at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:384)
      	at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:111)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      
      	at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:270)
      	... 5 more
      Caused by:   Released at Fri May 02 09:50:35 PDT 2014
      	at hudson.remoting.ExportTable$Entry.release(ExportTable.java:115)
      	at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:303)
      	at hudson.remoting.Channel.unexport(Channel.java:600)
      	at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:384)
      	at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:111)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:744)
      

          [JENKINS-22853] SEVERE: Trying to unexport an object that's already unexported

          Our Jenkins Master and Slave versions were running different versions of the JDK/JRE. (Specifically some slaves were 1.6!). After updating all slaves to 1.7 we are not seeing the Unexport problem any more we have even moved the master to OpenJDK 1.8 and it seems to be getting along just fine with Oracle JDK/JRE 1.7 on most slaves.

          We are now getting hammered by a different mega stack trace: MalformedURLException: no protocol:

          But the unexport exceptions are gone.

          Steven Christenson added a comment - Our Jenkins Master and Slave versions were running different versions of the JDK/JRE. (Specifically some slaves were 1.6!). After updating all slaves to 1.7 we are not seeing the Unexport problem any more we have even moved the master to OpenJDK 1.8 and it seems to be getting along just fine with Oracle JDK/JRE 1.7 on most slaves. We are now getting hammered by a different mega stack trace: MalformedURLException: no protocol: But the unexport exceptions are gone.

          Andy Dreyfuss added a comment -

          Is matching java versions between master/slave a prerequisite or expectation for full support?

          Andy Dreyfuss added a comment - Is matching java versions between master/slave a prerequisite or expectation for full support?

          Kevin Bube added a comment -

          Most of our slaves are also on Java 7, some are Java 8.

          Kevin Bube added a comment - Most of our slaves are also on Java 7, some are Java 8.

          Tatsuya Sato added a comment -

          I'm running into the same issue on Jenkins 2.9 and OpenJDK 1.8.0_91, not master/slave.

          Jenkins's log

          When I tried to do something with Jenkins CLI, the logs of this instance says:

          /var/log/jenkins/jenkins.log
          Jun 20, 2016 1:24:05 PM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException
          SEVERE: A thread (Thread-8/63) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
          org.acegisecurity.userdetails.UsernameNotFoundException: admin
                  at hudson.security.SecurityRealm$None$2.loadUserByUsername(SecurityRealm.java:549)
                  at jenkins.security.ImpersonatingUserDetailsService.loadUserByUsername(ImpersonatingUserDetailsService.java:32)
                  at hudson.model.User.impersonate(User.java:311)
                  at org.jenkinsci.main.modules.cli.auth.ssh.SshCliAuthenticator.authenticate(SshCliAuthenticator.java:44)
                  at hudson.cli.CliManagerImpl$2.run(CliManagerImpl.java:109)
          
          Jun 20, 2016 1:24:05 PM hudson.remoting.ExportTable unexportByOid
          SEVERE: Trying to unexport an object that's already unexported
          java.lang.IllegalStateException: Invalid object ID 4 iota=5
                  at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:386)
                  at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:409)
                  at hudson.remoting.Channel.unexport(Channel.java:641)
                  at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:366)
                  at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158)
                  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
                  at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
                  at hudson.cli.CliManagerImpl$1.call(CliManagerImpl.java:63)
                  at hudson.remoting.CallableDecoratorAdapter.call(CallableDecoratorAdapter.java:18)
                  at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
                  at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                  at java.lang.Thread.run(Thread.java:745)
          
          Jun 20, 2016 1:24:05 PM hudson.remoting.ExportTable unexportByOid
          SEVERE: 2nd unexport attempt is here
          Command ProxyOutputStream.EOF(4) created at
                  at hudson.remoting.Command.<init>(Command.java:67)
                  at hudson.remoting.Command.<init>(Command.java:50)
                  at hudson.remoting.ProxyOutputStream$EOF.<init>(ProxyOutputStream.java:355)
                  at hudson.remoting.ProxyOutputStream.doClose(ProxyOutputStream.java:171)
                  at hudson.remoting.ProxyOutputStream.error(ProxyOutputStream.java:166)
                  at hudson.remoting.ProxyOutputStream.close(ProxyOutputStream.java:157)
                  at hudson.cli.Connection.close(Connection.java:265)
                  at hudson.cli.CLI.authenticate(CLI.java:575)
                  at hudson.cli.CLI._main(CLI.java:478)
                  at hudson.cli.CLI.main(
          

          Situation

          I provisioned Jenkins into CentOS 7 with ansible, based on this codes. The provisioned instance loads some groovy scripts in booting it.

          createAdmin.groovy
          import jenkins.*
          import jenkins.model.*
          import hudson.*
          import hudson.model.*
          
          def userId = 'admin'
          def jenkinUserKey = "SSH public key appears here"
          
          admin = User.get(userId)
          def keysProperty = admin.getProperty(org.jenkinsci.main.modules.cli.auth.ssh.UserPropertyImpl)
          if (keysProperty == null) {
            keysProperty = new org.jenkinsci.main.modules.cli.auth.ssh.UserPropertyImpl("")
          }
          
          println(keysProperty.authorizedKeys)
          if (!keysProperty.authorizedKeys.contains(jenkinUserKey)) {
            keysProperty.authorizedKeys = jenkinUserKey + "\n" + keysProperty.authorizedKeys
          
            admin.addProperty(keysProperty)
            admin.save()
          
            println("Added SSH key for admin")
          } else {
            println("SSH key was already added for admin")
          }
          

          And one of the init scripts sets the configuration so that slave Agent's port is choosen randomly.

          slaveAgentPort.groovy
          import jenkins.*
          import jenkins.model.*
          import hudson.*
          import hudson.model.*
          
          Jenkins.instance.slaveAgentPort = 0
          

          When I provisioned the instance of Jenkins 2.8, I had never seen this issue. so I will try out Jenkins 2.8 again.

          Tatsuya Sato added a comment - I'm running into the same issue on Jenkins 2.9 and OpenJDK 1.8.0_91, not master/slave. Jenkins's log When I tried to do something with Jenkins CLI, the logs of this instance says: /var/log/jenkins/jenkins.log Jun 20, 2016 1:24:05 PM hudson.init.impl.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler uncaughtException SEVERE: A thread ( Thread -8/63) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code. org.acegisecurity.userdetails.UsernameNotFoundException: admin at hudson.security.SecurityRealm$None$2.loadUserByUsername(SecurityRealm.java:549) at jenkins.security.ImpersonatingUserDetailsService.loadUserByUsername(ImpersonatingUserDetailsService.java:32) at hudson.model.User.impersonate(User.java:311) at org.jenkinsci.main.modules.cli.auth.ssh.SshCliAuthenticator.authenticate(SshCliAuthenticator.java:44) at hudson.cli.CliManagerImpl$2.run(CliManagerImpl.java:109) Jun 20, 2016 1:24:05 PM hudson.remoting.ExportTable unexportByOid SEVERE: Trying to unexport an object that's already unexported java.lang.IllegalStateException: Invalid object ID 4 iota=5 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:386) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:409) at hudson.remoting.Channel.unexport(Channel.java:641) at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:366) at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at hudson.cli.CliManagerImpl$1.call(CliManagerImpl.java:63) at hudson.remoting.CallableDecoratorAdapter.call(CallableDecoratorAdapter.java:18) at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:745) Jun 20, 2016 1:24:05 PM hudson.remoting.ExportTable unexportByOid SEVERE: 2nd unexport attempt is here Command ProxyOutputStream.EOF(4) created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.ProxyOutputStream$EOF.<init>(ProxyOutputStream.java:355) at hudson.remoting.ProxyOutputStream.doClose(ProxyOutputStream.java:171) at hudson.remoting.ProxyOutputStream.error(ProxyOutputStream.java:166) at hudson.remoting.ProxyOutputStream.close(ProxyOutputStream.java:157) at hudson.cli.Connection.close(Connection.java:265) at hudson.cli.CLI.authenticate(CLI.java:575) at hudson.cli.CLI._main(CLI.java:478) at hudson.cli.CLI.main( Situation I provisioned Jenkins into CentOS 7 with ansible, based on this codes . The provisioned instance loads some groovy scripts in booting it. createAdmin.groovy import jenkins.* import jenkins.model.* import hudson.* import hudson.model.* def userId = 'admin' def jenkinUserKey = "SSH public key appears here" admin = User.get(userId) def keysProperty = admin.getProperty(org.jenkinsci.main.modules.cli.auth.ssh.UserPropertyImpl) if (keysProperty == null ) { keysProperty = new org.jenkinsci.main.modules.cli.auth.ssh.UserPropertyImpl("") } println(keysProperty.authorizedKeys) if (!keysProperty.authorizedKeys.contains(jenkinUserKey)) { keysProperty.authorizedKeys = jenkinUserKey + "\n" + keysProperty.authorizedKeys admin.addProperty(keysProperty) admin.save() println( "Added SSH key for admin" ) } else { println( "SSH key was already added for admin" ) } And one of the init scripts sets the configuration so that slave Agent's port is choosen randomly. slaveAgentPort.groovy import jenkins.* import jenkins.model.* import hudson.* import hudson.model.* Jenkins.instance.slaveAgentPort = 0 When I provisioned the instance of Jenkins 2.8, I had never seen this issue. so I will try out Jenkins 2.8 again.

          Vin v added a comment -

          Had the same problem on a Windows slave machine. Updated the slave.jar to 2.43 and the issue was resolved.

          Vin v added a comment - Had the same problem on a Windows slave machine. Updated the slave.jar to 2.43 and the issue was resolved.

          Oleg Nenashev added a comment -

          From what I see, the issue happens because Jenkins tries to unexport ProxyInputStream objects from two sides.

          My repro steps:
          1) Create a test slave with 20 executors
          2) Run the Pipeline script below
          3) Interrupt build once it starts performing short stashes

          // The map we'll store the parallel steps in before executing hem.
          def stepsForParallel = [:]
          
          stage "Prepare stashed file"
          node("test") {
              sh "cp /Users/nenashev/Documents/jenkins/demo/jenkins-2.7.1/jenkins.war ."
              for (int i=0; i<10; i++) {
                  sh "cp jenkins.war jenkins_${i}.war"
              }
              writeFile file: 'foo.txt', text: 'Hello'
              stash includes: 'foo.txt', name: 'foo_stashed'
              stash includes: '*', name: 'jenkins_stashed'
          }
          
          
          stage "Parallel unstash"
          for (int i = 0; i < 19; i++) {
              def step = {
                  node("test") {
                      ws("ws_$i") {
                          unstash includes: '*', name: 'jenkins_stashed'
                      }
                  }
              }
              stepsForParallel["unstash_$i"] = step
          }
              
          stepsForParallel["fast_unstash"] = {
              node("test") {
                  ws("ws_fast") {
                      for (int i=0; i<2000; i++) {
                          unstash includes: '*', name: 'foo_stashed'
                      }
                  }
              }
          }
          
          parallel stepsForParallel
          

          Side 1. Remote call releases exports at the end of the call. E.g. if it gets interrupted

          Released at Thu Aug 04 13:12:16 MSK 2016
              	at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131)
              	at hudson.remoting.ExportTable$ExportList.release(ExportTable.java:247)
              	at hudson.remoting.UserRequest.releaseExports(UserRequest.java:224)
              	at hudson.remoting.UserRequest.releaseExports(UserRequest.java:220)
              	at hudson.remoting.Channel.call(Channel.java:798)
              	at hudson.FilePath.act(FilePath.java:1007)
              	at hudson.FilePath.act(FilePath.java:996)
              	at hudson.FilePath.untarFrom(FilePath.java:728)
              	at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:114)
          

          Side 2. When ProxyInputStream gets closed, it sends asynchronous EOF command to the remote side. This EOF also tries to release the allocated object (FileInputStream in my case). By the time the request gets to the remote executor, UserRequest.releaseExports() unexports the object.

          ProxyInputStream is nested into RemoteInputStream, which is actually being used for many cases like stdin/stdout propagation in remote calls. So technically any remote call operation may overlap with UserRequest termination handler and cause an overlap in such case.

          So the solution would be to either make ProxyInputStream EOF command synchronous or to make it tolerant against race conditions.

          Reproducing the issue in tests

          Oleg Nenashev added a comment - From what I see, the issue happens because Jenkins tries to unexport ProxyInputStream objects from two sides. My repro steps: 1) Create a test slave with 20 executors 2) Run the Pipeline script below 3) Interrupt build once it starts performing short stashes // The map we'll store the parallel steps in before executing hem. def stepsForParallel = [:] stage "Prepare stashed file" node("test") { sh "cp /Users/nenashev/Documents/jenkins/demo/jenkins-2.7.1/jenkins.war ." for (int i=0; i<10; i++) { sh "cp jenkins.war jenkins_${i}.war" } writeFile file: 'foo.txt', text: 'Hello' stash includes: 'foo.txt', name: 'foo_stashed' stash includes: '*', name: 'jenkins_stashed' } stage "Parallel unstash" for (int i = 0; i < 19; i++) { def step = { node("test") { ws("ws_$i") { unstash includes: '*', name: 'jenkins_stashed' } } } stepsForParallel["unstash_$i"] = step } stepsForParallel["fast_unstash"] = { node("test") { ws("ws_fast") { for (int i=0; i<2000; i++) { unstash includes: '*', name: 'foo_stashed' } } } } parallel stepsForParallel Side 1. Remote call releases exports at the end of the call. E.g. if it gets interrupted Released at Thu Aug 04 13:12:16 MSK 2016 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) at hudson.remoting.ExportTable$ExportList.release(ExportTable.java:247) at hudson.remoting.UserRequest.releaseExports(UserRequest.java:224) at hudson.remoting.UserRequest.releaseExports(UserRequest.java:220) at hudson.remoting.Channel.call(Channel.java:798) at hudson.FilePath.act(FilePath.java:1007) at hudson.FilePath.act(FilePath.java:996) at hudson.FilePath.untarFrom(FilePath.java:728) at org.jenkinsci.plugins.workflow.flow.StashManager.unstash(StashManager.java:114) Side 2. When ProxyInputStream gets closed, it sends asynchronous EOF command to the remote side. This EOF also tries to release the allocated object (FileInputStream in my case). By the time the request gets to the remote executor, UserRequest.releaseExports() unexports the object. ProxyInputStream is nested into RemoteInputStream, which is actually being used for many cases like stdin/stdout propagation in remote calls. So technically any remote call operation may overlap with UserRequest termination handler and cause an overlap in such case. So the solution would be to either make ProxyInputStream EOF command synchronous or to make it tolerant against race conditions. Reproducing the issue in tests

          Oleg Nenashev added a comment -

          The fix has been release in remoting-2.62 and jenkins-2.17

          Oleg Nenashev added a comment - The fix has been release in remoting-2.62 and jenkins-2.17

          This fix will be part of 2.19.1 LTS naturally.

          Oliver Gondža added a comment - This fix will be part of 2.19.1 LTS naturally.

          Code changed in jenkins
          User: Bertrand Roussel
          Path:
          client/pom.xml
          http://jenkins-ci.org/commit/swarm-plugin/8ad209aaafad5ec6256aa547702870da083ff6cd
          Log:
          Upgrade remoting API to 2.62

          Version 2.62 contains a fix for the following bugs:

          This occurs mostly with Java 8.

          The remoting version matches with Jenkins 2.18/2.19/2.19.1/2.19.2.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Bertrand Roussel Path: client/pom.xml http://jenkins-ci.org/commit/swarm-plugin/8ad209aaafad5ec6256aa547702870da083ff6cd Log: Upgrade remoting API to 2.62 Version 2.62 contains a fix for the following bugs: https://issues.jenkins-ci.org/browse/JENKINS-22853 https://issues.jenkins-ci.org/browse/JENKINS-23271 (possibly) This occurs mostly with Java 8. The remoting version matches with Jenkins 2.18/2.19/2.19.1/2.19.2.

          bbonn added a comment -

          HI There,

          We are still seeing this on Jenkins ver. 2.7.1. Is there additional things we can do to troubleshoot. We have also been crashing with permgen errors since upgrade, not sure if related. Please advise if i should reopen this issue. Thanks!

          SEVERE: Trying to unexport an object that's already unexported
          java.lang.IllegalStateException: Invalid object ID 412 iota=773
          at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:386)
          at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:409)
          at hudson.remoting.Channel.unexport(Channel.java:641)
          at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43)
          at hudson.remoting.Channel$1.handle(Channel.java:501)
          at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
          Caused by: java.lang.Exception: Object was recently deallocated
          #412 (ref.0) : object=null type=hudson.remoting.FastPipedOutputStream interfaces=[java.lang.Object]
          Created at Thu Feb 02 09:24:16 EST 2017
          at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:99)
          at hudson.remoting.ExportTable.export(ExportTable.java:305)
          at hudson.remoting.Channel.internalExport(Channel.java:629)
          at hudson.remoting.Pipe.writeObject(Pipe.java:158)
          at sun.reflect.GeneratedMethodAccessor1141.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at java.io.ObjectStreamClass.invokeWriteObject(Unknown Source)
          at java.io.ObjectOutputStream.writeSerialData(Unknown Source)
          at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)
          at java.io.ObjectOutputStream.writeObject0(Unknown Source)
          at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source)
          at java.io.ObjectOutputStream.writeSerialData(Unknown Source)
          at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)
          at java.io.ObjectOutputStream.writeObject0(Unknown Source)
          at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source)
          at java.io.ObjectOutputStream.writeSerialData(Unknown Source)
          at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source)
          at java.io.ObjectOutputStream.writeObject0(Unknown Source)
          at java.io.ObjectOutputStream.writeObject(Unknown Source)
          at hudson.remoting.UserRequest._serialize(UserRequest.java:189)
          at hudson.remoting.UserRequest.serialize(UserRequest.java:198)
          at hudson.remoting.UserRequest.<init>(UserRequest.java:64)
          at hudson.remoting.Channel.callAsync(Channel.java:807)
          at hudson.FilePath.actAsync(FilePath.java:1087)
          at hudson.FilePath.read(FilePath.java:1782)
          at hudson.plugins.groovy.FileScriptSource.getScriptStream(FileScriptSource.java:52)
          at hudson.plugins.groovy.SystemGroovy.perform(SystemGroovy.java:93)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
          at hudson.model.Build$BuildExecution.build(Build.java:205)
          at hudson.model.Build$BuildExecution.doRun(Build.java:162)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
          at hudson.model.Run.execute(Run.java:1741)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:410)
          Released at Thu Feb 02 09:24:16 EST 2017
          at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131)
          at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414)
          at hudson.remoting.Channel.unexport(Channel.java:641)
          at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:366)
          at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158)
          at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
          at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
          at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
          at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.lang.Thread.run(Unknown Source)
          Caused by: Command ProxyOutputStream.EOF(412) created at
          at hudson.remoting.Command.<init>(Command.java:67)
          at hudson.remoting.Command.<init>(Command.java:50)
          at hudson.remoting.ProxyOutputStream$EOF.<init>(ProxyOutputStream.java:358)
          at hudson.remoting.ProxyOutputStream.doClose(ProxyOutputStream.java:177)
          at hudson.remoting.ProxyOutputStream.close(ProxyOutputStream.java:173)
          at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:303)
          at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:274)
          at hudson.FilePath$33.invoke(FilePath.java:1795)
          at hudson.FilePath$33.invoke(FilePath.java:1782)
          at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772)
          at hudson.remoting.UserRequest.perform(UserRequest.java:118)
          at hudson.remoting.UserRequest.perform(UserRequest.java:48)
          at hudson.remoting.Request$2.run(Request.java:328)
          at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
          at java.util.concurrent.FutureTask.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1$1.run(Engine.java:63)
          ... 1 more
          at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:379)

          bbonn added a comment - HI There, We are still seeing this on Jenkins ver. 2.7.1. Is there additional things we can do to troubleshoot. We have also been crashing with permgen errors since upgrade, not sure if related. Please advise if i should reopen this issue. Thanks! SEVERE: Trying to unexport an object that's already unexported java.lang.IllegalStateException: Invalid object ID 412 iota=773 at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:386) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:409) at hudson.remoting.Channel.unexport(Channel.java:641) at hudson.remoting.UnexportCommand.execute(UnexportCommand.java:43) at hudson.remoting.Channel$1.handle(Channel.java:501) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60) Caused by: java.lang.Exception: Object was recently deallocated #412 (ref.0) : object=null type=hudson.remoting.FastPipedOutputStream interfaces= [java.lang.Object] Created at Thu Feb 02 09:24:16 EST 2017 at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:99) at hudson.remoting.ExportTable.export(ExportTable.java:305) at hudson.remoting.Channel.internalExport(Channel.java:629) at hudson.remoting.Pipe.writeObject(Pipe.java:158) at sun.reflect.GeneratedMethodAccessor1141.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at java.io.ObjectStreamClass.invokeWriteObject(Unknown Source) at java.io.ObjectOutputStream.writeSerialData(Unknown Source) at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source) at java.io.ObjectOutputStream.writeSerialData(Unknown Source) at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.defaultWriteFields(Unknown Source) at java.io.ObjectOutputStream.writeSerialData(Unknown Source) at java.io.ObjectOutputStream.writeOrdinaryObject(Unknown Source) at java.io.ObjectOutputStream.writeObject0(Unknown Source) at java.io.ObjectOutputStream.writeObject(Unknown Source) at hudson.remoting.UserRequest._serialize(UserRequest.java:189) at hudson.remoting.UserRequest.serialize(UserRequest.java:198) at hudson.remoting.UserRequest.<init>(UserRequest.java:64) at hudson.remoting.Channel.callAsync(Channel.java:807) at hudson.FilePath.actAsync(FilePath.java:1087) at hudson.FilePath.read(FilePath.java:1782) at hudson.plugins.groovy.FileScriptSource.getScriptStream(FileScriptSource.java:52) at hudson.plugins.groovy.SystemGroovy.perform(SystemGroovy.java:93) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779) at hudson.model.Build$BuildExecution.build(Build.java:205) at hudson.model.Build$BuildExecution.doRun(Build.java:162) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534) at hudson.model.Run.execute(Run.java:1741) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:410) Released at Thu Feb 02 09:24:16 EST 2017 at hudson.remoting.ExportTable$Entry.release(ExportTable.java:131) at hudson.remoting.ExportTable.unexportByOid(ExportTable.java:414) at hudson.remoting.Channel.unexport(Channel.java:641) at hudson.remoting.ProxyOutputStream$EOF$1.run(ProxyOutputStream.java:366) at hudson.remoting.PipeWriter$1.run(PipeWriter.java:158) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18) at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: Command ProxyOutputStream.EOF(412) created at at hudson.remoting.Command.<init>(Command.java:67) at hudson.remoting.Command.<init>(Command.java:50) at hudson.remoting.ProxyOutputStream$EOF.<init>(ProxyOutputStream.java:358) at hudson.remoting.ProxyOutputStream.doClose(ProxyOutputStream.java:177) at hudson.remoting.ProxyOutputStream.close(ProxyOutputStream.java:173) at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:303) at org.apache.commons.io.IOUtils.closeQuietly(IOUtils.java:274) at hudson.FilePath$33.invoke(FilePath.java:1795) at hudson.FilePath$33.invoke(FilePath.java:1782) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2772) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:328) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1$1.run(Engine.java:63) ... 1 more at hudson.remoting.ExportTable.diagnoseInvalidId(ExportTable.java:379)

            oleg_nenashev Oleg Nenashev
            cg Christian Goetze
            Votes:
            62 Vote for this issue
            Watchers:
            86 Start watching this issue

              Created:
              Updated:
              Resolved: