Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54078

[JEP-210] Exception: Object was recently deallocated when using Git + timeout(activity: true)

    • workflow-basic-steps 2.12

      When upgrading to Pipeline: Job 2.26 I get errors when combining some wrappers and steps:

      timeout{
          timestamps{
              checkout [$class: 'GitSCM' ...]
          }
      }
      

      Jenkins version: 2.138.2
      Pipeline: Groovy 2.58
      Timestamper 1.8.10
      Git plugin 3.9.1

      Released at Mon Oct 15 13:44:45 MSK 2018
      	at hudson.remoting.ExportTable$Entry.release(ExportTable.java:163)
      	at hudson.remoting.ExportTable$ExportList.release(ExportTable.java:300)
      	at hudson.remoting.UserRequest.releaseExports(UserRequest.java:299)
      	at hudson.remoting.UserRequest.releaseExports(UserRequest.java:295)
      	at hudson.remoting.Channel.call(Channel.java:970)
      	at hudson.FilePath.act(FilePath.java:1071)
      	at hudson.FilePath.act(FilePath.java:1060)
      	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:137)
      	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:795)
      	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:786)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1154)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
      	at hudson.security.ACL.impersonate(ACL.java:290)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      Caused: java.lang.Exception: Object was recently deallocated
          #17 (ref.0) : object=null type=org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$ResetCallbackImpl interfaces=[org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$ResetCallback]
            Created at Mon Oct 15 13:44:45 MSK 2018
          	at hudson.remoting.ExportTable$Entry.<init>(ExportTable.java:114)
          	at hudson.remoting.ExportTable.export(ExportTable.java:365)
          	at hudson.remoting.Channel.internalExport(Channel.java:776)
          	at hudson.remoting.Channel.export(Channel.java:767)
          	at hudson.remoting.Channel.export(Channel.java:731)
          	at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$ConsoleLogFilterImpl.writeReplace(TimeoutStepExecution.java:255)
          	at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1218)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136)
          	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
          	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
          	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
          	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
          	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
          	at java.util.ArrayList.writeObject(ArrayList.java:766)
          	at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
          	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
          	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
          	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
          	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
          	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
          	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
          	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
          	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
          	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
          	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
          	at hudson.remoting.UserRequest._serialize(UserRequest.java:264)
          	at hudson.remoting.UserRequest.serialize(UserRequest.java:273)
          	at hudson.remoting.UserRequest.<init>(UserRequest.java:101)
          	at hudson.remoting.Channel.call(Channel.java:953)
          	at hudson.FilePath.act(FilePath.java:1071)
          	at hudson.FilePath.act(FilePath.java:1060)
          	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:137)
          	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:795)
          	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:786)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1154)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
          	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
          	at hudson.security.ACL.impersonate(ACL.java:290)
          	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
            Released at Mon Oct 15 13:44:45 MSK 2018
          	at hudson.remoting.ExportTable$Entry.release(ExportTable.java:163)
          	at hudson.remoting.ExportTable$ExportList.release(ExportTable.java:300)
          	at hudson.remoting.UserRequest.releaseExports(UserRequest.java:299)
          	at hudson.remoting.UserRequest.releaseExports(UserRequest.java:295)
          	at hudson.remoting.Channel.call(Channel.java:970)
          	at hudson.FilePath.act(FilePath.java:1071)
          	at hudson.FilePath.act(FilePath.java:1060)
          	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:137)
          	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:795)
          	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:786)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1154)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
          	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
          	at hudson.security.ACL.impersonate(ACL.java:290)
          	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
      	at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:468)
      Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to channel
      		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
      		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
      		at hudson.remoting.Channel.call(Channel.java:955)
      		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
      		at com.sun.proxy.$Proxy9.logWritten(Unknown Source)
      		at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$ConsoleLogFilterImpl$1.eol(TimeoutStepExecution.java:265)
      		at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
      		at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
      		at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:74)
      		at hudson.plugins.timestamper.TimestampNotesOutputStream.eol(TimestampNotesOutputStream.java:74)
      		at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
      		at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
      		at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:74)
      		at java.io.PrintStream.write(PrintStream.java:480)
      		at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
      		at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
      		at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
      		at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
      		at java.io.PrintStream.newLine(PrintStream.java:546)
      		at java.io.PrintStream.println(PrintStream.java:807)
      		at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:552)
      		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
      		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
      		at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      		at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      		at hudson.remoting.Request$2.run(Request.java:369)
      		at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      Caused: java.util.concurrent.ExecutionException: Invalid object ID 17 iota=19
      	at hudson.remoting.ExportTable.diagnoseInvalidObjectId(ExportTable.java:478)
      	at hudson.remoting.ExportTable.get(ExportTable.java:397)
      	at hudson.remoting.Channel.getExportedObject(Channel.java:780)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:920)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:903)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:855)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:19)
      	at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
      	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
      Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to stego02
      		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
      		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
      		at hudson.remoting.Channel.call(Channel.java:955)
      		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
      		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      		at java.lang.reflect.Method.invoke(Method.java:498)
      		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
      		at com.sun.proxy.$Proxy91.execute(Unknown Source)
      		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1120)
      		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1160)
      		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
      		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
      		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
      		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
      		at hudson.security.ACL.impersonate(ACL.java:290)
      		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
      		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      Caused: hudson.remoting.RemotingSystemException
      	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:299)
      	at com.sun.proxy.$Proxy9.logWritten(Unknown Source)
      	at org.jenkinsci.plugins.workflow.steps.TimeoutStepExecution$ConsoleLogFilterImpl$1.eol(TimeoutStepExecution.java:265)
      	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
      	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
      	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:74)
      	at hudson.plugins.timestamper.TimestampNotesOutputStream.eol(TimestampNotesOutputStream.java:74)
      	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:60)
      	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:56)
      	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:74)
      	at java.io.PrintStream.write(PrintStream.java:480)
      	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
      	at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
      	at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
      	at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
      	at java.io.PrintStream.newLine(PrintStream.java:546)
      	at java.io.PrintStream.println(PrintStream.java:807)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:552)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
      	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
      

          [JENKINS-54078] [JEP-210] Exception: Object was recently deallocated when using Git + timeout(activity: true)

          I won't be able to update the plugin so soon, so I can't reproduce it, but as a followup: 

          timeout(activity: true, time: 10, unit: 'SECONDS') {
          

          looks correct, as I had a similar configuration on the failing job:
          (only through shared libraries)

          script.timeout( time: timeout, unit: 'HOURS', activity: true ) {
            script.timestamps {
          
              ...
            }
          }

          Alexey Kiryanov added a comment - I won't be able to update the plugin so soon, so I can't reproduce it, but as a followup:  timeout(activity: true , time: 10, unit: 'SECONDS' ) { looks correct, as I had a similar configuration on the failing job: (only through shared libraries) script.timeout( time: timeout, unit: 'HOURS' , activity: true ) {   script.timestamps { ... } }

          Jesse Glick added a comment -

          The timestamper plugin is not involved. The trigger condition is using the Git plugin and timeout(activity: true, …).

          Jesse Glick added a comment - The timestamper plugin is not involved. The trigger condition is using the Git plugin and timeout(activity: true, …) .

          Jesse Glick added a comment -

          Here is a link to download my fix if you would like to validate it.

          Jesse Glick added a comment - Here is a link to download my fix if you would like to validate it.

          I can confirm your fix fixes the issue for me.

          Patrick Ruckstuhl added a comment - I can confirm your fix fixes the issue for me.

          After the troubles with huge number of "Computer.threadPoolForRemoting" threads I'm having as described on JENKINS-54073 I now rolled back to the latest released (2.11) in the hope of having this together with workflow-durable-task-step 2.25 it would go away but that's unfortunately not the case

          Patrick Ruckstuhl added a comment - After the troubles with huge number of "Computer.threadPoolForRemoting" threads I'm having as described on JENKINS-54073 I now rolled back to the latest released (2.11) in the hope of having this together with workflow-durable-task-step 2.25 it would go away but that's unfortunately not the case

          Figured out that I had to downgrade workflow-job from 2.26 to 2.25 as well, now I think the issue doesn't happen anymore.

          Patrick Ruckstuhl added a comment - Figured out that I had to downgrade workflow-job from 2.26 to 2.25 as well, now I think the issue doesn't happen anymore.

          Jesse Glick added a comment -

          tario I have a fix for your issue already, it is going through CI now.

          Jesse Glick added a comment - tario I have a fix for your issue already, it is going through CI now.

          Jesse Glick added a comment -

          tario if you are still in a position to try experimental builds, I think this update should correct your issue. activity: true as originally implemented (and in my fix from 2018-10-16) sent a distinct Remoting packet for every line of output, which made for substantial overhead in busy builds. I have now rewritten this to only send a “reset the timer please” message from the agent to the master at intervals of half of the idle timeout.

          There are still some fixes I plan to make in this PR, but that build ought to address the performance problem.

          Jesse Glick added a comment - tario if you are still in a position to try experimental builds, I think this update should correct your issue. activity: true as originally implemented (and in my fix from 2018-10-16) sent a distinct Remoting packet for every line of output, which made for substantial overhead in busy builds. I have now rewritten this to only send a “reset the timer please” message from the agent to the master at intervals of half of the idle timeout. There are still some fixes I plan to make in this PR, but that build ought to address the performance problem.

          Thanks for this, unfortunately I'm right now not in a position to try this experimental build out. Now with the down/upgrades (workflow-job 2.25 and workflow-durable-task-step 2.25) that you did it looks like things are running stable again.

          Patrick Ruckstuhl added a comment - Thanks for this, unfortunately I'm right now not in a position to try this experimental build out. Now with the down/upgrades (workflow-job 2.25 and workflow-durable-task-step 2.25) that you did it looks like things are running stable again.

          Devin Nusbaum added a comment -

          A fix for this was just released in version 2.12 of the Pipeline Basic Steps Plugin.

          Devin Nusbaum added a comment - A fix for this was just released in version 2.12 of the Pipeline Basic Steps Plugin.

            jglick Jesse Glick
            trueconf_kiryanov Alexey Kiryanov
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: