Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72973

Skip delta computation to reduce memory consumption of recordIssues step

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Minor Minor
    • warnings-ng-plugin
    • None
    • Jenkins 2.440.2
      analysis-mode-api 12.1.0
      forensics-api 2.4.0
      git-forensics 2.1.0
      warnings-ng 11.2.2
      Agent running on Windows

      In one project, the recordIssues step often fails with "hudson.remoting.ProxyException: java.nio.channels.ClosedChannelException", and the stack trace includes "io.jenkins.plugins.forensics.git.delta.GitDeltaCalculator.calculateDelta". I suspect the agent runs out of memory while it's trying to compare files in the Git history, and the error prevents it from responding correctly to the remote call from the Jenkins controller. The error does not terminate the agent Java process. I haven't yet found the out-of-memory error in any logs.

      I'd like a way to reduce the memory consumption by disabling the analysis of modified files. However, I'd like to keep using git-forensics-plugin for choosing the reference build; that doesn't need so much memory.

          [JENKINS-72973] Skip delta computation to reduce memory consumption of recordIssues step

          The stack trace in the build log:

          hudson.remoting.ProxyException: java.nio.channels.ClosedChannelException
          Also:   hudson.remoting.ProxyException: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from [REDACTED]
          		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
          		at hudson.remoting.Request.call(Request.java:199)
          		at hudson.remoting.Channel.call(Channel.java:1002)
          		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
          		at jdk.proxy84/jdk.proxy84.$Proxy176.withRepository(Unknown Source)
          		at org.jenkinsci.plugins.gitclient.RemoteGitImpl.withRepository(RemoteGitImpl.java:261)
          		at io.jenkins.plugins.forensics.git.delta.GitDeltaCalculator.calculateDelta(GitDeltaCalculator.java:64)
          		at io.jenkins.plugins.forensics.delta.DeltaCalculator.calculateDelta(DeltaCalculator.java:34)
          		at io.jenkins.plugins.analysis.core.steps.IssuesPublisher.markIssuesInModifiedFiles(IssuesPublisher.java:162)
          		at io.jenkins.plugins.analysis.core.steps.IssuesPublisher.computeDelta(IssuesPublisher.java:148)
          		at io.jenkins.plugins.analysis.core.steps.IssuesPublisher.attachAction(IssuesPublisher.java:92)
          		at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.publishResult(IssuesRecorder.java:742)
          		at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.record(IssuesRecorder.java:659)
          		at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.perform(IssuesRecorder.java:621)
          		at io.jenkins.plugins.analysis.core.steps.RecordIssuesStep$Execution.run(RecordIssuesStep.java:631)
          		at io.jenkins.plugins.analysis.core.steps.RecordIssuesStep$Execution.run(RecordIssuesStep.java:591)
          		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
          		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 00027a98-c925-4431-84b7-f39982f4d71c
          Caused: hudson.remoting.ProxyException: hudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException
          	at hudson.remoting.Request.abort(Request.java:346)
          	at hudson.remoting.Channel.terminate(Channel.java:1083)
          	at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:241)
          	at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221)
          	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
          	at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:289)
          	at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:168)
          	at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825)
          	at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:155)
          	at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:193)
          	at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:789)
          	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
          	at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51)
          	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          	at java.base/java.lang.Thread.run(Thread.java:840)
          

          Kalle Niemitalo added a comment - The stack trace in the build log: hudson.remoting.ProxyException: java.nio.channels.ClosedChannelException Also: hudson.remoting.ProxyException: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from [REDACTED] at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787) at hudson.remoting.Request.call(Request.java:199) at hudson.remoting.Channel.call(Channel.java:1002) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285) at jdk.proxy84/jdk.proxy84.$Proxy176.withRepository(Unknown Source) at org.jenkinsci.plugins.gitclient.RemoteGitImpl.withRepository(RemoteGitImpl.java:261) at io.jenkins.plugins.forensics.git.delta.GitDeltaCalculator.calculateDelta(GitDeltaCalculator.java:64) at io.jenkins.plugins.forensics.delta.DeltaCalculator.calculateDelta(DeltaCalculator.java:34) at io.jenkins.plugins.analysis.core.steps.IssuesPublisher.markIssuesInModifiedFiles(IssuesPublisher.java:162) at io.jenkins.plugins.analysis.core.steps.IssuesPublisher.computeDelta(IssuesPublisher.java:148) at io.jenkins.plugins.analysis.core.steps.IssuesPublisher.attachAction(IssuesPublisher.java:92) at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.publishResult(IssuesRecorder.java:742) at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.record(IssuesRecorder.java:659) at io.jenkins.plugins.analysis.core.steps.IssuesRecorder.perform(IssuesRecorder.java:621) at io.jenkins.plugins.analysis.core.steps.RecordIssuesStep$Execution.run(RecordIssuesStep.java:631) at io.jenkins.plugins.analysis.core.steps.RecordIssuesStep$Execution.run(RecordIssuesStep.java:591) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) Also: org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 00027a98-c925-4431-84b7-f39982f4d71c Caused: hudson.remoting.ProxyException: hudson.remoting.RequestAbortedException: java.nio.channels.ClosedChannelException at hudson.remoting.Request.abort(Request.java:346) at hudson.remoting.Channel.terminate(Channel.java:1083) at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:241) at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:221) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:289) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:168) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:825) at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:155) at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:193) at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:789) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at jenkins.util.ErrorLoggingExecutorService.lambda$wrap$0(ErrorLoggingExecutorService.java:51) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)

          Kalle Niemitalo added a comment - - edited

          As an experiment, I terminated the agent Java process. This caused the WinSW process (jenkins-agent.exe) to terminate as well, and the Service Control Manager of Windows logged this to the "System" event log. However, when I look at the "System" event log around the time stamps of the ClosedChannelException errors, there are no log entries about the agent service stopping. Thus, those errors cannot have terminated the agent Java process.

          I'm thinking about adding a Groovy hook script that removes the GitDeltaCalculatorFactory extension. That could be a quick workaround and likely to remain compatible with future plugin versions.

          Kalle Niemitalo added a comment - - edited As an experiment, I terminated the agent Java process. This caused the WinSW process (jenkins-agent.exe) to terminate as well, and the Service Control Manager of Windows logged this to the "System" event log. However, when I look at the "System" event log around the time stamps of the ClosedChannelException errors, there are no log entries about the agent service stopping. Thus, those errors cannot have terminated the agent Java process. I'm thinking about adding a Groovy hook script that removes the GitDeltaCalculatorFactory extension. That could be a quick workaround and likely to remain compatible with future plugin versions.

          Kalle Niemitalo added a comment - - edited

          Like this perhaps

          import jenkins.model.Jenkins;
          import io.jenkins.plugins.forensics.delta.DeltaCalculatorFactory;
          import io.jenkins.plugins.forensics.git.delta.GitDeltaCalculatorFactory;
          
          def factories = Jenkins.get().getExtensionList(DeltaCalculatorFactory.class);
          def gitFactory = factories.get(GitDeltaCalculatorFactory.class);
          if (gitFactory != null) {
            println("Removing ${gitFactory} to work around https://issues.jenkins.io/browse/JENKINS-72973");
            factories.remove(gitFactory);
          }
          

          Kalle Niemitalo added a comment - - edited Like this perhaps import jenkins.model.Jenkins; import io.jenkins.plugins.forensics.delta.DeltaCalculatorFactory; import io.jenkins.plugins.forensics.git.delta.GitDeltaCalculatorFactory; def factories = Jenkins.get().getExtensionList(DeltaCalculatorFactory.class); def gitFactory = factories.get(GitDeltaCalculatorFactory.class); if (gitFactory != null ) { println( "Removing ${gitFactory} to work around https: //issues.jenkins.io/browse/JENKINS-72973" ); factories.remove(gitFactory); }

          Ulli Hafner added a comment -

          It makes sense to use the existing skipPostProcessing flag for this part as well.

          Ulli Hafner added a comment - It makes sense to use the existing skipPostProcessing flag for this part as well.

          I see, skipPostProcessing is "Disable detection of missing package and module names" in the snippet generator UI. I don't think we've had any out-of-memory problems with that feature, and I'd like to keep it enabled.

          In my opinion, if an existing parameter is made to skip the delta computation, then the skipBlames parameter should be used for that.

          Kalle Niemitalo added a comment - I see, skipPostProcessing is "Disable detection of missing package and module names" in the snippet generator UI. I don't think we've had any out-of-memory problems with that feature, and I'd like to keep it enabled. In my opinion, if an existing parameter is made to skip the delta computation, then the skipBlames parameter should be used for that.

          A hypothesis came to mind: is it possible that the ClosedChannelException error is not caused by memory consumption, but instead by unusual characters in the FileChanges.fileContent string? However, when I look at https://github.com/jenkinsci/remoting/, it seems to be using java.io.ObjectOutputStream, whose documentation does not mention such a limitation.

          Kalle Niemitalo added a comment - A hypothesis came to mind: is it possible that the ClosedChannelException error is not caused by memory consumption, but instead by unusual characters in the FileChanges.fileContent string? However, when I look at https://github.com/jenkinsci/remoting/ , it seems to be using java.io.ObjectOutputStream, whose documentation does not mention such a limitation.

          The last time the ClosedChannelException occurred, the Jenkins build was for a pull request that included changes to a 20-megabyte XML file and to several binary files (some of which are in Git LFS and others not). Perhaps these caused excessive memory consumption in jgit, even though the output of "git diff" for the XML file is only four kilobytes long.  I don't think any of the modified large files had any warnings, but the DeltaCalculator API does not support passing in a list of relevant files, so git-forensics-plugin ends up comparing all files anyway.

          Kalle Niemitalo added a comment - The last time the ClosedChannelException occurred, the Jenkins build was for a pull request that included changes to a 20-megabyte XML file and to several binary files (some of which are in Git LFS and others not). Perhaps these caused excessive memory consumption in jgit, even though the output of "git diff" for the XML file is only four kilobytes long.  I don't think any of the modified large files had any warnings, but the DeltaCalculator API does not support passing in a list of relevant files, so git-forensics-plugin ends up comparing all files anyway.

            drulli Ulli Hafner
            kon Kalle Niemitalo
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: