-
Improvement
-
Resolution: Fixed
-
Minor
-
None
-
Jenkins 2.440.2
analysis-mode-api 12.1.0
forensics-api 2.4.0
git-forensics 2.1.0
warnings-ng 11.2.2
Agent running on Windows
In one project, the recordIssues step often fails with "hudson.remoting.ProxyException: java.nio.channels.ClosedChannelException", and the stack trace includes "io.jenkins.plugins.forensics.git.delta.GitDeltaCalculator.calculateDelta". I suspect the agent runs out of memory while it's trying to compare files in the Git history, and the error prevents it from responding correctly to the remote call from the Jenkins controller. The error does not terminate the agent Java process. I haven't yet found the out-of-memory error in any logs.
I'd like a way to reduce the memory consumption by disabling the analysis of modified files. However, I'd like to keep using git-forensics-plugin for choosing the reference build; that doesn't need so much memory.
- links to
[JENKINS-72973] Skip delta computation to reduce memory consumption of recordIssues step
Kalle Niemitalo
created issue -
Kalle Niemitalo
made changes -
| Description |
Original:
In one project, the recordIssues step often fails with "hudson.remoting.ProxyException: java.nio.channels.ClosedChannelException", and the stack trace includes "io.jenkins.plugins.forensics.git.delta.GitDeltaCalculator.calculateDelta". I suspect the agent runs out of memory while it's trying to compare files in the Git history, terminates, and is automatically restarted, but the recordIssues step does not recover. I haven't yet found the out-of-memory error in any logs, though.
I'd like a way to reduce the memory consumption by disabling the analysis of modified files. However, I'd like to keep using git-forensics-plugin for choosing the reference build; that doesn't need so much memory. |
New:
In one project, the recordIssues step often fails with "hudson.remoting.ProxyException: java.nio.channels.ClosedChannelException", and the stack trace includes "io.jenkins.plugins.forensics.git.delta.GitDeltaCalculator.calculateDelta". I suspect the agent runs out of memory while it's trying to compare files in the Git history, and the error prevents it from responding correctly to the remote call from the Jenkins controller. The error does not terminate the agent Java process. I haven't yet found the out-of-memory error in any logs.
I'd like a way to reduce the memory consumption by disabling the analysis of modified files. However, I'd like to keep using git-forensics-plugin for choosing the reference build; that doesn't need so much memory. |
As an experiment, I terminated the agent Java process. This caused the WinSW process (jenkins-agent.exe) to terminate as well, and the Service Control Manager of Windows logged this to the "System" event log. However, when I look at the "System" event log around the time stamps of the ClosedChannelException errors, there are no log entries about the agent service stopping. Thus, those errors cannot have terminated the agent Java process.
I'm thinking about adding a Groovy hook script that removes the GitDeltaCalculatorFactory extension. That could be a quick workaround and likely to remain compatible with future plugin versions.
Kalle Niemitalo
added a comment - - edited As an experiment, I terminated the agent Java process. This caused the WinSW process (jenkins-agent.exe) to terminate as well, and the Service Control Manager of Windows logged this to the "System" event log. However, when I look at the "System" event log around the time stamps of the ClosedChannelException errors, there are no log entries about the agent service stopping. Thus, those errors cannot have terminated the agent Java process.
I'm thinking about adding a Groovy hook script that removes the GitDeltaCalculatorFactory extension. That could be a quick workaround and likely to remain compatible with future plugin versions. Like this perhaps
import jenkins.model.Jenkins; import io.jenkins.plugins.forensics.delta.DeltaCalculatorFactory; import io.jenkins.plugins.forensics.git.delta.GitDeltaCalculatorFactory; def factories = Jenkins.get().getExtensionList(DeltaCalculatorFactory.class); def gitFactory = factories.get(GitDeltaCalculatorFactory.class); if (gitFactory != null) { println("Removing ${gitFactory} to work around https://issues.jenkins.io/browse/JENKINS-72973"); factories.remove(gitFactory); }
Kalle Niemitalo
added a comment - - edited Like this perhaps
import jenkins.model.Jenkins;
import io.jenkins.plugins.forensics.delta.DeltaCalculatorFactory;
import io.jenkins.plugins.forensics.git.delta.GitDeltaCalculatorFactory;
def factories = Jenkins.get().getExtensionList(DeltaCalculatorFactory.class);
def gitFactory = factories.get(GitDeltaCalculatorFactory.class);
if (gitFactory != null ) {
println( "Removing ${gitFactory} to work around https: //issues.jenkins.io/browse/JENKINS-72973" );
factories.remove(gitFactory);
}
It makes sense to use the existing skipPostProcessing flag for this part as well.
Kalle Niemitalo
added a comment - I see, skipPostProcessing is "Disable detection of missing package and module names" in the snippet generator UI. I don't think we've had any out-of-memory problems with that feature, and I'd like to keep it enabled.
In my opinion, if an existing parameter is made to skip the delta computation, then the skipBlames parameter should be used for that.
Kalle Niemitalo
added a comment - I see, skipPostProcessing is "Disable detection of missing package and module names" in the snippet generator UI. I don't think we've had any out-of-memory problems with that feature, and I'd like to keep it enabled.
In my opinion, if an existing parameter is made to skip the delta computation, then the skipBlames parameter should be used for that. Kalle Niemitalo
added a comment - A hypothesis came to mind: is it possible that the ClosedChannelException error is not caused by memory consumption, but instead by unusual characters in the FileChanges.fileContent string? However, when I look at https://github.com/jenkinsci/remoting/, it seems to be using java.io.ObjectOutputStream, whose documentation does not mention such a limitation.
Kalle Niemitalo
added a comment - A hypothesis came to mind: is it possible that the ClosedChannelException error is not caused by memory consumption, but instead by unusual characters in the FileChanges.fileContent string? However, when I look at https://github.com/jenkinsci/remoting/ , it seems to be using java.io.ObjectOutputStream, whose documentation does not mention such a limitation. Kalle Niemitalo
added a comment - The last time the ClosedChannelException occurred, the Jenkins build was for a pull request that included changes to a 20-megabyte XML file and to several binary files (some of which are in Git LFS and others not). Perhaps these caused excessive memory consumption in jgit, even though the output of "git diff" for the XML file is only four kilobytes long. I don't think any of the modified large files had any warnings, but the DeltaCalculator API does not support passing in a list of relevant files, so git-forensics-plugin ends up comparing all files anyway.
Kalle Niemitalo
added a comment - The last time the ClosedChannelException occurred, the Jenkins build was for a pull request that included changes to a 20-megabyte XML file and to several binary files (some of which are in Git LFS and others not). Perhaps these caused excessive memory consumption in jgit, even though the output of "git diff" for the XML file is only four kilobytes long. I don't think any of the modified large files had any warnings, but the DeltaCalculator API does not support passing in a list of relevant files, so git-forensics-plugin ends up comparing all files anyway. | Status | Original: Open [ 1 ] | New: In Progress [ 3 ] |
The stack trace in the build log: