There are several ways to deal with this issue

• disable changelog calculation
• configure jenkins use 64-bit JVM with maximum heap size 2 GB or more
• split CVS repository to several smaller ones

I'm not sure 2 of this options are viable - it's not acceptable to expect people to split CVS repositories and polling requires the same functions as changelog generation so preventing changelogs may not fix issues. For users on 32 bit systems, the other option isn't possible either meaning this issue needs fixed properly.

I've encountered the same problem - it looks like there is a memory leak either in jenkins (1.465) or cvs plugin(2.4-SNAPSHOT (private-05/25/2012 14:52)). Restarting Jenkins fixed the problem, probably only temporarily.

java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding$StringDecoder.decode(Unknown Source)
at java.lang.StringCoding.decode(Unknown Source)
at java.lang.StringCoding.decode(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at java.io.ByteArrayOutputStream.toString(Unknown Source)
at hudson.scm.CVSSCM.getRemoteLogForModule(CVSSCM.java:540)
at hudson.scm.CVSSCM.calculateChangeLog(CVSSCM.java:415)
at hudson.scm.CVSSCM.checkout(CVSSCM.java:825)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1218)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:586)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:475)
at hudson.model.Run.run(Run.java:1434)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:239)

The issue is being caused by the CVS rlog command not having any filters set so returning headers for every file within the module. This will affect any repository with a lot of files in it (both active files and those in the attic) and is most likely to result in an out of memory error at the point the byte array the log is created in is converted to a String). I'm going to be committing a change tonight that sets filters so should prevent this issue.

Code changed in jenkins
User: mc1arke
Path:
src/main/java/hudson/scm/CVSSCM.java
src/main/java/hudson/scm/CvsChangeLogHelper.java
src/test/java/hudson/scm/CvsChangeLogHelperTest.java
http://jenkins-ci.org/commit/cvs-plugin/cd31307e33bc3749a71e5ad8d9f2a09cf3fd733d
Log:
[Fixed JENKINS-13227] change log and polling pick up branch changes properly
[Fixed JENKINS-13814] change log applies filers to prevent OOM Exceptions

Code changed in jenkins
User: mc1arke
Path:
src/test/resources/hudson/scm/cvsRlogOutput2.txt
http://jenkins-ci.org/commit/cvs-plugin/05e417c0c13367bda49761f9cab7618fdfba28ca
Log:
[Fixed JENKINS-13227] change log and polling pick up branch changes properly
[Fixed JENKINS-13814] change log applies filers to prevent OOM Exceptions

Fixed by re-applying previously set '-S' filter on rlog command

The fixes in the latest 2.4 Snapshot work for my build now. Thanks a lot!

I got the same issue with the 2.4 upgrade today. Can someone let me know where I could download a latest 2.4 (or 2.5) Snapshot version?

Thanks!

Gene: There haven't been any changes in the 2.5 Snapshot that would fix this (and I can't replicate the issue on my current test CVS Repsitory). Are you able to provide a stack trace/job log for your error so I can try and narrow down what's happening?

Michael:
I got one same error as "java.lang.OutOfMemoryError exception when getting the remote log" yesterday. I came to this page via a google search.
However this morning I tried to reproduce the error, unfortunately I did not get an error like that but the jenkins hangs there forever with cvs rlog ..... operation. I am now trying a test as Urban suggested. (use 64 bits jvm with 3G heap size)

I saw you have fixes in latest 2.4 snapshot already so I downloaded the source from github and tried to build. The version from github is 2.5-SNAPSHOT. My build failed with error as –

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4:37.557s
[INFO] Finished at: Tue Jun 19 15:31:53 EDT 2012
[INFO] Final Memory: 14M/318M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.jenkins-ci.tools:maven-hpi-plugin:1.64:apt-compile (default-apt-compile) on project cvs: Execution default-apt-compile of goal org.jenkins-ci.tools:maven-hpi-plugin:1.64:apt-compile failed: A required class was missing while executing org.jenkins-ci.tools:maven-hpi-plugin:1.64:apt-compile: com/sun/mirror/apt/AnnotationProcessorFactory
[ERROR] -----------------------------------------------------
[ERROR] realm = plugin>org.jenkins-ci.tools:maven-hpi-plugin:1.64
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy

Michael,

Environment -
OS: OS Linux unknown, i386/32 (4 cores)
Java: Java(TM) SE Runtime Environment, 1.6.0_31-b04
JVM: Java HotSpot(TM) Server VM, 20.6-b01, mixed mode
Jenkins ver. 1.424.6.10-SNAPSHOT
cvs plugin 2.4

Here is the job log--
10:04:07 cvs rlog: Logging nms/javasrc/com/xxxxxx/nms/server/mediator/generated/xxxxxaos6850/v6_4_4/ethernetoam
10:04:07 cvs rlog: Logging nms/javasrc/com/xxxxxx/nms/server/mediator/generated/xxxxxaos6850/v6_4_4/ethernetoam/imp
10:04:08 FATAL: Java heap space
10:04:08 java.lang.OutOfMemoryError: Java heap space
10:04:08 at java.util.Arrays.copyOf(Arrays.java:2786)
10:04:08 at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
10:04:08 at java.io.PrintStream.write(PrintStream.java:430)
10:04:08 at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
10:04:08 at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
10:04:08 at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85)
10:04:08 at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168)
10:04:08 at java.io.PrintStream.write(PrintStream.java:477)
10:04:08 at java.io.PrintStream.print(PrintStream.java:619)
10:04:08 at java.io.PrintStream.println(PrintStream.java:756)
10:04:08 at org.netbeans.lib.cvsclient.commandLine.BasicListener.messageSent(BasicListener.java:104)
10:04:08 at org.netbeans.lib.cvsclient.event.MessageEvent.fireEvent(MessageEvent.java:161)
10:04:08 at org.netbeans.lib.cvsclient.event.EventManager.fireCVSEvent(EventManager.java:170)
10:04:08 at org.netbeans.lib.cvsclient.response.MessageResponse.process(MessageResponse.java:104)
10:04:08 at org.netbeans.lib.cvsclient.Client.handleResponse(Client.java:648)
10:04:08 at org.netbeans.lib.cvsclient.Client.processRequests(Client.java:598)
10:04:08 at org.netbeans.lib.cvsclient.command.log.RlogCommand.execute(RlogCommand.java:402)
10:04:08 at org.netbeans.lib.cvsclient.Client.executeCommand(Client.java:710)
10:04:08 at hudson.scm.CVSSCM.getRemoteLogForModule(CVSSCM.java:524)
10:04:08 at hudson.scm.CVSSCM.calculateChangeLog(CVSSCM.java:415)
10:04:08 at hudson.scm.CVSSCM.checkout(CVSSCM.java:831)
10:04:08 at hudson.model.AbstractProject.checkout(AbstractProject.java:1193)
10:04:08 at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:565)
10:04:08 at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:453)
10:04:08 at hudson.model.Run.run(Run.java:1376)
10:04:08 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
10:04:08 at hudson.model.ResourceController.execute(ResourceController.java:88)
10:04:08 at hudson.model.Executor.run(Executor.java:175)

Michael: Thanks! You are right, I used jdk1.7 to build the plugin. The build succeeds with jdk1.6.

I backed to 1.6 for my other testing. So my question is –
Does the current 2.5-snapshot include your fix?

I do not know how to run RLOG command from a command line, could you please suggest? I could do that with 2.4 after my test currently on version 1.6

Thanks!

Gene: if you look back in the job log then you should find a statement of 'cvs rlog -S -d 00/00/00 00:00:00+0000<11/11/11 11:11:11+1111 moduleName' where the numbers are a correctly formatted timestamp and 'moduleName' is the name of the module currently being operated on. If you paste this command into a terminal then it should invoke your machine's CVS client to run rlog (you may want to change into the directory in your workspace that one of your modules is in so it picks up the correct authentication settings).

Michael: As you suggested I run the cvs rlog from command line and works normal. There are values which we consider them as private data. I am sorry that I can not past it here.

I will have a try of my newly build 2.5-SNAPSHOT version. I suppose that your fix has been checked into.

Installed 2.5-SNAPSHOT (build from the latest source), still get following error. Seems like the fix is not in yet.
-----------
11:23:51 FATAL: Java heap space
11:23:51 java.lang.OutOfMemoryError: Java heap space
11:23:51 at java.util.Arrays.copyOf(Arrays.java:2786)
11:23:51 at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
11:23:51 at java.io.PrintStream.write(PrintStream.java:430)
11:23:51 at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
11:23:51 at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
11:23:51 at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85)
11:23:51 at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168)
11:23:51 at java.io.PrintStream.write(PrintStream.java:477)
11:23:51 at java.io.PrintStream.print(PrintStream.java:619)
11:23:51 at java.io.PrintStream.println(PrintStream.java:756)
11:23:51 at org.netbeans.lib.cvsclient.commandLine.BasicListener.messageSent(BasicListener.java:104)
11:23:51 at org.netbeans.lib.cvsclient.event.MessageEvent.fireEvent(MessageEvent.java:161)
11:23:51 at org.netbeans.lib.cvsclient.event.EventManager.fireCVSEvent(EventManager.java:170)
11:23:51 at org.netbeans.lib.cvsclient.response.MessageResponse.process(MessageResponse.java:104)
11:23:51 at org.netbeans.lib.cvsclient.Client.handleResponse(Client.java:648)
11:23:51 at org.netbeans.lib.cvsclient.Client.processRequests(Client.java:598)
11:23:51 at org.netbeans.lib.cvsclient.command.log.RlogCommand.execute(RlogCommand.java:402)
11:23:51 at org.netbeans.lib.cvsclient.Client.executeCommand(Client.java:710)
11:23:51 at hudson.scm.CVSSCM.getRemoteLogForModule(CVSSCM.java:524)
11:23:51 at hudson.scm.CVSSCM.calculateChangeLog(CVSSCM.java:415)
11:23:51 at hudson.scm.CVSSCM.checkout(CVSSCM.java:834)
11:23:51 at hudson.model.AbstractProject.checkout(AbstractProject.java:1193)
11:23:51 at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:565)
11:23:51 at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:453)
11:23:51 at hudson.model.Run.run(Run.java:1376)
11:23:51 at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
11:23:51 at hudson.model.ResourceController.execute(ResourceController.java:88)
11:23:51 at hudson.model.Executor.run(Executor.java:175)

Michael: I have sent you the cvs rlog output via an email.

More inputs of cvs rlog stuck (for ever) –

Executed method:
java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3366)

Executor #1 for sambstage : executing MAIN_DEV #13
java.util.regex.Pattern$BmpCharProperty.match(Pattern.java:3366)
java.util.regex.Pattern$Curly.match(Pattern.java:3737)
java.util.regex.Pattern$GroupTail.match(Pattern.java:4227)
java.util.regex.Pattern$Curly.match1(Pattern.java:3797)
java.util.regex.Pattern$Curly.match(Pattern.java:3746)
java.util.regex.Pattern$GroupHead.match(Pattern.java:4168)
java.util.regex.Pattern$Curly.match0(Pattern.java:3782)
java.util.regex.Pattern$Curly.match(Pattern.java:3744)
java.util.regex.Pattern$Start.match(Pattern.java:3055)
java.util.regex.Matcher.search(Matcher.java:1105)
java.util.regex.Matcher.find(Matcher.java:535)
hudson.scm.CvsChangeLogHelper.mapCvsLog(CvsChangeLogHelper.java:169)
hudson.scm.CVSSCM.calculateChangeLog(CVSSCM.java:419)
hudson.scm.CVSSCM.checkout(CVSSCM.java:831)
hudson.model.AbstractProject.checkout(AbstractProject.java:1193)
hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:565)
hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:453)
hudson.model.Run.run(Run.java:1376)
hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
hudson.model.ResourceController.execute(ResourceController.java:88)
hudson.model.Executor.run(Executor.java:175)

Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
src/main/java/hudson/scm/CVSChangeLogSet.java
src/main/java/hudson/scm/CVSSCM.java
src/main/java/hudson/scm/CvsChangeLogHelper.java
src/main/java/hudson/scm/CvsLog.java
src/test/java/hudson/scm/CvsChangeLogHelperTest.java
http://jenkins-ci.org/commit/cvs-plugin/2565a8f550f61bbca9311aff60293806f4d6f67b
Log:
JENKINS-13814 don't buffer the entire rlog output in memory.

The size of the data "cvs rlog" produces is roughtly O(N*M) where
N is the # of files in the directory and M is the amount of changes.

So even when a delta is small, on a large repository this can produce
significant amount of data. Use of ByteArrayOutputStream causes a large
spike that can kill a VM, so spill the data over to disk if we are
getting large output.

I went ahead and made some performance improvements. I hope this didn't step on the toe of Michael.

My CVS knowledge is rusty, but there's inherent problem in retaining the entire "cvs rlog" output in memory. CVS rlog returns some data for every file in the repository, so on a large repository, it'll produce huge data even if there haven't been many files that have changed. I fixed this.

I'm also bit disappointed that the optimization we had in the CVS plugin 1.x appears to be gone, where we observed what files have changed during update and use that list to reduce the target of the log command. This was very useful for typical CI situation where your changes between builds are small. I suspect this also has a performance implication.

Finally, I'd like to advise against relying regular expression matching on entire "cvs log" output block, as it is both error prone and fragile.

When I look at https://github.com/jenkinsci/cvs-plugin/blob/master/src/main/java/hudson/scm/CvsLog.java#L273 I spot a number of problems right away — for example,

[\r|\n]+

is a mistake of

[\r\n]+

and there are various

.+?

that can incorrectly match multiple lines when the commit message contains stuff that looks suspiciously like cvs log output.

And in some cases regular expression matching will result in very inefficient backtracking, which is what I suspect to be the cause when people report "cvs rlog stuck".

I thought we used to bundle package-renamed CVS log parsing code taken from Ant, which I thought did the job better. I'm curious what the motivation is for replacing that with a custom parsing code.

We expect an upgrade of cvs plugin with a shorter interval of changelog. However the result is that the upgraded version (2.x) takes much longer. If we want to use the upgraded one, we have to turn off the changelog feature.