Limiting the number of files is probably the easiest option here. Though another option is to somehow stream the information from disk on-demand, rather than deserializing the entire thing into the heap.

For now, what would a good limit be?

I was thinking of letting user to decide the limit. I have an implementation for that but I'll need to investigate further does it help and how the memory gets reserved.

We seem to have the same problem as well. Our files are not that big and it start to halt the webpage around log part 1560.
The biggest impact was, the webpage was not responsive and the only way we can do is kill the java process. — Is there a better way?

Thanks.
=========================
Mar 23, 2012 2:33:15 PM hudson.plugins.logparser.LogParserReader readLogPart
INFO: Done reading log part 1560
Mar 23, 2012 2:33:15 PM hudson.plugins.logparser.LogParserThread getLineStatuses
INFO: LogParserThread: Start parsing log part 1560
Mar 23, 2012 2:33:15 PM hudson.triggers.SCMTrigger$Runner runPolling
SEVERE: Failed to record SCM polling
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.io.BufferedReader.<init>(BufferedReader.java:80)
at java.io.BufferedReader.<init>(BufferedReader.java:91)
at hudson.plugins.perforce.HudsonP4DefaultExecutor.getReader(HudsonP4DefaultExecutor.java:123)
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:322)
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:292)
at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:61)
at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1325)
at hudson.plugins.perforce.PerforceSCM.compareRemoteRevisionWith(PerforceSCM.java:1019)
at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:356)
at hudson.scm.SCM.poll(SCM.java:373)
at hudson.model.AbstractProject.poll(AbstractProject.java:1340)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:420)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:449)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Mar 23, 2012 2:33:22 PM hudson.plugins.logparser.LogParserThread getLineStatuses
INFO: LogParserThread: Done parsing log part 1553
Mar 23, 2012 2:33:15 PM hudson.plugins.logparser.LogParserReader readLogPart
INFO: Start reading log part 1561 in thread #104
====================================

Code changed in jenkins
User: Rob Petti
Path:
src/main/java/com/tek42/perforce/parse/ChangelistBuilder.java
src/main/java/com/tek42/perforce/parse/Changes.java
src/main/java/hudson/plugins/perforce/PerforceSCM.java
src/main/resources/hudson/plugins/perforce/PerforceSCM/config.jelly
src/main/webapp/help/fileLimit.html
src/test/java/hudson/plugins/perforce/PerforceSCMTest.java
http://jenkins-ci.org/commit/perforce-plugin/a2523100d02ca360fd23b382af4e531c4dc6ac59
Log:
Merge pull request #18 from miktap/master

JENKINS-13109 Huge changelogs eat all the Java memory

Compare: https://github.com/jenkinsci/perforce-plugin/compare/11249ca...a252310

I noticed the same error today on 1.450 w/ perforce plugin 1.3.7. In my case, it isn't caused by huge changelogs - at least that I can tell. We don't have huge changelogs. We do have 2 slaves and all polling is done on the master. Here is the stack trace:

Apr 1, 2012 7:48:48 PM hudson.triggers.SCMTrigger$Runner runPolling
SEVERE: Failed to record SCM polling
java.lang.OutOfMemoryError: Java heap space
at hudson.remoting.FastPipedInputStream.<init>(FastPipedInputStream.java:78)
at hudson.remoting.FastPipedInputStream.<init>(FastPipedInputStream.java:66)
at hudson.plugins.perforce.HudsonP4RemoteExecutor.exec(HudsonP4RemoteExecutor.java:97)
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:321)
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:292)
at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:54)
at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1208)
at hudson.plugins.perforce.PerforceSCM.compareRemoteRevisionWith(PerforceSCM.java:903)
at hudson.scm.SCM._compareRemoteRevisionWith(SCM.java:356)
at hudson.scm.SCM.poll(SCM.java:373)
at hudson.model.AbstractProject.poll(AbstractProject.java:1323)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:420)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:449)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

We do keep Jenkins running without periodic reboots so I'm thinking there may be a memory leak somewhere as well. If anyone can give me hints how to get more information, I'll try to provide it.

-tim

Tim, the wiki has some information on how to debug memory leaks:
https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError

I'll also note that you are getting a different error from the one filed in the ticket, so your cause might be different.

Ok, I'll move my stack trace to a new issue and I'll update with OOM Error info when/if it occurs again. Thanks again, Rob.

Created this issue: https://issues.jenkins-ci.org/browse/JENKINS-13332