-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Platform: All, OS: Linux
-
Powered by SuggestiMate
We don't see these with p4win, which is why we suspect it is something with the
hudson plugin.
Performing sync with Perforce for: //xxx/...
[workspace] $ p4 workspace -o xxx_inspections Changing P4 Client Root to:
http://xxx.com:8082/job/xxx/ws/
Changing P4 Client View to: //xxx/... //xxx/...
[workspace] $ p4 -s client -i
Last sync'd change: 101540
[workspace] $ p4 changes -m 25 //xxx/...
[workspace] $ p4 describe -s 101661
[workspace] $ p4 describe -s 101656
Caught Exception communicating with perforce. Failed to communicate with
p4FATAL: Unable to communicate with perforce. Failed to communicate with p4
java.io.IOException: Unable to communicate with perforce. Failed to communicate
with p4
at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:340)
at hudson.model.AbstractProject.checkout(AbstractProject.java:574)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:251)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:225)
at hudson.model.Run.run(Run.java:771)
at hudson.model.Build.run(Build.java:85)
at hudson.model.ResourceController.execute(ResourceController.java:70)
at hudson.model.Executor.run(Executor.java:82)
[JENKINS-2062] Occasional Perforce Connection Errors
Are there any adverse effects when this happens? It should be noted that this is a Warning, nothing more...
It was noted earlier that pipes were getting leaked, but the plugin have become much more aggressive with cleaning those up. Is this still happening?
I'm not seeing any adverse effects other than all of these stack traces filling up my log file (48MB in two days)
I am seeing this issue on a recent Hudson installation. I see the warnings in the log files and using lsof I can see the number of open pipes increasing, in time with the warnings. Eventually we reach a point where we get a 'too many open files' error and the server has to be restarted.
The warning is always due to the getChangeNumbers call.
Hudson version 1.379
Perforce plugin version 1.1.9
Perforce server version P4D/SOLARIS10X86_64/2010.1/251161 (2010/06/16)
Hudson running on Linux : uname -a
Linux sj10lo01 2.6.9-78.0.25.ELlargesmp #1 SMP Fri Jun 26 07:56:47 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
Perforce server running on Solaris: uname -a
SunOS sj10ut01 5.10 Generic_137138-09 i86pc i386 i86pc
4561451 [SCM polling for hudson.model.FreeStyleProject@4bbf8a41[xxxx-block]] WARN perforce - Perforce process terminated suddenly
4561452 [SCM polling for hudson.model.FreeStyleProject@4bbf8a41[xxxx-block]] WARN perforce - java.io.IOException: Write end dead
at java.io.PipedInputStream.read(PipedInputStream.java:294)
at java.io.PipedInputStream.read(PipedInputStream.java:361)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.readLine(BufferedReader.java:299)
at java.io.BufferedReader.readLine(BufferedReader.java:362)
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:297)
at com.tek42.perforce.parse.Changes.getChangeNumbers(Changes.java:137)
at hudson.plugins.perforce.PerforceSCM.needToBuild(PerforceSCM.java:865)
at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:761)
at hudson.scm.SCM.poll(SCM.java:372)
at hudson.model.AbstractProject.poll(AbstractProject.java:1195)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:417)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:446)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
@gordonmcgregor: are your jobs configured to use slaves, and if so, what operating system are they running?
EDIT: also what operating system is your hudson server running on? It's difficult to tell with just a uname dump.
@rpetti
>Are your jobs configured to use slaves, and if so, what operating system are they running?
The server is set up as a single Master. All the jobs run on linux (see below)
>Also what operating system is your hudson server running on? It's difficult to tell with just a uname dump.
The hudson server (and sole build master) is a RedHat Linux system
/etc/redhat-release : Red Hat Enterprise Linux WS release 4 (Nahant Update 7)
/proc/version : Linux version 2.6.9-78.0.25.ELlargesmp (mockbuild@ls20-bc1-14.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-10)) #1 SMP Fri Jun 26 07:56:47 EDT 2009
Because of this issue, we have to manually restart Hudson daily. Otherwise it eventually gets to the point shown here.
[105] sj10lo01 ../hudson > ps -aux | grep hudson
verify 22719 16.7 0.6 1506004 768616 ? SNl Oct05 220:08 /usr/bin/java -jar /home/verify/hudson/hudson.war
[107] sj10lo01 ../hudson > lsof -p 22719
[... truncated ... ]
exe 22719 verify 1011r FIFO 0,7 64686985 pipe
exe 22719 verify 1012r FIFO 0,7 64673938 pipe
exe 22719 verify 1013r FIFO 0,7 64766263 pipe
exe 22719 verify 1014r FIFO 0,7 64762154 pipe
exe 22719 verify 1015r FIFO 0,7 64687011 pipe
exe 22719 verify 1016r FIFO 0,7 64712594 pipe
exe 22719 verify 1017r FIFO 0,7 64758276 pipe
exe 22719 verify 1019r FIFO 0,7 64687040 pipe
exe 22719 verify 1020r FIFO 0,7 64766310 pipe
[... truncated ... ]
[107] sj10lo01 ../hudson > lsof -p 22719 | grep pipe | wc
805 6440 60375
With those 805 pipes open (after about a day of uptime, single machine/master, 11 jobs, all polling once a minute)
the hudson.log then starts to fill with this error, for each job.
com.tek42.perforce.PerforceException: Could not run perforce command.
at hudson.plugins.perforce.HudsonP4Executor.exec(HudsonP4Executor.java:83)
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:289)
at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:53)
at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:973)
at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:755)
at hudson.scm.SCM.poll(SCM.java:372)
at hudson.model.AbstractProject.poll(AbstractProject.java:1195)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:417)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:446)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Cannot run program "/proj/merlot/bin/p4" (in directory "/proj/merlot/work/verify/hudson-ci-data/jobs/RVL-block/workspace"): java.io.IOException: error=24, Too many open files
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at hudson.Proc$LocalProc.<init>(Proc.java:192)
at hudson.Proc$LocalProc.<init>(Proc.java:164)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:638)
at hudson.Launcher$ProcStarter.start(Launcher.java:273)
at hudson.plugins.perforce.HudsonP4Executor.exec(HudsonP4Executor.java:74)
... 15 more
Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 20 more
If you need any further information to resolve this issue, please let me know.
For now I've scripted Hudson to reboot every day.
If I miss a reboot for a day, the server locks up with the 'too many files open' error messages
Seeing a similar issue on OS X 10.6.5
Hudson: 1.392
Perforce Plugin: 1.1.13
Java: 1.6.0_22 (running 64bit)
Let me know if there is additional information you'd like to have.
jmcintyre:
I'll need a full description of your problem including logs and version information for all systems and binaries (p4d, p4, perforce server operating system). If it's a connectivity issue, a high level description of where the servers are physically located in relation to one another would also be beneficial.
This particular ticket has been hijacked many times by many people each with a different problem, so when you say you're having a similar issue, I actually have no idea which one you are referring to.
To help keep this simple: I use no slaves, just a master.