• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • p4-plugin
    • None
    • Platform: All, OS: Linux

      We don't see these with p4win, which is why we suspect it is something with the
      hudson plugin.

      Performing sync with Perforce for: //xxx/...
      [workspace] $ p4 workspace -o xxx_inspections Changing P4 Client Root to:
      http://xxx.com:8082/job/xxx/ws/
      Changing P4 Client View to: //xxx/... //xxx/...
      [workspace] $ p4 -s client -i
      Last sync'd change: 101540
      [workspace] $ p4 changes -m 25 //xxx/...
      [workspace] $ p4 describe -s 101661
      [workspace] $ p4 describe -s 101656
      Caught Exception communicating with perforce. Failed to communicate with
      p4FATAL: Unable to communicate with perforce. Failed to communicate with p4
      java.io.IOException: Unable to communicate with perforce. Failed to communicate
      with p4
      at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:340)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:574)
      at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:251)
      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:225)
      at hudson.model.Run.run(Run.java:771)
      at hudson.model.Build.run(Build.java:85)
      at hudson.model.ResourceController.execute(ResourceController.java:70)
      at hudson.model.Executor.run(Executor.java:82)

          [JENKINS-2062] Occasional Perforce Connection Errors

          jchristi added a comment -

          To help keep this simple: I use no slaves, just a master.

          jchristi added a comment - To help keep this simple: I use no slaves, just a master.

          Rob Petti added a comment -

          Are there any adverse effects when this happens? It should be noted that this is a Warning, nothing more...

          It was noted earlier that pipes were getting leaked, but the plugin have become much more aggressive with cleaning those up. Is this still happening?

          Rob Petti added a comment - Are there any adverse effects when this happens? It should be noted that this is a Warning, nothing more... It was noted earlier that pipes were getting leaked, but the plugin have become much more aggressive with cleaning those up. Is this still happening?

          jchristi added a comment - - edited

          I'm not seeing any adverse effects other than all of these stack traces filling up my log file (48MB in two days)

          jchristi added a comment - - edited I'm not seeing any adverse effects other than all of these stack traces filling up my log file (48MB in two days)

          I am seeing this issue on a recent Hudson installation. I see the warnings in the log files and using lsof I can see the number of open pipes increasing, in time with the warnings. Eventually we reach a point where we get a 'too many open files' error and the server has to be restarted.

          The warning is always due to the getChangeNumbers call.

          Hudson version 1.379
          Perforce plugin version 1.1.9
          Perforce server version P4D/SOLARIS10X86_64/2010.1/251161 (2010/06/16)

          Hudson running on Linux : uname -a
          Linux sj10lo01 2.6.9-78.0.25.ELlargesmp #1 SMP Fri Jun 26 07:56:47 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux

          Perforce server running on Solaris: uname -a
          SunOS sj10ut01 5.10 Generic_137138-09 i86pc i386 i86pc

          4561451 [SCM polling for hudson.model.FreeStyleProject@4bbf8a41[xxxx-block]] WARN perforce - Perforce process terminated suddenly
          4561452 [SCM polling for hudson.model.FreeStyleProject@4bbf8a41[xxxx-block]] WARN perforce - java.io.IOException: Write end dead
          at java.io.PipedInputStream.read(PipedInputStream.java:294)
          at java.io.PipedInputStream.read(PipedInputStream.java:361)
          at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
          at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
          at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
          at java.io.InputStreamReader.read(InputStreamReader.java:167)
          at java.io.BufferedReader.fill(BufferedReader.java:136)
          at java.io.BufferedReader.readLine(BufferedReader.java:299)
          at java.io.BufferedReader.readLine(BufferedReader.java:362)
          at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:297)
          at com.tek42.perforce.parse.Changes.getChangeNumbers(Changes.java:137)
          at hudson.plugins.perforce.PerforceSCM.needToBuild(PerforceSCM.java:865)
          at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:761)
          at hudson.scm.SCM.poll(SCM.java:372)
          at hudson.model.AbstractProject.poll(AbstractProject.java:1195)
          at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:417)
          at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:446)
          at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
          at java.lang.Thread.run(Thread.java:619)

          GordonMcGregor added a comment - I am seeing this issue on a recent Hudson installation. I see the warnings in the log files and using lsof I can see the number of open pipes increasing, in time with the warnings. Eventually we reach a point where we get a 'too many open files' error and the server has to be restarted. The warning is always due to the getChangeNumbers call. Hudson version 1.379 Perforce plugin version 1.1.9 Perforce server version P4D/SOLARIS10X86_64/2010.1/251161 (2010/06/16) Hudson running on Linux : uname -a Linux sj10lo01 2.6.9-78.0.25.ELlargesmp #1 SMP Fri Jun 26 07:56:47 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux Perforce server running on Solaris: uname -a SunOS sj10ut01 5.10 Generic_137138-09 i86pc i386 i86pc 4561451 [SCM polling for hudson.model.FreeStyleProject@4bbf8a41 [xxxx-block] ] WARN perforce - Perforce process terminated suddenly 4561452 [SCM polling for hudson.model.FreeStyleProject@4bbf8a41 [xxxx-block] ] WARN perforce - java.io.IOException: Write end dead at java.io.PipedInputStream.read(PipedInputStream.java:294) at java.io.PipedInputStream.read(PipedInputStream.java:361) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158) at java.io.InputStreamReader.read(InputStreamReader.java:167) at java.io.BufferedReader.fill(BufferedReader.java:136) at java.io.BufferedReader.readLine(BufferedReader.java:299) at java.io.BufferedReader.readLine(BufferedReader.java:362) at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:297) at com.tek42.perforce.parse.Changes.getChangeNumbers(Changes.java:137) at hudson.plugins.perforce.PerforceSCM.needToBuild(PerforceSCM.java:865) at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:761) at hudson.scm.SCM.poll(SCM.java:372) at hudson.model.AbstractProject.poll(AbstractProject.java:1195) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:417) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:446) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619)

          Rob Petti added a comment - - edited

          @gordonmcgregor: are your jobs configured to use slaves, and if so, what operating system are they running?

          EDIT: also what operating system is your hudson server running on? It's difficult to tell with just a uname dump.

          Rob Petti added a comment - - edited @gordonmcgregor: are your jobs configured to use slaves, and if so, what operating system are they running? EDIT: also what operating system is your hudson server running on? It's difficult to tell with just a uname dump.

          @rpetti

          >Are your jobs configured to use slaves, and if so, what operating system are they running?

          The server is set up as a single Master. All the jobs run on linux (see below)

          >Also what operating system is your hudson server running on? It's difficult to tell with just a uname dump.

          The hudson server (and sole build master) is a RedHat Linux system
          /etc/redhat-release : Red Hat Enterprise Linux WS release 4 (Nahant Update 7)
          /proc/version : Linux version 2.6.9-78.0.25.ELlargesmp (mockbuild@ls20-bc1-14.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-10)) #1 SMP Fri Jun 26 07:56:47 EDT 2009

          GordonMcGregor added a comment - @rpetti >Are your jobs configured to use slaves, and if so, what operating system are they running? The server is set up as a single Master. All the jobs run on linux (see below) >Also what operating system is your hudson server running on? It's difficult to tell with just a uname dump. The hudson server (and sole build master) is a RedHat Linux system /etc/redhat-release : Red Hat Enterprise Linux WS release 4 (Nahant Update 7) /proc/version : Linux version 2.6.9-78.0.25.ELlargesmp (mockbuild@ls20-bc1-14.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-10)) #1 SMP Fri Jun 26 07:56:47 EDT 2009

          Because of this issue, we have to manually restart Hudson daily. Otherwise it eventually gets to the point shown here.

          [105] sj10lo01 ../hudson > ps -aux | grep hudson
          verify 22719 16.7 0.6 1506004 768616 ? SNl Oct05 220:08 /usr/bin/java -jar /home/verify/hudson/hudson.war

          [107] sj10lo01 ../hudson > lsof -p 22719
          [... truncated ... ]
          exe 22719 verify 1011r FIFO 0,7 64686985 pipe
          exe 22719 verify 1012r FIFO 0,7 64673938 pipe
          exe 22719 verify 1013r FIFO 0,7 64766263 pipe
          exe 22719 verify 1014r FIFO 0,7 64762154 pipe
          exe 22719 verify 1015r FIFO 0,7 64687011 pipe
          exe 22719 verify 1016r FIFO 0,7 64712594 pipe
          exe 22719 verify 1017r FIFO 0,7 64758276 pipe
          exe 22719 verify 1019r FIFO 0,7 64687040 pipe
          exe 22719 verify 1020r FIFO 0,7 64766310 pipe
          [... truncated ... ]

          [107] sj10lo01 ../hudson > lsof -p 22719 | grep pipe | wc
          805 6440 60375

          With those 805 pipes open (after about a day of uptime, single machine/master, 11 jobs, all polling once a minute)
          the hudson.log then starts to fill with this error, for each job.

          com.tek42.perforce.PerforceException: Could not run perforce command.
          at hudson.plugins.perforce.HudsonP4Executor.exec(HudsonP4Executor.java:83)
          at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:289)
          at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:53)
          at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:973)
          at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:755)
          at hudson.scm.SCM.poll(SCM.java:372)
          at hudson.model.AbstractProject.poll(AbstractProject.java:1195)
          at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:417)
          at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:446)
          at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
          at java.lang.Thread.run(Thread.java:619)
          Caused by: java.io.IOException: Cannot run program "/proj/merlot/bin/p4" (in directory "/proj/merlot/work/verify/hudson-ci-data/jobs/RVL-block/workspace"): java.io.IOException: error=24, Too many open files
          at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
          at hudson.Proc$LocalProc.<init>(Proc.java:192)
          at hudson.Proc$LocalProc.<init>(Proc.java:164)
          at hudson.Launcher$LocalLauncher.launch(Launcher.java:638)
          at hudson.Launcher$ProcStarter.start(Launcher.java:273)
          at hudson.plugins.perforce.HudsonP4Executor.exec(HudsonP4Executor.java:74)
          ... 15 more
          Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files
          at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
          at java.lang.ProcessImpl.start(ProcessImpl.java:65)
          at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
          ... 20 more

          GordonMcGregor added a comment - Because of this issue, we have to manually restart Hudson daily. Otherwise it eventually gets to the point shown here. [105] sj10lo01 ../hudson > ps -aux | grep hudson verify 22719 16.7 0.6 1506004 768616 ? SNl Oct05 220:08 /usr/bin/java -jar /home/verify/hudson/hudson.war [107] sj10lo01 ../hudson > lsof -p 22719 [... truncated ... ] exe 22719 verify 1011r FIFO 0,7 64686985 pipe exe 22719 verify 1012r FIFO 0,7 64673938 pipe exe 22719 verify 1013r FIFO 0,7 64766263 pipe exe 22719 verify 1014r FIFO 0,7 64762154 pipe exe 22719 verify 1015r FIFO 0,7 64687011 pipe exe 22719 verify 1016r FIFO 0,7 64712594 pipe exe 22719 verify 1017r FIFO 0,7 64758276 pipe exe 22719 verify 1019r FIFO 0,7 64687040 pipe exe 22719 verify 1020r FIFO 0,7 64766310 pipe [... truncated ... ] [107] sj10lo01 ../hudson > lsof -p 22719 | grep pipe | wc 805 6440 60375 With those 805 pipes open (after about a day of uptime, single machine/master, 11 jobs, all polling once a minute) the hudson.log then starts to fill with this error, for each job. com.tek42.perforce.PerforceException: Could not run perforce command. at hudson.plugins.perforce.HudsonP4Executor.exec(HudsonP4Executor.java:83) at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:289) at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:53) at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:973) at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:755) at hudson.scm.SCM.poll(SCM.java:372) at hudson.model.AbstractProject.poll(AbstractProject.java:1195) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:417) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:446) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) Caused by: java.io.IOException: Cannot run program "/proj/merlot/bin/p4" (in directory "/proj/merlot/work/verify/hudson-ci-data/jobs/RVL-block/workspace"): java.io.IOException: error=24, Too many open files at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at hudson.Proc$LocalProc.<init>(Proc.java:192) at hudson.Proc$LocalProc.<init>(Proc.java:164) at hudson.Launcher$LocalLauncher.launch(Launcher.java:638) at hudson.Launcher$ProcStarter.start(Launcher.java:273) at hudson.plugins.perforce.HudsonP4Executor.exec(HudsonP4Executor.java:74) ... 15 more Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) ... 20 more

          If you need any further information to resolve this issue, please let me know.

          For now I've scripted Hudson to reboot every day.
          If I miss a reboot for a day, the server locks up with the 'too many files open' error messages

          GordonMcGregor added a comment - If you need any further information to resolve this issue, please let me know. For now I've scripted Hudson to reboot every day. If I miss a reboot for a day, the server locks up with the 'too many files open' error messages

          jmcintyre added a comment -

          Seeing a similar issue on OS X 10.6.5

          Hudson: 1.392
          Perforce Plugin: 1.1.13
          Java: 1.6.0_22 (running 64bit)

          Let me know if there is additional information you'd like to have.

          jmcintyre added a comment - Seeing a similar issue on OS X 10.6.5 Hudson: 1.392 Perforce Plugin: 1.1.13 Java: 1.6.0_22 (running 64bit) Let me know if there is additional information you'd like to have.

          Rob Petti added a comment -

          jmcintyre:

          I'll need a full description of your problem including logs and version information for all systems and binaries (p4d, p4, perforce server operating system). If it's a connectivity issue, a high level description of where the servers are physically located in relation to one another would also be beneficial.

          This particular ticket has been hijacked many times by many people each with a different problem, so when you say you're having a similar issue, I actually have no idea which one you are referring to.

          Rob Petti added a comment - jmcintyre: I'll need a full description of your problem including logs and version information for all systems and binaries (p4d, p4, perforce server operating system). If it's a connectivity issue, a high level description of where the servers are physically located in relation to one another would also be beneficial. This particular ticket has been hijacked many times by many people each with a different problem, so when you say you're having a similar issue, I actually have no idea which one you are referring to.

            Unassigned Unassigned
            yschimke yschimke
            Votes:
            18 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated: