-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Hudson 1.379, Perforce plugin 1.1.9
-
Powered by SuggestiMate
I've seen a number of problems communicating with Perforce since upgrading to 1.1.9.
The build log mentions "No output for: " and then a perforce command (e.g. counter/workspace). The build then fails.
Usually the next build succeeds.
See attached file.
I've only seen it on slaves, i.e. remote clients.
- HUDSON-7664.diff
- 2 kB
- perforce.hpi
- 295 kB
- perforce-rpetti-54.hpi
- 299 kB
- perforce-rpetti-58.hpi
- 305 kB
- depends on
-
JENKINS-7809 Remote Launcher randomly returns no data.
-
- Resolved
-
- is duplicated by
-
JENKINS-7700 Perforce plugin failing occasionally with PerforceException: No output for:
-
- Resolved
-
-
JENKINS-8254 Intermittent errors where Perforce plugin can't communicate with Perforce
-
- Resolved
-
- is related to
-
JENKINS-8064 Perforce plugin incorrectly finds changes
-
- Resolved
-
[JENKINS-7664] Exceptions when communicating with Perforce "No output for:" (probably slave-only)
No, I can't build plugins. I would be willing to test builds for you if that would help, and it fits your timezone (I'm in UTC+2 at the moment). I'm still seeing it happen from time to time, although mainly when the company is "busy" (vague term defined as many builds in Hudson and supposedly much activity on the perforce server(s)).
Perhaps some (default?) value for timeout has changed?
I have downgraded to perforce plugin to 1.1.8. That seemed to work for a while, but in the meantime the builds fail again.
In my scenarion I ugraded from hudson 1.346 with perforce plugin 1.1.8 to hudson 1.379 with perforce plugin 1.1.9. Downgrading the perforce plugin did not help. Unfortunately I don't think I can easily downgrade the hudson version, since I suppose the configuration file format changed.
(More details in JENKINS-7700)
I only went from Hudson 1.375 via 1.378 (for an hour or so) to 1.379.
There has been some work on the master/slave communication, as far as I remember ... ?
Yeah, looks like 1.378 and 1.379 had some 'fixes' for master-slave communication.
Sorry, no. I only ran 1.378 for a short while - I read here in Jira that artifact archiving was prone to hanging, and sure enough it did so on my first build ... so I upgraded quickly to 1.379 which was out at that time.
Would downgrading to 1.377 even be possible? I could try that. I'll check the changelog for any new features that I use already. Downgrading the p4 plugin won't be easy
Seems OK to downgrade to 1.377. I'll do so first thing tomorrow morning and keep a sharp eye out for problems.
You might want to hold off on that, and I'll try and reproduce the issue on my end first. Since the problem seems intermittent, you can probably set the "Retry Count" option in Advanced Project Options so hudson will try a couple times to checkout before giving up. At least then you can go about your business without too much trouble until we can get this fixed.
OK, will do. I had completely forgotten the "Retry Count"; will see if that helps.
Will that handle problems with "p4 workspace", "p4 counter", etc, as well?
I downgraded hudson to 1.377. I get lots of error messages at startup, but it seems to work.
I still see
WARN perforce - Perforce process terminated suddenly WARN perforce - java.io.IOException: Write end dead
in the hudson.log, but I have not seen any builds failing as with 1.379.
So my current setup is 1.377 with perforce plugin 1.1.8
Increasing the Retry Count (to 3) has helped for me. I've seen it retry on a number of builds, but never more than once for the same build.
So the problem is still there somewhere, even if I can mask out its symptoms
I saw this as well ("No output for...") going from 1.376 to 1.380. Rolling back to 1.376 made the problem go away.
Upping the retries sort of made it tolerable without rolling back, except the "workspace" commands when polling seem to be only tried once, and if they get no output, a build is triggered. So our slaves were getting swamped with builds that had no changes.
Does it only affects builds on slaves, also with a master only Hudson?
Yes, we believe it only affects slaves.
– Posted from Bugbox for Android
Risking muddying the waters with duplicate info, on the chance this might help you debug...
With 1.379 / 1.1.10 intermittently seeing on slaves
Caught exception communicating with perforce. No output for: p4 -P 081D60BBE62A54C109493A7A6329B84D describe -s 114822 com.tek42.perforce.PerforceException: No output for: p4 -P 081D60BBE62A54C109493A7A6329B84D describe -s 114822
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:341)
at com.tek42.perforce.parse.Changes.getChangelist(Changes.java:63)
at com.tek42.perforce.parse.Changes.getChangelistsFromNumbers(Changes.java:381)
at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:552)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1082)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:479)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:411)
at hudson.model.Run.run(Run.java:1280)
at hudson.matrix.MatrixRun.run(MatrixRun.java:137)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:137)
I was able to reproduce this behavior for 1.378-1.380. I tested with perforce plugin 1.1.8 and 1.1.10 with the same results, so it doesn't seem related to any plugin changes.
The root cause is being sent to the hudson server log. I've opened a new issue (JENKINS-7809) against core, since this seems to be a problem with the remoting api changes made in 1.378. I'll continue to search for a work-around (especially for polling) but until then try not to upgrade past 1.377 if you can help it. Users who don't need polling can use torbent's Retry Count work-around.
Thank you. I'll downgrade to 1.377 for the time being; I'll risk JENKINS-5977 over the current situation ...
It looks like that HudsonPipedOutputStream closes the stream before it can get processed. this caused by the new threading model between of channel communication introduced in 1.378-1.379.
If we put a static sleep (1s) before the close then the no output problem has occurred rarely. Unfortunately this naive patch did not fix P4 vs Hudson issue. It created a new issue when Hudson is trying to setup P4 workspaces.
Code changed in hudson
User: : rpetti
Path:
trunk/hudson/plugins/perforce/src/main/java/hudson/plugins/perforce/HudsonPipedOutputStream.java
http://jenkins-ci.org/commit/36985
Log:
JENKINS-7664 committing potential fix for errors caused by misbehaving remote api.
Wow. We dug ourselves so desperately into how the threading is done after 378-379 that we not even thought of flushing the stream! Thanks pointing that out! I've put the patch into production on our Hudson farm and it seems that it has passed our first tests.
I'm glad to announce that our heavy duty Hudson grid works well now with Perforce and 25 nodes stable without any error during the last day with the patch applied. I vote for release!
That's awesome to hear, lkishalmi! Thanks for putting it through it's paces. I'll close the bug and run a release right now.
I'm sorry to say, but I was too euphoric after the first day success. Unfortunately we still see this issue as random encounter. However it's far less frequent than before, so it worth a release anyway. So I believe that JENKINS-7809 is a different issue than flushing and are responsible for the rest.
My proposed fix and the binary attached.
I suspect the issue is caused by the use of PipedInput/OutputStream, which has various known issues (such as a requirement that writer/reader be always the same thread.) The proposed patch fixes that.
If people seeing this problem can try the binary and report back their experience, that would be greatly appreciated.
I have a dev instance of Hudson that I tried the fix out on, and it looked like it worked A-OK! I've put the new plugin (and an upgraded Hudson .war) in place in our production Hudson environment. We have a large number of builds using Perforce and about 10 build nodes - I haven't seen any problems so far. I'll give it a few days to bake and post an update.
Thanks!!
We've just gave 1.1.12-SNAPSHOT (private-11/23/2010 03:13-kohsuke) a chance. Unfortunately it seems not solving this stream truncation issue.
At first glance, 1.1.12-SNAPSHOT (private-11/23/2010 03:13-kohsuke) seems to have fixed my issues, at least from my log file perspective I do not see any "WARN perforce - java.io.IOException: Write end dead" messages anymore. I have no slaves, just a master.
I did see one exception that may have been caused by the fix:
5576473 Handling GET /scm/PerforceSCM/checkChangeList : RequestHandlerThread[#1] INFO perforce - Executing: /usr/local/bin/p4 counter change
Nov 28, 2010 9:36:00 PM hudson.plugins.perforce.PerforceSCM processWorkspaceBeforeDeletion
INFO: Workspace is being deleted; enabling one-time force sync.
Problem: Connect to server failed; check $P4PORT
com.tek42.perforce.PerforceException: Connect to server failed; check $P4PORT
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:339)
at com.tek42.perforce.parse.Changes.getChangeNumbers(Changes.java:137)
at hudson.plugins.perforce.PerforceSCM.needToBuild(PerforceSCM.java:897)
at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:793)
at hudson.scm.SCM.poll(SCM.java:373)
at hudson.model.AbstractProject.poll(AbstractProject.java:1237)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454)
at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
I also no longer have "Write end dead" messages from perforce in my Hudson logs. It looks like builds have been working w/o any problems since installing the new perforce plugin. I do have messages in my logs like below, but they seem to be benign:
Problem: No output for: /var/hudson/bin/p4 changes -m 2 //JobName--1660205398/... com.tek42.perforce.PerforceException: No output for: /var/hudson/bin/p4 changes -m 2 //JobName--16 at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:341) at com.tek42.perforce.parse.Changes.getChangeNumbers(Changes.java:137) at hudson.plugins.perforce.PerforceSCM.needToBuild(PerforceSCM.java:897) at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:793) at hudson.scm.SCM.poll(SCM.java:373) at hudson.model.AbstractProject.poll(AbstractProject.java:1237) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:425) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:454) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Nov 23, 2010 4:11:30 PM hudson.remoting.ProxyOutputStream$Chunk$1 run WARNING: Failed to write to stream java.io.IOException: Pipe is already closed at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:147) at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:131) at hudson.util.DelegatingOutputStream.write(DelegatingOutputStream.java:51) at hudson.remoting.ProxyOutputStream$Chunk$1.run(ProxyOutputStream.java:185) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: hudson.remoting.FastPipedInputStream$ClosedBy: The pipe was closed at... at hudson.remoting.FastPipedOutputStream.close(FastPipedOutputStream.java:94) at hudson.plugins.perforce.HudsonPipedOutputStream$1.run(HudsonPipedOutputStream.java:43) ... 1 more
@duckyd: Unless there are no files in your view spec, this is the exact same problem the bug is describing.
@jchristi: "Connect to server failed; check $P4PORT" is an error message from the p4 client, so the plugin is able to receive output through the pipe without any issues. The error message is likely because your perforce server has become unavailable temporarily for whatever reason. I don't believe it has anything to do with this issue, or kohsuke's patch.
@rpetti the error (in the logs) I'm seeing currently isn't causing builds to fail. The error that I was getting, which caused builds to fail (and which appears to be fixed) was:
Caught exception communicating with perforce. No output for: /usr/local/bin/p4 workspace -o JobName-612004424 com.tek42.perforce.PerforceException: No output for: /usr/local/bin/p4 workspace -o JobName-612004424 at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:341) at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:53) at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:992) at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:504) at hudson.model.AbstractProject.checkout(AbstractProject.java:1119) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:480) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:412) at hudson.model.Run.run(Run.java:1325) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:293) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:139) ERROR: Unable to communicate with perforce. No output for: /usr/local/bin/p4 workspace -o JobName-612004424
The error "No output for: /usr/local/bin/p4" is the underlying cause, and affects all perforce commands run. It just happens to not be failing during p4 sync for you so far.
@rpetti I see. Thankfully, it doesn't appear to cause build to fail, which is a significant improvement. Prior to the fix, it was unusable (we had to revert to an earlier version of Hudson & the perforce plugin)
All it's done is reduce the frequency of the problem... I'm betting you will still see failed builds in the future.
I have the exact same issue after upgrading from version 1.1.1 to 1.1.12. That is a large span of versions but I did a full Hudson upgrade this past weekend to the latest 1.387 and this seemed like a good time to try the new version of the p4 plugin. There have been no changes to the jobs since the upgrade and all is OK except for this error happening several times since then. The previous version 1.1.1 had never shown this error at all for months. It was very reliable in all aspects.
I see this in Ubuntu, CentOS, Windows Server 64bit slaves. All exactly the same with the same exception. This sometimes passes with a few retries but it happens almost every build. It is just an unlucky coincidence when it passes with retries than fails completely.
I am not sure if I could downgrade and how far back I would be able to get this to be back in stable mode.
Any hints as to the root cause of this?
FYI, you may remember I could build this plugin and try it out if that helps. I have not done it in some time because it has been very stable but it would be worth a try. Do you have any pointers as to what I should be looking for? It is very reproducible in my environment.
As mentioned in the comments above, this is an issue in hudson that was introduced in 1.378 that has yet to be fixed. Apart from the workarounds suggested, you have no option but to either deal with the random failures, or downgrade to 1.377.
I tried the binary patch provided by kohsuke and it seems to have fixed all issues. I am now running Hudson 1.387 with the 1.1.12-SNAPSHOT (private-11/23/2010 03:13-kohsuke) with 10 slaves and all the errors are gone. Builds are happy again.
Getting a lot of mixed reports about that patch... I still think there is a problem with the underlying remoting functions in hudson, but if there's even a slight chance of improvement, then it's probably worth committing at this point.
Yes, although initially the patch seemed to help the errors are back but the frequency is reduced. It does point to something related to the remoting functions.
We recently upgraded to the new version of the Perforce plugin (v 1.1.13) and have experienced it locking up our Perforce server (requiring a restart). It looks like a client -i command never completes, holding a lock on the perforce side that then blocks all subsequent commands.
Edit: Downgrading to 1.1.11 hasn't fixed the issue of the server locking with multiple, parallel 'client -i' commands. I thought it had, seems I was wrong - just took longer to lock up.
Hudson version 1.392
Perforce plug-in versions 1.1.13 and 1.1.11
../perforce > ./p4d -V
Perforce - The Fast Software Configuration Management System.
Copyright 1995-2010 Perforce Software. All rights reserved.
Rev. P4D/SOLARIS10X86_64/2010.1/273938 (2010/11/16).
Gordon: please file a new issue. Your problem is substantially different from this one. In any case, I would recommend trying hudson 1.377, just in case the root cause is the same for both issues. Also ensure that the plugin is configured to use the raw p4 client binary, and not some wrapper script.
As can be seen in the changelog, my patch made it into 1.1.13. For those of you who are still seeing a problem, please provide a stack trace so that we can better understand if the problem is still there or not.
So ... would it be worth it to try upgrading?
We're still running core 1.377 and p4 plugin 1.1.10, waiting for this issue to be fixed. I might try upgrading, but I really don't want to end up with these problems again.
Actually I haven't upgraded anything Hudson-y at all since October because of this.
If I should upgrade, what kind of input from me would help most? Just upgrade the p4 plugin with core, see what happens, then later update core? Or all at once?
Let me also note that I also saw problems with the Cobertura plugin, as mentioned in the core issue (JENKINS-7809). Is there also an update of that?
The latest perforce plugin should work just fine, it's only after upgrading hudson past 1.377 that the problem arises.
With kohsuke's patch, I wasn't able to reproduce the problem, but then again the rate of failure without it was only 1-2% in my tests. Others have said that the problem is still there, and that it just happens less frequently.
I doubt it's fully fixed, torbent, but if you want to help then you could try upgrading and providing a stack trace if the problem presents itself again. If your hudson deployment is mission critical, then you should probably stay at 1.377...
Here is an output using the latest 1.394 Hudson and latest Perforce Plugin 1.1.14. I hope that this will help speed up the resolution for the issue which has halted out upgrades since 1.374...
Using remote perforce client: HudsonSlaveWebTestClient--1608677735
[SlaveAutotestHelper] $ "C:\Program Files\Perforce\p4.exe" workspace -o HudsonSlaveWebTestClient--1608677735
[SlaveAutotestHelper] $ "C:\Program Files\Perforce\p4.exe" login -p
Caught exception communicating with perforce. No output for: C:\Program Files\Perforce\p4.exe -P Enter password: workspace -o HudsonSlaveWebTestClient--1608677735 com.tek42.perforce.PerforceException: No output for: C:\Program Files\Perforce\p4.exe -P Enter password: workspace -o HudsonSlaveWebTestClient--1608677735
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:341)
at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:53)
at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1004)
at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:518)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1171)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:480)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:412)
at hudson.model.Run.run(Run.java:1362)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)
ERROR: Unable to communicate with perforce. No output for: C:\Program Files\Perforce\p4.exe -P Enter password: workspace -o HudsonSlaveWebTestClient--1608677735
[SlaveAutotestHelper] $ "C:\Program Files\Perforce\p4.exe" -P "Enter password: " workspace -o HudsonSlaveWebTestClient--1608677735
jonasbovin: How frequently does this occur? Your problem looks different...
We just upgraded to Hudson 1.394 and Perforce Plugin 1.1.14, and our builds are also starting to fail, for example:
Using remote perforce client: hudson-main-member-e2e-957586325 [build111_main] $ p4 workspace -o hudson-main-member-e2e-957586325 Last sync'd change: 109895 [build111_main] $ p4 counter change [build111_main] $ p4 -s changes //hudson-main-member-e2e-957586325/...@109896,@109913 [build111_main] $ p4 describe -s 109913 Caught exception communicating with perforce. No output for: p4 describe -s 109913 com.tek42.perforce.PerforceException: No output for: p4 describe -s 109913 at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:341) at com.tek42.perforce.parse.Changes.getChangelist(Changes.java:63) at com.tek42.perforce.parse.Changes.getChangelistsFromNumbers(Changes.java:389) at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:566) at hudson.model.AbstractProject.checkout(AbstractProject.java:1171) at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:480) at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:412) at hudson.model.Run.run(Run.java:1362) at hudson.matrix.MatrixRun.run(MatrixRun.java:137) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:145) ERROR: Unable to communicate with perforce. No output for: p4 describe -s 109913
So far I've only seen this on 2 of the matrix builds, at a rate of about 1 in 20 configurations throwing this error.
I also see a lot of this in the master hudson log:
Jan 20, 2011 1:51:05 PM hudson.remoting.ProxyOutputStream$Chunk$1 run WARNING: Failed to write to stream java.io.IOException: Pipe is already closed at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:147) at hudson.remoting.FastPipedOutputStream.write(FastPipedOutputStream.java:131) at hudson.util.DelegatingOutputStream.write(DelegatingOutputStream.java:51) at hudson.remoting.ProxyOutputStream$Chunk$1.run(ProxyOutputStream.java:185) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: hudson.remoting.FastPipedInputStream$ClosedBy: The pipe was closed at... at hudson.remoting.FastPipedInputStream.close(FastPipedInputStream.java:112) at sun.nio.cs.StreamDecoder.implClose(StreamDecoder.java:358) at sun.nio.cs.StreamDecoder.close(StreamDecoder.java:173) at java.io.InputStreamReader.close(InputStreamReader.java:182) at java.io.BufferedReader.close(BufferedReader.java:497) at com.tek42.perforce.parse.AbstractPerforceTemplate.saveToPerforce(AbstractPerforceTemplate.java:222) at com.tek42.perforce.parse.Workspaces.saveWorkspace(Workspaces.java:68) at hudson.plugins.perforce.PerforceSCM.saveWorkspaceIfDirty(PerforceSCM.java:1162) at hudson.plugins.perforce.PerforceSCM.pollChanges(PerforceSCM.java:790) at hudson.scm.SCM.poll(SCM.java:373) at hudson.model.AbstractProject.poll(AbstractProject.java:1289) at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:420) at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:449) at hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118) ... 6 more
Perforce client version is:
Rev. P4/LINUX26X86_64/2010.1/265509 (2010/09/23).
Server:
Server version: P4D/LINUX26X86_64/2010.1/265509 (2010/09/23)
Java Version:
Java(TM) SE Runtime Environment (build 1.6.0_16-b01)
Java HotSpot(TM) 64-Bit Server VM (build 14.2-b01, mixed mode)
rpetti: The reason it looks different is probably due to the fact that we use ticket based authentication - which means it always fails on login and never gets beyond that.
With regards to frequency, this is output from an automated WebTest run through Hudson itself and I use it on the test environment server. It verifies the existence of the bug and fails every time it is run (@daily). Before this test passes we cannot upgrade beyond the current version.
I hope this helps. Please feel free to inquire again.
Looks like I jumped the gun earlier, when running it more frequently I can achieve as much as 80% successes and 20% "no output error".
Hope this helps.
If anyone still able to reproduce this problem would care to try the attached build. I've made a minor change that will hopefully do the trick. perforce-rpetti-54.hpi
rpetti: I've tried the perforce-rpetti-54.hpi on a 1.394 Hudson, but unfortunately it just hangs after:
...
Using master perforce client: HudsonWebTestClient
--[workspace] $ "C:\Program Files\Perforce\p4.exe" workspace -o HudsonWebTestClient
never gets beyond the workspace command - usually what happens is that it fails - as it is not logged in and then issues the login statement an gets a ticket (see below):
When using the 1.1.14 plugin which every now and then get 'No output...'
Using master perforce client: HudsonWebTestClient
--[workspace] $ "C:\Program Files\Perforce\p4.exe" workspace -o HudsonWebTestClient
--[workspace] $ "C:\Program Files\Perforce\p4.exe" login -p
--[workspace] $ "C:\Program Files\Perforce\p4.exe" -P 20DADC12CB3FEA1432C78D34E289BAB2 workspace -o HudsonWebTestClient
Last sync'd change: 183801
When I try to cancel the hanging job, it resumes next line as above on 1.1.14 plugin. This also affects non-slaves, master will also hang until job is canceled manually from Hudson web-interface.
Using remote perforce client: HudsonSlaveWebTestClient--1608677735
[SlaveAutotestHelper] $ "C:\Program Files\Perforce\p4.exe" workspace o HudsonSlaveWebTestClient-1608677735 (first halted, then I canceled job manually)
[SlaveAutotestHelper] $ "C:\Program Files\Perforce\p4.exe" login -p (haltes, then I canceled job manually)
Caught exception communicating with perforce. Unable to login via p4 login due to IOException: nullcom.tek42.perforce.PerforceException: Unable to login via p4 login due to IOException: null
at com.tek42.perforce.parse.AbstractPerforceTemplate.login(AbstractPerforceTemplate.java:488)
at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:350)
at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:53)
at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1081)
at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:541)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1171)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:480)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:412)
at hudson.model.Run.run(Run.java:1362)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:145)
ERROR: Unable to communicate with perforce. Unable to login via p4 login due to IOException: null
Finished: FAILURE
I tried 1.1.15 and got the same result: hang on the workspace command. With 1.1.14 I get the "no output" problem 3 out of 12 times.
I'm fully willing to keep trying beta versions if you keep posting them. Alternatively, my email is "shoughton at highmoonstudios.com".
I was able to reproduce the problem (or one strikingly similar) and was able to fix it by reimplementing the remote calls performed by the plugin. Can someone else test it? (perforce-rpetti-58.hpi)
If it works for others, then I'll release it and update JENKINS-7809 with my findings.
Well I've installed the new plugin last friday morning. Since then we did not experience any perforce "no output" related problems. This is a good sign. We keep watching it!
That's SO great! This means I may finally move on from Hudson 1.377 – to Jenkins, naturally
This issue started on Hudson, and our Hudson installation is actually still at 1.377 because of it. We've since moved some of our jobs to Jenkins and haven't seen the slightest trace of this problem, so I think it's (about) time to close it.
I don't suppose anyone who can reproduce this problem has the ability to build the perforce plugin? We may have to bisect the changes between 1.1.8 and 1.1.9 to see what is causing this. If not, I can provide a set of builds that you can test.