-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Platform: All, OS: All
-
Powered by SuggestiMate
If I'm using the Perforce plugin and have a setting for this project
(FPHibernate-Head) with a workspace like this:
Single Workspace: wyliek.hudson-clients.fphibernate-head.london
View: //fotech/shared/lib/java/FPHibernate/head/...
Hudson Home: /scratch/betasx2/wyliek/hudson-home
And am using a configuration matrix of jdk1.6.0_03 and jdk1.5.0_13, what seems
to happen is this:
1) Perforce plugin does a sync to
/scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace
2) Matrix runner creates axis directory of
/scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace/jdk/jdk1.5.0_13
3) Perforce Plugin changes the root of the workspace to
/scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace/jdk/jdk1.5.0_13
and does a p4 sync. P4 says nothing to sync, but there's nothing there.
4) Ant doesn't find the build.xml file to run.
Here's the output on the overall project running:
started
Performing sync with Perforce for: //fotech/shared/lib/java/FPHibernate/head/...
Changing P4 Client Root to:
/scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace/
Changing P4 Client View to: //fotech/shared/lib/java/FPHibernate/head/...
//wyliek.hudson-clients.fphibernate-head.london/...
Last sync'd change: 76672
Sync'ing workspace to depot.
ForceSync flag is set, forcing: p4 sync
//fotech/shared/lib/java/FPHibernate/head/...
Sync complete, took 3392 MS
Triggering jdk=jdk1.6.0_03
Triggering jdk=jdk1.5.0_13
finished: FAILURE
Here's the output on the jdk1.6.0_03 axis run:
started
Performing sync with Perforce for: //fotech/shared/lib/java/FPHibernate/head/...
Changing P4 Client Root to:
/scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace/jdk/jdk1.6.0_03/
Changing P4 Client View to: //fotech/shared/lib/java/FPHibernate/head/...
//wyliek.hudson-clients.fphibernate-head.london/...
Last sync'd change: 76672
No changes since last build.
[jdk1.6.0_03] $ /scratch/betasx2/wyliek/apache-ant-1.7.0/bin/ant
-Djdk=jdk1.6.0_03 tests
Buildfile: build.xml does not exist!
Build failed
Recording test results
finished: FAILURE
Here's the output on the jdk1.5.0_13 axis run:
started
Performing sync with Perforce for: //fotech/shared/lib/java/FPHibernate/head/...
Changing P4 Client Root to:
/scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace/jdk/jdk1.5.0_13/
Changing P4 Client View to: //fotech/shared/lib/java/FPHibernate/head/...
//wyliek.hudson-clients.fphibernate-head.london/...
Last sync'd change: 76672
No changes since last build.
[jdk1.5.0_13] $ /scratch/betasx2/wyliek/apache-ant-1.7.0/bin/ant
-Djdk=jdk1.5.0_13 tests
Buildfile: build.xml does not exist!
Build failed
Recording test results
finished: FAILURE
[JENKINS-1022] Interaction between Perforce plugin and Matrix builds
I've just noted the same behavior in this thread on the users mailing list:
http://www.nabble.com/Perforce-workspace-issues-in-matrix-project-td25198191.html
I don't understand how you can say this is just the behavior of Perforce and not
something you can fix. Hudson already has behavior specific to Perforce in
creating and managing custom Perforce workspaces, but in this instance it's
using them incorrectly. Hudson is trying to use the same Perforce workspace for
two different roots without re-syncing.
In the thread, I offered two solutions:
1) Don't run the main job on the same node/slave as the axis jobs – this would
also have the benefit of not blocking itself when there is only one executor on
a slave.
2) Create additional Perforce workspaces for the axis jobs – just as the main
job syncs to "workspace" and the axis jobs sync to "workspace/axis", the main
job could use Perforce workspace "x" and the axis jobs could use Perforce
workspace "x-axis".
Please reconsider this issue. Otherwise you should probably document on the
matrix feature that it is not inteded to work with Perforce.
I saw other people asking for the ability to tie the parent matrix job to a
particular node in Issue 936, so I thought I'd try that as a workaround for this
issue as well. I just copied the "master/slave" section from the
AbstractItem/configure-common.jelly to the
MatrixProject/configure-entries.jelly, so that the "Tie this project to a node"
option would be exposed for Matrix projects as well. Then I tied the Matrix
project itself to my Master node, and for the axis I just selected two
"Individual Nodes" on other machines. It looks like it should be working now
– I see three progress bars in the "Build Executor Status" section as I would
expect. I can see from the logs that it is now using three distinct perforce
workspaces, but one of them still doesn't sync. Here are the relevant portions
of the build logs:
--------------------
Started by user anonymous
Building on master
Using master perforce client: experimental_boogaloo_4
[workspace] $ p4 workspace -o experimental_boogaloo_4
Changing P4 Client Root to: /home/cruise/hudson/jobs/experimental boogaloo
4/workspace/
Changing P4 Client View to: //cj/personal/jemmer/ExperimentalAndUtility/...
//experimental_boogaloo_4/...
Saving new client experimental_boogaloo_4
[workspace] $ p4 -s client -i
Last sync'd change: 95162
[workspace] $ p4 counter change
[workspace] $ p4 -s changes //experimental_boogaloo_4/...@95163,@95878
Sync'ing workspace to changelist 95878 (forcing sync of unchanged files).
[workspace] $ p4 sync -f //experimental_boogaloo_4/...@95878
Sync complete, took 5173 ms
Triggering build106_bugfix
Triggering build107_bugfix
Finished: FAILURE
--------------------
Started by upstream project "experimental boogaloo 4" build number 1
Building remotely on build106_bugfix
Using remote perforce client: experimental_boogaloo_4-build106
Changing P4 Client Root to: /home/cruise/hudson_bugfix/workspace/experimental
boogaloo 4/label/build106_bugfix/
Changing P4 Client View to: //cj/personal/jemmer/ExperimentalAndUtility/...
//experimental_boogaloo_4-build106/...
Saving new client experimental_boogaloo_4-build106
Last sync'd change: 0
Sync'ing workspace to changelist 95878.
Sync complete, took 68 ms
[maven output complaining that there's no pom.xml]
Finished: FAILURE
--------------------
Started by upstream project "experimental boogaloo 4" build number 1
Building remotely on build107_bugfix
Using remote perforce client: experimental_boogaloo_4-build107
Changing P4 Client Root to: /home/cruise/hudson_bugfix/workspace/experimental
boogaloo 4/label/build107_bugfix/
Changing P4 Client View to: //cj/personal/jemmer/ExperimentalAndUtility/...
//experimental_boogaloo_4-build107/...
Saving new client experimental_boogaloo_4-build107
Last sync'd change: 0
Sync'ing workspace to changelist 95878.
Sync complete, took 4880 ms
[maven output showing successful build]
Finished: SUCCESS
--------------------
If I go onto the 'build106' machine, which didn't sync anything to its
workspace, and issue the following command:
p4 -c experimental_boogaloo_4-build106 sync -n
It does report that it would add all the appropriate files.
So now I'm stumped and it seems my proposed workaround doesn't work and
something will indeed probably have to be done in the Perforce plugin itself.
I got access to the perforce server logs and can now demonstrate that the plugin
is not issuing the commands from the correct servers. Here are all the perforce
commands issued during two change-triggered builds:
2009/10/13 17:21:00 pid 19985
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-workspace -o
hudson_sample_matrix-build106'
2009/10/13 17:21:01 pid 19986
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-changes -m 1
//hudson_sample_matrix-build106/...'
2009/10/13 17:21:06 pid 19992
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-workspace -o
hudson_sample_matrix-build106'
2009/10/13 17:21:06 pid 19993
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-client -i'
2009/10/13 17:21:06 pid 19994
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-counter change'
2009/10/13 17:21:06 pid 19995
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-changes
//hudson_sample_matrix-build106/...@96579,@96584'
2009/10/13 17:21:06 pid 19996
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-describe -s 96584'
2009/10/13 17:21:07 pid 19997
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-sync
//hudson_sample_matrix-build106/...@96584'
2009/10/13 17:21:08 pid 20002
build.automation@hudson_sample_matrix-build108 x.x.x..83 'user-workspace -o
hudson_sample_matrix-build108'
2009/10/13 17:21:08 pid 20003
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-workspace -o
hudson_sample_matrix-build107'
2009/10/13 17:21:08 pid 20004
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-counter change'
2009/10/13 17:21:08 pid 20005
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-workspace -o
hudson_sample_matrix-build106'
2009/10/13 17:21:08 pid 20006
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-counter change'
2009/10/13 17:21:08 pid 20007
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-changes
//hudson_sample_matrix-build107/...@96581,@96584'
2009/10/13 17:21:08 pid 20008
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-changes
//hudson_sample_matrix-build108/...@96581,@96584'
2009/10/13 17:21:08 pid 20009
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-client -i'
2009/10/13 17:21:08 pid 20010
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-sync
//hudson_sample_matrix-build107/...@96584'
2009/10/13 17:21:08 pid 20011
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-sync
//hudson_sample_matrix-build108/...@96584'
2009/10/13 17:21:08 pid 20012
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-counter change'
2009/10/13 17:21:08 pid 20013
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-changes
//hudson_sample_matrix-build106/...@96579,@96584'
2009/10/13 17:21:08 pid 20014
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-describe -s 96584'
2009/10/13 17:21:08 pid 20015
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-sync
//hudson_sample_matrix-build106/...@96584'
2009/10/13 17:31:01 pid 20878
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-workspace -o
hudson_sample_matrix-build106'
2009/10/13 17:31:01 pid 20879
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-changes -m 1
//hudson_sample_matrix-build106/...'
2009/10/13 17:31:06 pid 20885
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-workspace -o
hudson_sample_matrix-build106'
2009/10/13 17:31:06 pid 20886
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-client -i'
2009/10/13 17:31:06 pid 20887
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-counter change'
2009/10/13 17:31:06 pid 20888
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-changes
//hudson_sample_matrix-build106/...@96585,@96586'
2009/10/13 17:31:06 pid 20889
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-describe -s 96585'
2009/10/13 17:31:06 pid 20890
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-sync
//hudson_sample_matrix-build106/...@96585'
2009/10/13 17:31:07 pid 20892
build.automation@hudson_sample_matrix-build108 x.x.x..83 'user-workspace -o
hudson_sample_matrix-build108'
2009/10/13 17:31:07 pid 20893
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-workspace -o
hudson_sample_matrix-build107'
2009/10/13 17:31:07 pid 20894
build.automation@hudson_sample_matrix-build106 x.x.x..79 'user-workspace -o
hudson_sample_matrix-build106'
2009/10/13 17:31:07 pid 20895
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-counter change'
2009/10/13 17:31:08 pid 20896
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-counter change'
2009/10/13 17:31:08 pid 20897
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-client -i'
2009/10/13 17:31:08 pid 20898
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-changes
//hudson_sample_matrix-build108/...@96585,@96586'
2009/10/13 17:31:08 pid 20899
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-changes
//hudson_sample_matrix-build107/...@96585,@96586'
2009/10/13 17:31:08 pid 20903
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-counter change'
2009/10/13 17:31:08 pid 20904
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-sync
//hudson_sample_matrix-build108/...@96586'
2009/10/13 17:31:08 pid 20905
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-describe -s 96585'
2009/10/13 17:31:08 pid 20906
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-changes
//hudson_sample_matrix-build106/...@96585,@96586'
2009/10/13 17:31:08 pid 20907
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-sync
//hudson_sample_matrix-build107/...@96585'
2009/10/13 17:31:08 pid 20908
build.automation@hudson_sample_matrix-build107 x.x.x..81 'user-sync
//hudson_sample_matrix-build106/...@96586'
The build is running on three slaves on three different machines build106
(x.x.x.79), build107 (x.x.x.81), and build108 (x.x.x.83). You can see that the
plugin is using the wrong client from the wrong machine most of the time.
OK, I've tracked it down to the fact that the same PerforceSCM object instance
is being used by all the build threads, and they're all simultaneously modifying
and using the same 'depot' instance variable. If I remove that instance
variable, it actually works. As demonstrated by the following server log, each
slave instance/machine is now issuing independent commands:
2009/10/14 17:51:03 pid 28461 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-workspace -o hudson_sample_matrix-build106'
2009/10/14 17:51:03 pid 28462 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-changes -m 1 //hudson_sample_matrix-build106/...'
2009/10/14 17:51:10 pid 28477 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-workspace -o hudson_sample_matrix-build106'
2009/10/14 17:51:10 pid 28478 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-counter change'
2009/10/14 17:51:10 pid 28479 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-changes //hudson_sample_matrix-build106/...@96623,@96635'
2009/10/14 17:51:10 pid 28480 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-describe -s 96635'
2009/10/14 17:51:10 pid 28481 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-sync //hudson_sample_matrix-build106/...@96635'
2009/10/14 17:51:12 pid 28483 build.automation@hudson_sample_matrix-build108
x.x.x.83 'user-workspace -o hudson_sample_matrix-build108'
2009/10/14 17:51:12 pid 28484 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-workspace -o hudson_sample_matrix-build106'
2009/10/14 17:51:12 pid 28485 build.automation@hudson_sample_matrix-build107
x.x.x.81 'user-workspace -o hudson_sample_matrix-build107'
2009/10/14 17:51:12 pid 28486 build.automation@hudson_sample_matrix-build108
x.x.x.83 'user-counter change'
2009/10/14 17:51:12 pid 28487 build.automation@hudson_sample_matrix-build107
x.x.x.81 'user-counter change'
2009/10/14 17:51:12 pid 28488 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-counter change'
2009/10/14 17:51:12 pid 28489 build.automation@hudson_sample_matrix-build108
x.x.x.83 'user-changes //hudson_sample_matrix-build108/...@96623,@96635'
2009/10/14 17:51:12 pid 28490 build.automation@hudson_sample_matrix-build107
x.x.x.81 'user-changes //hudson_sample_matrix-build107/...@96623,@96635'
2009/10/14 17:51:12 pid 28491 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-changes //hudson_sample_matrix-build106/...@96623,@96635'
2009/10/14 17:51:12 pid 28492 build.automation@hudson_sample_matrix-build108
x.x.x.83 'user-describe -s 96635'
2009/10/14 17:51:12 pid 28493 build.automation@hudson_sample_matrix-build107
x.x.x.81 'user-describe -s 96635'
2009/10/14 17:51:12 pid 28494 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-describe -s 96635'
2009/10/14 17:51:12 pid 28495 build.automation@hudson_sample_matrix-build108
x.x.x.83 'user-sync //hudson_sample_matrix-build108/...@96635'
2009/10/14 17:51:12 pid 28496 build.automation@hudson_sample_matrix-build107
x.x.x.81 'user-sync //hudson_sample_matrix-build107/...@96635'
2009/10/14 17:51:12 pid 28497 build.automation@hudson_sample_matrix-build106
x.x.x.79 'user-sync //hudson_sample_matrix-build106/...@96635'
I also had to change PerforceMailResolver, which was also depending on the
instance attribute, but I'm not sure where this is used to test whether my
change to that class worked.
If I can get commit access to subversion (user 'emmulator') I can submit these
changes.
Code changed in hudson
User: : emmulator
Path:
trunk/hudson/plugins/perforce/src/main/java/hudson/plugins/perforce/HudsonP4Executor.java
trunk/hudson/plugins/perforce/src/main/java/hudson/plugins/perforce/PerforceMailResolver.java
trunk/hudson/plugins/perforce/src/main/java/hudson/plugins/perforce/PerforceSCM.java
trunk/hudson/plugins/perforce/src/test/java/hudson/plugins/perforce/PerforceSCMTest.java
http://fisheye4.cenqua.com/changelog/hudson/?cs=22945
Log:
JENKINS-1022 removed 'depot' instance variable that was being modified by multiple threads simultaneously, this allows perforce commands to be issued from the correct slave nodes; also had master matrix job not alter the perforce client root, so that the child matrix job would get the update
I'm not going to mark this fixed, since I realized that it probably only works
with one axis. Since the plugin still only uses one client-spec per node, if
the matrix includes multiple axes and two are built on the same node, the
perforce clients will probably still interfere with each other. If I get a
chance, I will test that.
I suspect the mentioned changes to PerforceMailResolver has caused it to not
work anymore - please see JENKINS-4933.
Due to the complexities of dealing with multiconfiguration builds, I've changed the behavior of the plugin to always force sync when performing a matrix build. This eliminates the need of having a different client workspace for each and every combination, while still allowing matrix builds to work correctly.
Marking this as WONTFIX because it is a symptom of behavior of the Perforce client. There isn't anything
we can do from the Hudson plugin's standpoint.