Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-1022

Interaction between Perforce plugin and Matrix builds

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • p4-plugin
    • None
    • Platform: All, OS: All

      If I'm using the Perforce plugin and have a setting for this project
      (FPHibernate-Head) with a workspace like this:
      Single Workspace: wyliek.hudson-clients.fphibernate-head.london
      View: //fotech/shared/lib/java/FPHibernate/head/...
      Hudson Home: /scratch/betasx2/wyliek/hudson-home

      And am using a configuration matrix of jdk1.6.0_03 and jdk1.5.0_13, what seems
      to happen is this:
      1) Perforce plugin does a sync to
      /scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace
      2) Matrix runner creates axis directory of
      /scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace/jdk/jdk1.5.0_13
      3) Perforce Plugin changes the root of the workspace to
      /scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace/jdk/jdk1.5.0_13
      and does a p4 sync. P4 says nothing to sync, but there's nothing there.
      4) Ant doesn't find the build.xml file to run.

      Here's the output on the overall project running:
      started
      Performing sync with Perforce for: //fotech/shared/lib/java/FPHibernate/head/...
      Changing P4 Client Root to:
      /scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace/
      Changing P4 Client View to: //fotech/shared/lib/java/FPHibernate/head/...
      //wyliek.hudson-clients.fphibernate-head.london/...
      Last sync'd change: 76672
      Sync'ing workspace to depot.
      ForceSync flag is set, forcing: p4 sync
      //fotech/shared/lib/java/FPHibernate/head/...
      Sync complete, took 3392 MS
      Triggering jdk=jdk1.6.0_03
      Triggering jdk=jdk1.5.0_13
      finished: FAILURE

      Here's the output on the jdk1.6.0_03 axis run:
      started
      Performing sync with Perforce for: //fotech/shared/lib/java/FPHibernate/head/...
      Changing P4 Client Root to:
      /scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace/jdk/jdk1.6.0_03/
      Changing P4 Client View to: //fotech/shared/lib/java/FPHibernate/head/...
      //wyliek.hudson-clients.fphibernate-head.london/...
      Last sync'd change: 76672
      No changes since last build.
      [jdk1.6.0_03] $ /scratch/betasx2/wyliek/apache-ant-1.7.0/bin/ant
      -Djdk=jdk1.6.0_03 tests
      Buildfile: build.xml does not exist!
      Build failed
      Recording test results
      finished: FAILURE

      Here's the output on the jdk1.5.0_13 axis run:
      started
      Performing sync with Perforce for: //fotech/shared/lib/java/FPHibernate/head/...
      Changing P4 Client Root to:
      /scratch/betasx2/wyliek/hudson-home/jobs/FPHibernate-Head/workspace/jdk/jdk1.5.0_13/
      Changing P4 Client View to: //fotech/shared/lib/java/FPHibernate/head/...
      //wyliek.hudson-clients.fphibernate-head.london/...
      Last sync'd change: 76672
      No changes since last build.
      [jdk1.5.0_13] $ /scratch/betasx2/wyliek/apache-ant-1.7.0/bin/ant
      -Djdk=jdk1.5.0_13 tests
      Buildfile: build.xml does not exist!
      Build failed
      Recording test results
      finished: FAILURE

          [JENKINS-1022] Interaction between Perforce plugin and Matrix builds

          kirkwylie created issue -

          kirkwylie added a comment -

          This is using Hudson 1.156 and the Perforce 1.0.5 plugin.

          kirkwylie added a comment - This is using Hudson 1.156 and the Perforce 1.0.5 plugin.

          digerata added a comment -

          The problem here is that Perforce is keeping track of the revisions the workspace has. When the
          workspace was sync'd with the latest in step #1, it will not download any more changes for step #3.

          A possible work around would be to force the sync for step #3. However, that would cause the entire
          project to be brought down every time. Not necessarily the most efficient way to do it. And, there is no
          setting in the plugin to always force a sync.

          I wonder if there is a way in the matrix build to copy the workspace from step #1 out to the
          jdk/jdk1.5.0_13 directory when step #3 executes?

          digerata added a comment - The problem here is that Perforce is keeping track of the revisions the workspace has. When the workspace was sync'd with the latest in step #1, it will not download any more changes for step #3. A possible work around would be to force the sync for step #3. However, that would cause the entire project to be brought down every time. Not necessarily the most efficient way to do it. And, there is no setting in the plugin to always force a sync. I wonder if there is a way in the matrix build to copy the workspace from step #1 out to the jdk/jdk1.5.0_13 directory when step #3 executes?

          digerata added a comment -

          Marking this as WONTFIX because it is a symptom of behavior of the Perforce client. There isn't anything
          we can do from the Hudson plugin's standpoint.

          digerata added a comment - Marking this as WONTFIX because it is a symptom of behavior of the Perforce client. There isn't anything we can do from the Hudson plugin's standpoint.
          digerata made changes -
          Resolution New: Won't Fix [ 2 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]

          emmulator added a comment -

          I've just noted the same behavior in this thread on the users mailing list:

          http://www.nabble.com/Perforce-workspace-issues-in-matrix-project-td25198191.html

          I don't understand how you can say this is just the behavior of Perforce and not
          something you can fix. Hudson already has behavior specific to Perforce in
          creating and managing custom Perforce workspaces, but in this instance it's
          using them incorrectly. Hudson is trying to use the same Perforce workspace for
          two different roots without re-syncing.

          In the thread, I offered two solutions:

          1) Don't run the main job on the same node/slave as the axis jobs – this would
          also have the benefit of not blocking itself when there is only one executor on
          a slave.

          2) Create additional Perforce workspaces for the axis jobs – just as the main
          job syncs to "workspace" and the axis jobs sync to "workspace/axis", the main
          job could use Perforce workspace "x" and the axis jobs could use Perforce
          workspace "x-axis".

          Please reconsider this issue. Otherwise you should probably document on the
          matrix feature that it is not inteded to work with Perforce.

          emmulator added a comment - I've just noted the same behavior in this thread on the users mailing list: http://www.nabble.com/Perforce-workspace-issues-in-matrix-project-td25198191.html I don't understand how you can say this is just the behavior of Perforce and not something you can fix. Hudson already has behavior specific to Perforce in creating and managing custom Perforce workspaces, but in this instance it's using them incorrectly. Hudson is trying to use the same Perforce workspace for two different roots without re-syncing. In the thread, I offered two solutions: 1) Don't run the main job on the same node/slave as the axis jobs – this would also have the benefit of not blocking itself when there is only one executor on a slave. 2) Create additional Perforce workspaces for the axis jobs – just as the main job syncs to "workspace" and the axis jobs sync to "workspace/axis", the main job could use Perforce workspace "x" and the axis jobs could use Perforce workspace "x-axis". Please reconsider this issue. Otherwise you should probably document on the matrix feature that it is not inteded to work with Perforce.
          emmulator made changes -
          Resolution Original: Won't Fix [ 2 ]
          Status Original: Resolved [ 5 ] New: Reopened [ 4 ]

          digerata added a comment -

          Reassigning to issues@hudson as I no longer work on this project.

          digerata added a comment - Reassigning to issues@hudson as I no longer work on this project.
          digerata made changes -
          Status Original: Reopened [ 4 ] New: Open [ 1 ]

          emmulator added a comment -

          I saw other people asking for the ability to tie the parent matrix job to a
          particular node in Issue 936, so I thought I'd try that as a workaround for this
          issue as well. I just copied the "master/slave" section from the
          AbstractItem/configure-common.jelly to the
          MatrixProject/configure-entries.jelly, so that the "Tie this project to a node"
          option would be exposed for Matrix projects as well. Then I tied the Matrix
          project itself to my Master node, and for the axis I just selected two
          "Individual Nodes" on other machines. It looks like it should be working now
          – I see three progress bars in the "Build Executor Status" section as I would
          expect. I can see from the logs that it is now using three distinct perforce
          workspaces, but one of them still doesn't sync. Here are the relevant portions
          of the build logs:

          --------------------

          Started by user anonymous
          Building on master
          Using master perforce client: experimental_boogaloo_4
          [workspace] $ p4 workspace -o experimental_boogaloo_4
          Changing P4 Client Root to: /home/cruise/hudson/jobs/experimental boogaloo
          4/workspace/
          Changing P4 Client View to: //cj/personal/jemmer/ExperimentalAndUtility/...
          //experimental_boogaloo_4/...

          Saving new client experimental_boogaloo_4
          [workspace] $ p4 -s client -i
          Last sync'd change: 95162
          [workspace] $ p4 counter change
          [workspace] $ p4 -s changes //experimental_boogaloo_4/...@95163,@95878
          Sync'ing workspace to changelist 95878 (forcing sync of unchanged files).
          [workspace] $ p4 sync -f //experimental_boogaloo_4/...@95878
          Sync complete, took 5173 ms
          Triggering build106_bugfix
          Triggering build107_bugfix
          Finished: FAILURE

          --------------------

          Started by upstream project "experimental boogaloo 4" build number 1
          Building remotely on build106_bugfix
          Using remote perforce client: experimental_boogaloo_4-build106
          Changing P4 Client Root to: /home/cruise/hudson_bugfix/workspace/experimental
          boogaloo 4/label/build106_bugfix/
          Changing P4 Client View to: //cj/personal/jemmer/ExperimentalAndUtility/...
          //experimental_boogaloo_4-build106/...

          Saving new client experimental_boogaloo_4-build106
          Last sync'd change: 0
          Sync'ing workspace to changelist 95878.
          Sync complete, took 68 ms
          [maven output complaining that there's no pom.xml]
          Finished: FAILURE

          --------------------

          Started by upstream project "experimental boogaloo 4" build number 1
          Building remotely on build107_bugfix
          Using remote perforce client: experimental_boogaloo_4-build107
          Changing P4 Client Root to: /home/cruise/hudson_bugfix/workspace/experimental
          boogaloo 4/label/build107_bugfix/
          Changing P4 Client View to: //cj/personal/jemmer/ExperimentalAndUtility/...
          //experimental_boogaloo_4-build107/...

          Saving new client experimental_boogaloo_4-build107
          Last sync'd change: 0
          Sync'ing workspace to changelist 95878.
          Sync complete, took 4880 ms
          [maven output showing successful build]
          Finished: SUCCESS

          --------------------

          If I go onto the 'build106' machine, which didn't sync anything to its
          workspace, and issue the following command:

          p4 -c experimental_boogaloo_4-build106 sync -n

          It does report that it would add all the appropriate files.

          So now I'm stumped and it seems my proposed workaround doesn't work and
          something will indeed probably have to be done in the Perforce plugin itself.

          emmulator added a comment - I saw other people asking for the ability to tie the parent matrix job to a particular node in Issue 936, so I thought I'd try that as a workaround for this issue as well. I just copied the "master/slave" section from the AbstractItem/configure-common.jelly to the MatrixProject/configure-entries.jelly, so that the "Tie this project to a node" option would be exposed for Matrix projects as well. Then I tied the Matrix project itself to my Master node, and for the axis I just selected two "Individual Nodes" on other machines. It looks like it should be working now – I see three progress bars in the "Build Executor Status" section as I would expect. I can see from the logs that it is now using three distinct perforce workspaces, but one of them still doesn't sync. Here are the relevant portions of the build logs: -------------------- Started by user anonymous Building on master Using master perforce client: experimental_boogaloo_4 [workspace] $ p4 workspace -o experimental_boogaloo_4 Changing P4 Client Root to: /home/cruise/hudson/jobs/experimental boogaloo 4/workspace/ Changing P4 Client View to: //cj/personal/jemmer/ExperimentalAndUtility/... //experimental_boogaloo_4/... Saving new client experimental_boogaloo_4 [workspace] $ p4 -s client -i Last sync'd change: 95162 [workspace] $ p4 counter change [workspace] $ p4 -s changes //experimental_boogaloo_4/...@95163,@95878 Sync'ing workspace to changelist 95878 (forcing sync of unchanged files). [workspace] $ p4 sync -f //experimental_boogaloo_4/...@95878 Sync complete, took 5173 ms Triggering build106_bugfix Triggering build107_bugfix Finished: FAILURE -------------------- Started by upstream project "experimental boogaloo 4" build number 1 Building remotely on build106_bugfix Using remote perforce client: experimental_boogaloo_4-build106 Changing P4 Client Root to: /home/cruise/hudson_bugfix/workspace/experimental boogaloo 4/label/build106_bugfix/ Changing P4 Client View to: //cj/personal/jemmer/ExperimentalAndUtility/... //experimental_boogaloo_4-build106/... Saving new client experimental_boogaloo_4-build106 Last sync'd change: 0 Sync'ing workspace to changelist 95878. Sync complete, took 68 ms [maven output complaining that there's no pom.xml] Finished: FAILURE -------------------- Started by upstream project "experimental boogaloo 4" build number 1 Building remotely on build107_bugfix Using remote perforce client: experimental_boogaloo_4-build107 Changing P4 Client Root to: /home/cruise/hudson_bugfix/workspace/experimental boogaloo 4/label/build107_bugfix/ Changing P4 Client View to: //cj/personal/jemmer/ExperimentalAndUtility/... //experimental_boogaloo_4-build107/... Saving new client experimental_boogaloo_4-build107 Last sync'd change: 0 Sync'ing workspace to changelist 95878. Sync complete, took 4880 ms [maven output showing successful build] Finished: SUCCESS -------------------- If I go onto the 'build106' machine, which didn't sync anything to its workspace, and issue the following command: p4 -c experimental_boogaloo_4-build106 sync -n It does report that it would add all the appropriate files. So now I'm stumped and it seems my proposed workaround doesn't work and something will indeed probably have to be done in the Perforce plugin itself.

            Unassigned Unassigned
            kirkwylie kirkwylie
            Votes:
            2 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: