Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50434

Misbehaviour of feature: Pin the workspace to the build host

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • p4-plugin
    • Jenkins v2.62, p4-plugin v1.8.4, freestyle jobs

      I'm using a Manual (custom view) setup for my freestyle jobs so far (see screenshot of an example job).

      I have problems with the feature Pin the workspace to the build host: when looking at the workspaces I often see that workspaces get the wrong host set. This leads to synching problems and in the end typically to (C++) linker errors.

      I've created a Python script report_p4_workspaces_with_wrong_host_assignment.py for this. It gives me e.g.:

      jenkins_frinavci_FRI00040598D_spine2d.w64-vc14-qt56x.debug-coverage: FRIjenkins04
      jenkins_frinavci_FRI00040598D_stp-makospine.w64-vc14-qt5.9.4-release: FRNAVDEV01
      jenkins_frinavci_FRI00903207D_shared-communications-sfb_unit.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_FRI00903207D_shared-render_engine.w64-vc14-qt56x.debug-coverage: FRINAVDEV01
      jenkins_frinavci_frijenkins04_ce.win64-vc14-qt56: FRNAVDEV01
      jenkins_frinavci_frijenkins04_shared-report_engine.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins04_Tools-f8scope.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins06_Linux_Jobs-shared-threading-testframe.Linux: frijenkins11
      jenkins_frinavci_frijenkins06_Linux_Jobs-spine2d.linux64-gcc-qt56x: frijenkins11
      jenkins_frinavci_frijenkins08_fess.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins08_plugins-import.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins09_plugins-test_driver_plugin.w32-vc14-qt56x: FRINAVDEV01
      jenkins_frinavci_frijenkins09_plugins-videolib_ng.w3264-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins09_shared-threed.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins11_Linux_Jobs-cranial.linux64-gcc-qt56x-debugmako: frijenkins06
      jenkins_frinavci_frijenkins11_Linux_Jobs-cranial.linux64-gcc-qt56x-debugquick: frijenkins06
      jenkins_frinavci_frijenkins11_Linux_Jobs-cranial.linux64-gcc-qt56x-releasemako: frijenkins06
      jenkins_frinavci_frijenkins11_Linux_Jobs-cranial.linux64-gcc-qt56x-releasequick: frijenkins06
      jenkins_frinavci_frijenkins11_Linux_Jobs-shared-classlib.linux64-gcc-qt56x: frijenkins06
      jenkins_frinavci_frijenkins11_Linux_Jobs-shared-sys_util.linux64-gcc-qt56x: frijenkins06
      jenkins_frinavci_frinavdev01_shared-communications-udp_multicast.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frinavdev01_shared-qt_util.w64-vc14-qt56x: FRIJENKINS09
      jenkins_frinavci_frinavdev01_shared-threading.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_comp-image_acquisition_engine.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_multi-shared-CEMsg: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_plugins-plugin_manager.w32-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_plugins-plugin_manager.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_plugins-videolib.w3264-vc14-qt56x: FRIJENKINS09
      jenkins_frinavci_Haswell-NUC_shared-CEMsg.w3264-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_shared-communications-socket.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_shared-meshlib.w3264-vc14-qt56x: FRIjenkins04
      jenkins_frinavci_Haswell-NUC_shared-smart_tool.w32-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_shared-tool_description.w64-vc14-qt56x: FRIjenkins04
      jenkins_frinavci_Haswell-NUC_Tools-iso_c_simulator.w64-vc14-qt56x: FRNAVDEV01
      

      I'm going to attach a screenshot of the SCM configuration next.

          [JENKINS-50434] Misbehaviour of feature: Pin the workspace to the build host

          So far i regularly detect this problem using the script and fix it manually.

          Heiko Nardmann added a comment - So far i regularly detect this problem using the script and fix it manually.

          Paul Allen added a comment -

          What value are you seeing as the wrong host? The Jenkins master or another slave?  I presume that freestyle job is fixed to build on one slave.

          Paul Allen added a comment - What value are you seeing as the wrong host? The Jenkins master or another slave?  I presume that freestyle job is fixed to build on one slave.

          Exampletaken from above:

          jenkins_frinavci_FRI00040598D_spine2d.w64-vc14-qt56x.debug-coverage: FRIjenkins04
          

          The name of the workspace is jenkins_frinavci_FRI00040598D_spine2d.w64-vc14-qt56x.debug-coverage. This is derived from the workspace name format jenkins_frinavci_${NODE_NAME}_${JOB_NAME}. So I would expect that the host entered into the client spec is FRI00040598D. But it is FRIjenkins04.

          Does this explain my problem?

          Heiko Nardmann added a comment - Exampletaken from above: jenkins_frinavci_FRI00040598D_spine2d.w64-vc14-qt56x.debug-coverage: FRIjenkins04 The name of the workspace is jenkins_frinavci_FRI00040598D_spine2d.w64-vc14-qt56x.debug-coverage . This is derived from the workspace name format jenkins_frinavci_${NODE_NAME}_${JOB_NAME }. So I would expect that the host entered into the client spec is FRI00040598D . But it is FRIjenkins04 . Does this explain my problem?

          Both FRI00040598D and FRIjenkins04 are slave names.

          Heiko Nardmann added a comment - Both FRI00040598D and FRIjenkins04 are slave names.

          If necessary: we are a commercial Perforce customer. So we could open an official ticket if this helps.

          Heiko Nardmann added a comment - If necessary: we are a commercial Perforce customer. So we could open an official ticket if this helps.

          Paul Allen added a comment -

          A support ticket would be great, our support team can work with you on a reproduction. It seems strange a different node is used to create the workspace.  Are you sure the Jenkins Job is restricted to one slave?

          Paul Allen added a comment - A support ticket would be great, our support team can work with you on a reproduction. It seems strange a different node is used to create the workspace.  Are you sure the Jenkins Job is restricted to one slave?

          Heiko Nardmann added a comment - - edited

          No, the job is not restricted to one slave. That is why I added NODE_NAME to the workspace name ... to have a unique client spec per node ...

          Hmmm ... time to talk about the intended behaviour of that feature Pin the workspace to the build host ...

          My perception is that the following activities happen (based on jenkins_${NODE_NAME}_${JOB_NAME} as the workspace name format):

          1. initially the job MyJob runs on SlaveA => a new client spec is created with jenkins_SlaveA_MyJob
          2. second time it runs on SlaveB => a new client spec is created with jenkins_SlaveB_MyJob
          3. third time it runs on SlaveA again => the client spec jenkins_SlaveA_MyJob is used

          My understanding is that for each (Slave,Job) pair a dedicated client spec is created?

          Totally wrong?

          Heiko Nardmann added a comment - - edited No, the job is not restricted to one slave. That is why I added NODE_NAME to the workspace name ... to have a unique client spec per node ... Hmmm ... time to talk about the intended behaviour of that feature Pin the workspace to the build host ... My perception is that the following activities happen (based on jenkins_${NODE_NAME}_${JOB_NAME } as the workspace name format): initially the job MyJob runs on SlaveA => a new client spec is created with jenkins_SlaveA_MyJob second time it runs on SlaveB => a new client spec is created with jenkins_SlaveB_MyJob third time it runs on SlaveA again => the client spec jenkins_SlaveA_MyJob is used My understanding is that for each (Slave,Job) pair a dedicated client spec is created? Totally wrong?

          Is the workspace name created only once or is that expression evaluated each time?

          Heiko Nardmann added a comment - Is the workspace name created only once or is that expression evaluated each time?

          Paul Allen added a comment -

          It's been a while since I looked at the host feature - so a quick look at the code (AbstractTask):

          public Workspace setEnvironment(Run<?, ?> run, Workspace wsType, FilePath buildWorkspace)
                throws IOException, InterruptedException {
          
             Workspace ws = (Workspace) wsType.clone();
          
             // Set Node environment
             EnvVars envVars = run.getEnvironment(listener);
             String nodeName = NodeHelper.getNodeName(buildWorkspace);
             envVars.put("NODE_NAME", envVars.get("NODE_NAME", nodeName));
             String executor = ExecutorHelper.getExecutorID(buildWorkspace);
             envVars.put("EXECUTOR_NUMBER", envVars.get("EXECUTOR_NUMBER", executor));
          
             ws.setExpand(envVars);
          
             // Set workspace root (check for parallel execution)
             String root = buildWorkspace.getRemote();
             if (root.contains("@")) {
                root = root.replace("@", "%40");
             }
             ws.setRootPath(root);
          
             if (ws.isPinHost()) {
                String hostname = getHostName(buildWorkspace);
                ws.setHostName(hostname);
             } else {
                ws.setHostName("");
             }
             return ws;
          }

          The setEnvironment method is called every time the workspace is used (that could be for checkout, cleanup, publish, etc...) there is a remote task to evaluate the host 'getHostName(buildWorkspace)', perhaps Jenkins is executing this on a different host?  The simplest option to fix this is to use the 'nodeName', however the user MUST set the node name the same as the host (I know that seems obvious, but you would be amazed at what some users will configure).

          Paul Allen added a comment - It's been a while since I looked at the host feature - so a quick look at the code (AbstractTask): public Workspace setEnvironment(Run<?, ?> run, Workspace wsType, FilePath buildWorkspace) throws IOException, InterruptedException { Workspace ws = (Workspace) wsType.clone(); // Set Node environment EnvVars envVars = run.getEnvironment(listener); String nodeName = NodeHelper.getNodeName(buildWorkspace); envVars.put( "NODE_NAME" , envVars.get( "NODE_NAME" , nodeName)); String executor = ExecutorHelper.getExecutorID(buildWorkspace); envVars.put( "EXECUTOR_NUMBER" , envVars.get( "EXECUTOR_NUMBER" , executor)); ws.setExpand(envVars); // Set workspace root (check for parallel execution) String root = buildWorkspace.getRemote(); if (root.contains( "@" )) { root = root.replace( "@" , "%40" ); } ws.setRootPath(root); if (ws.isPinHost()) { String hostname = getHostName(buildWorkspace); ws.setHostName(hostname); } else { ws.setHostName(""); } return ws; } The setEnvironment method is called every time the workspace is used (that could be for checkout, cleanup, publish, etc...) there is a remote task to evaluate the host 'getHostName(buildWorkspace)', perhaps Jenkins is executing this on a different host?  The simplest option to fix this is to use the 'nodeName', however the user MUST set the node name the same as the host (I know that seems obvious, but you would be amazed at what some users will configure).

          Not sure whether it is related: why do I see this inside the jenkins.log:

          [root@frinavci01 jenkins]# grep appman.w32-vc14-qt56x jenkins.log | tail -30
          INFO: P4: poking: appman.w32-vc14-qt56x
          INFO: P4: probing: appman.w32-vc14-qt56x
          INFO: P4: poking: appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved.
          INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved.
          INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved.
          INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          

          Note: the job appman.w32-vc14-qt56x is disabled!!!

          Heiko Nardmann added a comment - Not sure whether it is related: why do I see this inside the jenkins.log : [root@frinavci01 jenkins]# grep appman.w32-vc14-qt56x jenkins.log | tail -30 INFO: P4: poking: appman.w32-vc14-qt56x INFO: P4: probing: appman.w32-vc14-qt56x INFO: P4: poking: appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved. INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved. INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved. INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x Note: the job appman.w32-vc14-qt56x is disabled!!!

          Maybe one could add a check inside P4Hook.java whether a job is enabled before doing anything?

          Or dou you expect to get only active jobs inside the jobs parameter of probeJobs()?

          Heiko Nardmann added a comment - Maybe one could add a check inside P4Hook.java whether a job is enabled before doing anything? Or dou you expect to get only active jobs inside the jobs parameter of probeJobs() ?

          Paul Allen added a comment -

          Yes it looks like an oversight on the triggered probe option.  Please can you raise that as a separate JIRA.

          Paul Allen added a comment - Yes it looks like an oversight on the triggered probe option.  Please can you raise that as a separate JIRA.

          See JENKINS-50634 for the separate improvement issue.

          Heiko Nardmann added a comment - See JENKINS-50634 for the separate improvement issue.

          heiko_nardmann thanks for you continuous valuable contribution to perforce Jenkins plugin. In referee to your question regarding raising a support ticket, since you are a paying Perforce customer, please raise a support ticket whenever you raise a Perforce related issue on this forum. While widely impacting issues raised by anyone get high priority on merit, we do unashamedly provide special priority to our paying customers

          W Basu Perforce added a comment - heiko_nardmann thanks for you continuous valuable contribution to perforce Jenkins plugin. In referee to your question regarding raising a support ticket, since you are a paying Perforce customer, please raise a support ticket whenever you raise a Perforce related issue on this forum. While widely impacting issues raised by anyone get high priority on merit, we do unashamedly provide special priority to our paying customers

          Karl Wirth added a comment -

          The change in Host field is being triggered by the Workspace cleanup:

             https://github.com/jenkinsci/jenkins/blob/a1cad0b839828e15d21a41c8ed83194e4c3eec9d/core/src/main/java/hudson/model/WorkspaceCleanupThread.java#L136

          This calls:

          https://github.com/jenkinsci/p4-plugin/blob/9c7cbf52a3a1468a7cd517a00f13e11ccb16cce3/src/main/java/org/jenkinsci/plugins/p4/PerforceScm.java#L629

          Heiko proved this by comparing the spec depot updates to the time of the automated cleaning seen in '/var/lib/jenkins/logs/task/Workspace\ clean-up.log:

          Started at Thu Apr 19 17:11:25 UTC 2018
          Multibranch_test
          Checking FilteredPipeline
          Checking Freestyle_test
          
          

          Moving to VERIFIED.

           

           

          Karl Wirth added a comment - The change in Host field is being triggered by the Workspace cleanup:    https://github.com/jenkinsci/jenkins/blob/a1cad0b839828e15d21a41c8ed83194e4c3eec9d/core/src/main/java/hudson/model/WorkspaceCleanupThread.java#L136 This calls: https://github.com/jenkinsci/p4-plugin/blob/9c7cbf52a3a1468a7cd517a00f13e11ccb16cce3/src/main/java/org/jenkinsci/plugins/p4/PerforceScm.java#L629 Heiko proved this by comparing the spec depot updates to the time of the automated cleaning seen in '/var/lib/jenkins/logs/task/Workspace\ clean-up.log: Started at Thu Apr 19 17:11:25 UTC 2018 Multibranch_test Checking FilteredPipeline Checking Freestyle_test Moving to VERIFIED.    

          Karl Wirth added a comment -

          Proving very difficult to fix. The concept of a job having different workspaces is alien to Jenkins so the correct information is currently not (and unlikely to ever be) provided to P4Jenkins to clean up properly.

           

          For now we are going to have to recommend that pin at build host is no longer used and {NODE_NAME} is used in workspace name instead.

          Karl Wirth added a comment - Proving very difficult to fix. The concept of a job having different workspaces is alien to Jenkins so the correct information is currently not (and unlikely to ever be) provided to P4Jenkins to clean up properly.   For now we are going to have to recommend that pin at build host is no longer used and {NODE_NAME} is used in workspace name instead.

          Doc required.

          W Basu Perforce added a comment - Doc required.

          Heiko Nardmann added a comment - Maybe https://github.com/jenkinsci/p4-plugin/blob/master/docs/WORKSPACEMANUAL.md needs to be changed here?

            p4karl Karl Wirth
            heiko_nardmann Heiko Nardmann
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: