Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50434

Misbehaviour of feature: Pin the workspace to the build host

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • p4-plugin
    • Jenkins v2.62, p4-plugin v1.8.4, freestyle jobs

      I'm using a Manual (custom view) setup for my freestyle jobs so far (see screenshot of an example job).

      I have problems with the feature Pin the workspace to the build host: when looking at the workspaces I often see that workspaces get the wrong host set. This leads to synching problems and in the end typically to (C++) linker errors.

      I've created a Python script report_p4_workspaces_with_wrong_host_assignment.py for this. It gives me e.g.:

      jenkins_frinavci_FRI00040598D_spine2d.w64-vc14-qt56x.debug-coverage: FRIjenkins04
      jenkins_frinavci_FRI00040598D_stp-makospine.w64-vc14-qt5.9.4-release: FRNAVDEV01
      jenkins_frinavci_FRI00903207D_shared-communications-sfb_unit.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_FRI00903207D_shared-render_engine.w64-vc14-qt56x.debug-coverage: FRINAVDEV01
      jenkins_frinavci_frijenkins04_ce.win64-vc14-qt56: FRNAVDEV01
      jenkins_frinavci_frijenkins04_shared-report_engine.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins04_Tools-f8scope.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins06_Linux_Jobs-shared-threading-testframe.Linux: frijenkins11
      jenkins_frinavci_frijenkins06_Linux_Jobs-spine2d.linux64-gcc-qt56x: frijenkins11
      jenkins_frinavci_frijenkins08_fess.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins08_plugins-import.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins09_plugins-test_driver_plugin.w32-vc14-qt56x: FRINAVDEV01
      jenkins_frinavci_frijenkins09_plugins-videolib_ng.w3264-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins09_shared-threed.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frijenkins11_Linux_Jobs-cranial.linux64-gcc-qt56x-debugmako: frijenkins06
      jenkins_frinavci_frijenkins11_Linux_Jobs-cranial.linux64-gcc-qt56x-debugquick: frijenkins06
      jenkins_frinavci_frijenkins11_Linux_Jobs-cranial.linux64-gcc-qt56x-releasemako: frijenkins06
      jenkins_frinavci_frijenkins11_Linux_Jobs-cranial.linux64-gcc-qt56x-releasequick: frijenkins06
      jenkins_frinavci_frijenkins11_Linux_Jobs-shared-classlib.linux64-gcc-qt56x: frijenkins06
      jenkins_frinavci_frijenkins11_Linux_Jobs-shared-sys_util.linux64-gcc-qt56x: frijenkins06
      jenkins_frinavci_frinavdev01_shared-communications-udp_multicast.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_frinavdev01_shared-qt_util.w64-vc14-qt56x: FRIJENKINS09
      jenkins_frinavci_frinavdev01_shared-threading.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_comp-image_acquisition_engine.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_multi-shared-CEMsg: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_plugins-plugin_manager.w32-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_plugins-plugin_manager.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_plugins-videolib.w3264-vc14-qt56x: FRIJENKINS09
      jenkins_frinavci_Haswell-NUC_shared-CEMsg.w3264-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_shared-communications-socket.w64-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_shared-meshlib.w3264-vc14-qt56x: FRIjenkins04
      jenkins_frinavci_Haswell-NUC_shared-smart_tool.w32-vc14-qt56x: FRNAVDEV01
      jenkins_frinavci_Haswell-NUC_shared-tool_description.w64-vc14-qt56x: FRIjenkins04
      jenkins_frinavci_Haswell-NUC_Tools-iso_c_simulator.w64-vc14-qt56x: FRNAVDEV01
      

      I'm going to attach a screenshot of the SCM configuration next.

          [JENKINS-50434] Misbehaviour of feature: Pin the workspace to the build host

          Paul Allen added a comment -

          It's been a while since I looked at the host feature - so a quick look at the code (AbstractTask):

          public Workspace setEnvironment(Run<?, ?> run, Workspace wsType, FilePath buildWorkspace)
                throws IOException, InterruptedException {
          
             Workspace ws = (Workspace) wsType.clone();
          
             // Set Node environment
             EnvVars envVars = run.getEnvironment(listener);
             String nodeName = NodeHelper.getNodeName(buildWorkspace);
             envVars.put("NODE_NAME", envVars.get("NODE_NAME", nodeName));
             String executor = ExecutorHelper.getExecutorID(buildWorkspace);
             envVars.put("EXECUTOR_NUMBER", envVars.get("EXECUTOR_NUMBER", executor));
          
             ws.setExpand(envVars);
          
             // Set workspace root (check for parallel execution)
             String root = buildWorkspace.getRemote();
             if (root.contains("@")) {
                root = root.replace("@", "%40");
             }
             ws.setRootPath(root);
          
             if (ws.isPinHost()) {
                String hostname = getHostName(buildWorkspace);
                ws.setHostName(hostname);
             } else {
                ws.setHostName("");
             }
             return ws;
          }

          The setEnvironment method is called every time the workspace is used (that could be for checkout, cleanup, publish, etc...) there is a remote task to evaluate the host 'getHostName(buildWorkspace)', perhaps Jenkins is executing this on a different host?  The simplest option to fix this is to use the 'nodeName', however the user MUST set the node name the same as the host (I know that seems obvious, but you would be amazed at what some users will configure).

          Paul Allen added a comment - It's been a while since I looked at the host feature - so a quick look at the code (AbstractTask): public Workspace setEnvironment(Run<?, ?> run, Workspace wsType, FilePath buildWorkspace) throws IOException, InterruptedException { Workspace ws = (Workspace) wsType.clone(); // Set Node environment EnvVars envVars = run.getEnvironment(listener); String nodeName = NodeHelper.getNodeName(buildWorkspace); envVars.put( "NODE_NAME" , envVars.get( "NODE_NAME" , nodeName)); String executor = ExecutorHelper.getExecutorID(buildWorkspace); envVars.put( "EXECUTOR_NUMBER" , envVars.get( "EXECUTOR_NUMBER" , executor)); ws.setExpand(envVars); // Set workspace root (check for parallel execution) String root = buildWorkspace.getRemote(); if (root.contains( "@" )) { root = root.replace( "@" , "%40" ); } ws.setRootPath(root); if (ws.isPinHost()) { String hostname = getHostName(buildWorkspace); ws.setHostName(hostname); } else { ws.setHostName(""); } return ws; } The setEnvironment method is called every time the workspace is used (that could be for checkout, cleanup, publish, etc...) there is a remote task to evaluate the host 'getHostName(buildWorkspace)', perhaps Jenkins is executing this on a different host?  The simplest option to fix this is to use the 'nodeName', however the user MUST set the node name the same as the host (I know that seems obvious, but you would be amazed at what some users will configure).

          Not sure whether it is related: why do I see this inside the jenkins.log:

          [root@frinavci01 jenkins]# grep appman.w32-vc14-qt56x jenkins.log | tail -30
          INFO: P4: poking: appman.w32-vc14-qt56x
          INFO: P4: probing: appman.w32-vc14-qt56x
          INFO: P4: poking: appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved.
          INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved.
          INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved.
          INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x
          

          Note: the job appman.w32-vc14-qt56x is disabled!!!

          Heiko Nardmann added a comment - Not sure whether it is related: why do I see this inside the jenkins.log : [root@frinavci01 jenkins]# grep appman.w32-vc14-qt56x jenkins.log | tail -30 INFO: P4: poking: appman.w32-vc14-qt56x INFO: P4: probing: appman.w32-vc14-qt56x INFO: P4: poking: appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved. INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved. INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: Client jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x saved. INFO: ... client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: (p4):cmd:... p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: p4 client -o jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x INFO: P4 Task: cleanup client: jenkins_frinavci_frijenkins09_appman.w32-vc14-qt56x Note: the job appman.w32-vc14-qt56x is disabled!!!

          Maybe one could add a check inside P4Hook.java whether a job is enabled before doing anything?

          Or dou you expect to get only active jobs inside the jobs parameter of probeJobs()?

          Heiko Nardmann added a comment - Maybe one could add a check inside P4Hook.java whether a job is enabled before doing anything? Or dou you expect to get only active jobs inside the jobs parameter of probeJobs() ?

          Paul Allen added a comment -

          Yes it looks like an oversight on the triggered probe option.  Please can you raise that as a separate JIRA.

          Paul Allen added a comment - Yes it looks like an oversight on the triggered probe option.  Please can you raise that as a separate JIRA.

          See JENKINS-50634 for the separate improvement issue.

          Heiko Nardmann added a comment - See JENKINS-50634 for the separate improvement issue.

          heiko_nardmann thanks for you continuous valuable contribution to perforce Jenkins plugin. In referee to your question regarding raising a support ticket, since you are a paying Perforce customer, please raise a support ticket whenever you raise a Perforce related issue on this forum. While widely impacting issues raised by anyone get high priority on merit, we do unashamedly provide special priority to our paying customers

          W Basu Perforce added a comment - heiko_nardmann thanks for you continuous valuable contribution to perforce Jenkins plugin. In referee to your question regarding raising a support ticket, since you are a paying Perforce customer, please raise a support ticket whenever you raise a Perforce related issue on this forum. While widely impacting issues raised by anyone get high priority on merit, we do unashamedly provide special priority to our paying customers

          Karl Wirth added a comment -

          The change in Host field is being triggered by the Workspace cleanup:

             https://github.com/jenkinsci/jenkins/blob/a1cad0b839828e15d21a41c8ed83194e4c3eec9d/core/src/main/java/hudson/model/WorkspaceCleanupThread.java#L136

          This calls:

          https://github.com/jenkinsci/p4-plugin/blob/9c7cbf52a3a1468a7cd517a00f13e11ccb16cce3/src/main/java/org/jenkinsci/plugins/p4/PerforceScm.java#L629

          Heiko proved this by comparing the spec depot updates to the time of the automated cleaning seen in '/var/lib/jenkins/logs/task/Workspace\ clean-up.log:

          Started at Thu Apr 19 17:11:25 UTC 2018
          Multibranch_test
          Checking FilteredPipeline
          Checking Freestyle_test
          
          

          Moving to VERIFIED.

           

           

          Karl Wirth added a comment - The change in Host field is being triggered by the Workspace cleanup:    https://github.com/jenkinsci/jenkins/blob/a1cad0b839828e15d21a41c8ed83194e4c3eec9d/core/src/main/java/hudson/model/WorkspaceCleanupThread.java#L136 This calls: https://github.com/jenkinsci/p4-plugin/blob/9c7cbf52a3a1468a7cd517a00f13e11ccb16cce3/src/main/java/org/jenkinsci/plugins/p4/PerforceScm.java#L629 Heiko proved this by comparing the spec depot updates to the time of the automated cleaning seen in '/var/lib/jenkins/logs/task/Workspace\ clean-up.log: Started at Thu Apr 19 17:11:25 UTC 2018 Multibranch_test Checking FilteredPipeline Checking Freestyle_test Moving to VERIFIED.    

          Karl Wirth added a comment -

          Proving very difficult to fix. The concept of a job having different workspaces is alien to Jenkins so the correct information is currently not (and unlikely to ever be) provided to P4Jenkins to clean up properly.

           

          For now we are going to have to recommend that pin at build host is no longer used and {NODE_NAME} is used in workspace name instead.

          Karl Wirth added a comment - Proving very difficult to fix. The concept of a job having different workspaces is alien to Jenkins so the correct information is currently not (and unlikely to ever be) provided to P4Jenkins to clean up properly.   For now we are going to have to recommend that pin at build host is no longer used and {NODE_NAME} is used in workspace name instead.

          Doc required.

          W Basu Perforce added a comment - Doc required.

          Heiko Nardmann added a comment - Maybe https://github.com/jenkinsci/p4-plugin/blob/master/docs/WORKSPACEMANUAL.md needs to be changed here?

            p4karl Karl Wirth
            heiko_nardmann Heiko Nardmann
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: