Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9697

Perforce Plugin running on a Windows 2008R2 slave fails to poll or check out.

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • p4-plugin
    • None
    • Windows 2008R2 Slave.

      We've been trying to configure a Windows 2008R2 slave (our first slave with this OS) to run a job that is already known to run successfully on other Windows XP and 2003 machines. We know that the plugin runs fine on a Windows 2008 master.

      We have confirmed that Perforce has been installed successfully by running all the equivalent commands from the command line and have checked and double checked the configuration and executable locations.

      We suspect that the problem is associated with the fact that with Windows 2008 the command line syntax has become far more strict and that the Perforce plugin uses different code paths to build the command line and execute the perforce process. On the master the plugin uses the Launcher classes to run the Perforce binary but on the slave it appears to use CmdLineExecutor class.

      The following exception is seen in the logs:
      [TEST_util] $ C:\PROGRA~1\Perforce\p4.exe workspace -o hudson_1_glp_4.0_utility_test-3510769
      Caught exception communicating with perforce. Connect to server failed; check $P4PORTcom.tek42.perforce.PerforceException: Connect to server failed; check $P4PORT
      at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:381)
      at com.tek42.perforce.parse.AbstractPerforceTemplate.getPerforceResponse(AbstractPerforceTemplate.java:291)
      at com.tek42.perforce.parse.Workspaces.getWorkspace(Workspaces.java:53)
      at hudson.plugins.perforce.PerforceSCM.getPerforceWorkspace(PerforceSCM.java:1183)
      at hudson.plugins.perforce.PerforceSCM.checkout(PerforceSCM.java:574)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1180)
      at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:506)
      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:422)
      at hudson.model.Run.run(Run.java:1362)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:145)

          [JENKINS-9697] Perforce Plugin running on a Windows 2008R2 slave fails to poll or check out.

          Rob Petti added a comment -

          I can probably shed some light on this for you.

          Basically when it runs

          c:\\Users\\nbem3al\\p4-catch.bat workspace -o jenkins-AO_SISO_AT-smoketest-CIT1-1858902131
          

          It doesn't return any error message that the plugin knows about, so it proceeds as if nothing is wrong. This should throw some kind of parsing error early on, but I guess it's not robust enough. From there it 'changes' the client view from nothing (since nothing intelligible was returned) to the new view. The batch script just eats it, and the plugin continues on happily.

          The Last build changeset is from the previous successful build (I guess you had it on a different slave before) so it's not actually getting it from perforce.

          It's not until it tries to parse the output of the counter command that it fails. There's a check in there to make sure the result is in fact a number. In this case, the batch script was returning "Access is denied." because of the illegal file write.

          So you aren't actually getting any further, the plugin just doesn't know to look for that kind of error.

          So back to the issue at hand.

          If the environment variables are being set correctly, but the connection is still failing that can really only mean one thing. Windows is likely blocking connections from child processes. That would explain why it works by hand, but not through the slave agent.

          This really wouldn't surprise me, because I think you need to add it as an exception to the windows firewall for outgoing connections. It might be working on my test machine because I'm using the Administrator account with the firewall disabled. Tomorrow I'll try to see if I can replicate the results by re-enabling the firewall and running the slave as a general user.

          Rob Petti added a comment - I can probably shed some light on this for you. Basically when it runs c:\\Users\\nbem3al\\p4- catch .bat workspace -o jenkins-AO_SISO_AT-smoketest-CIT1-1858902131 It doesn't return any error message that the plugin knows about, so it proceeds as if nothing is wrong. This should throw some kind of parsing error early on, but I guess it's not robust enough. From there it 'changes' the client view from nothing (since nothing intelligible was returned) to the new view. The batch script just eats it, and the plugin continues on happily. The Last build changeset is from the previous successful build (I guess you had it on a different slave before) so it's not actually getting it from perforce. It's not until it tries to parse the output of the counter command that it fails. There's a check in there to make sure the result is in fact a number. In this case, the batch script was returning "Access is denied." because of the illegal file write. So you aren't actually getting any further, the plugin just doesn't know to look for that kind of error. So back to the issue at hand. If the environment variables are being set correctly, but the connection is still failing that can really only mean one thing. Windows is likely blocking connections from child processes. That would explain why it works by hand, but not through the slave agent. This really wouldn't surprise me, because I think you need to add it as an exception to the windows firewall for outgoing connections. It might be working on my test machine because I'm using the Administrator account with the firewall disabled. Tomorrow I'll try to see if I can replicate the results by re-enabling the firewall and running the slave as a general user.

          Glenn Mayer added a comment -

          Thanks for that explanation. "Who are you that are so wise in the ways of science?"

          BTW, I tried running Jenkins as a master instance on my slave machine, and got the same error. So I can request a firewall rule change, but I haven't been able to find any reference to firewall rules that pertain to child processes. Do you know how I would characterize this Jenkins slave process when configuring a firewall change?

          Glenn Mayer added a comment - Thanks for that explanation. "Who are you that are so wise in the ways of science?" BTW, I tried running Jenkins as a master instance on my slave machine, and got the same error. So I can request a firewall rule change, but I haven't been able to find any reference to firewall rules that pertain to child processes. Do you know how I would characterize this Jenkins slave process when configuring a firewall change?

          Rob Petti added a comment -

          It would just be the p4 executable. You would basically add an outbound rule allowing all connections from the perforce client binary in "Windows Firewall with Advanced Security". Alternatively, you could add a new outbound rule allowing all connections on remote port 1666 (or whatever port number you are using for perforce).

          In any case, I'm not able to replicate your results when using an unprivileged user with the windows firewall enabled with the default settings. Something else must be going on...

          Rob Petti added a comment - It would just be the p4 executable. You would basically add an outbound rule allowing all connections from the perforce client binary in "Windows Firewall with Advanced Security". Alternatively, you could add a new outbound rule allowing all connections on remote port 1666 (or whatever port number you are using for perforce). In any case, I'm not able to replicate your results when using an unprivileged user with the windows firewall enabled with the default settings. Something else must be going on...

          Glenn Mayer added a comment -

          Well, I think we probably are using something other than the default settings; I don't have the privileges to view them.

          There's one more thing I'm unclear on: from the firewall's perspective, what's the difference between me running the "p4 workspace -o <name-of-my-workspace>" command from the powershell versus the same command being run by the Jenkins' slave agent? I will request a firewall rule change, but I want to make sure I'm requesting the right thing.

          Glenn Mayer added a comment - Well, I think we probably are using something other than the default settings; I don't have the privileges to view them. There's one more thing I'm unclear on: from the firewall's perspective, what's the difference between me running the "p4 workspace -o <name-of-my-workspace>" command from the powershell versus the same command being run by the Jenkins' slave agent? I will request a firewall rule change, but I want to make sure I'm requesting the right thing.

          Rob Petti added a comment -

          I'm unclear on that as well, and I admit I'm starting to grasp at straws here. Windows permissions are pretty alien to me, but I think when you run it explicitly it somehow gets more privileges than when you run it as a child of another process. I could be wrong of course...

          Rob Petti added a comment - I'm unclear on that as well, and I admit I'm starting to grasp at straws here. Windows permissions are pretty alien to me, but I think when you run it explicitly it somehow gets more privileges than when you run it as a child of another process. I could be wrong of course...

          Glenn Mayer added a comment -

          Here's some more info: On the slave machine, I created a batch file that sets my environment variables (P4PORT, P4USER, etc), and runs the necessary p4 commands to sync up my workspace (basically doing the most basic work of the p4 plugin). I turned off the p4 plugin in my job config, and set up this batch file to run as a pre-build step. It worked! Now, this is not a workaround, as it in no way replicates what the p4 plugin can do, but it's an interesting diagnostic step. Basically, I can run p4 by executing a batch file from a slave instance, but I can't connect with the p4 plugin.

          Glenn Mayer added a comment - Here's some more info: On the slave machine, I created a batch file that sets my environment variables (P4PORT, P4USER, etc), and runs the necessary p4 commands to sync up my workspace (basically doing the most basic work of the p4 plugin). I turned off the p4 plugin in my job config, and set up this batch file to run as a pre-build step. It worked! Now, this is not a workaround, as it in no way replicates what the p4 plugin can do, but it's an interesting diagnostic step. Basically, I can run p4 by executing a batch file from a slave instance, but I can't connect with the p4 plugin.

          Rob Petti added a comment -

          If it works from a batch file being executed by the slave agent, then how about this:

          @echo off
          p4.exe %*
          

          Try creating that and setting it as the perforce executable in the job configuration. If that still doesn't work, try adding the environment variables to the script.

          Rob Petti added a comment - If it works from a batch file being executed by the slave agent, then how about this: @echo off p4.exe %* Try creating that and setting it as the perforce executable in the job configuration. If that still doesn't work, try adding the environment variables to the script.

          Glenn Mayer added a comment -

          Aha, progress! This approach works, as long as I set P4PORT and P4USER in the script. And I can set the P4TICKETS variable in the script, so I don't have to include a password. I'm not sure why this behaves the way it does, but this is an acceptable workaround. Thank you so much for working with me on this.

          Unfortunately, I'm not out of the woods yet. Once the p4 sync happens, I'm getting a "Classloading from system classloader disabled" message which seems to lead to an error invoking MavenModuleSetBuild, and a failure to parse the POM. I'm still researching, but it seems like this is material for a separate JIRA issue, if I can't figure it out.

          Glenn Mayer added a comment - Aha, progress! This approach works, as long as I set P4PORT and P4USER in the script. And I can set the P4TICKETS variable in the script, so I don't have to include a password. I'm not sure why this behaves the way it does, but this is an acceptable workaround. Thank you so much for working with me on this. Unfortunately, I'm not out of the woods yet. Once the p4 sync happens, I'm getting a "Classloading from system classloader disabled" message which seems to lead to an error invoking MavenModuleSetBuild, and a failure to parse the POM. I'm still researching, but it seems like this is material for a separate JIRA issue, if I can't figure it out.

          Rob Petti added a comment -

          The plugin will automatically issue a 'p4 login' if it finds it's not authenticated, so you shouldn't need to include P4TICKET.

          In any case, that's awesome that you got it working! I'll leave this ticket open in case anyone else runs into it, or someone manages to find the root cause.

          Rob Petti added a comment - The plugin will automatically issue a 'p4 login' if it finds it's not authenticated, so you shouldn't need to include P4TICKET. In any case, that's awesome that you got it working! I'll leave this ticket open in case anyone else runs into it, or someone manages to find the root cause.

          Sorry to be silent - but I've been off for a little while.

          We do have an effective workaround that worked for us. Following a similar approach to that you've been working on.

          We simply created a batch file that looks like this and pointed Jenkins at it instead of the perforce executable. Took us ages to work it out. It seemed that when we set the environment up on the Jenkins server for some reason the SYSTEMROOT variable wasn't being successfully passed through.

          @echo off

          SETLOCAL

          set SYSTEMROOT=C:\Windows

          C:\Progra~1\Perforce\p4.exe %*

          ENDLOCAL

          Robert Boothby added a comment - Sorry to be silent - but I've been off for a little while. We do have an effective workaround that worked for us. Following a similar approach to that you've been working on. We simply created a batch file that looks like this and pointed Jenkins at it instead of the perforce executable. Took us ages to work it out. It seemed that when we set the environment up on the Jenkins server for some reason the SYSTEMROOT variable wasn't being successfully passed through. @echo off SETLOCAL set SYSTEMROOT=C:\Windows C:\Progra~1\Perforce\p4.exe %* ENDLOCAL

            Unassigned Unassigned
            robertboothby Robert Boothby
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: