Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-16176

ShiningPanda PATH update uses wrong separator on Windows

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • shiningpanda-plugin
    • None
    • Linux master, Windows XP slave with Python 2.7

      When running ShiningPanda on a Windows slave, the plugin (or virtualenv?) updates the PATH to include the virtualenv directory.

      0. Install ShiningPanda plugin (this result is for v1.15)
      1. Create a build with a Virtualenv build step
      2. Use the Shell or XShell command with a minimal script:

      set 
      pip.exe install -r pkg_requirements.txt
      

      The build fails because the pip command can not be found. The output shows the Path environment variable uses the : (colon) separator for the virtualenv Scripts directory. On Windows, the PATH separator should be ; (semicolon).

      [1-SMSDK-apps-venv] $ cmd.exe /c call C:\DOCUME~1\builder\LOCALS~1\Temp\shiningpanda4030889806584404855.bat
      
      c:\jenkins-dev\workspace\1-SMSDK-apps-venv>set
      ... [edited for relevance] ...
      Path=c:\jenkins-dev\workspace\1-SMSDK-apps-venv:c:\jenkins-dev\shiningpanda\deb23754\virtualenvs\d41d8cd9\Scripts:C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\FLEXlm;C:\Program Files\CollabNet\Subversion Client;c:\cygwin\bin;c:\cygwin\usr\bin;C:\Program Files\Java\jre6\bin
      PYTHON_EXE=c:\jenkins-dev\shiningpanda\deb23754\virtualenvs\d41d8cd9\Scripts\python.exe
      VIRTUAL_ENV=c:\jenkins-dev\shiningpanda\deb23754\virtualenvs\d41d8cd9
      WORKSPACE=c:\jenkins-dev\workspace\1-SMSDK-apps-venv
      
      c:\jenkins-dev\workspace\1-SMSDK-apps-venv>pip.exe install -r pkg_requirements.pip 
      'pip.exe' is not recognized as an internal or external command,
      operable program or batch file.
      

          [JENKINS-16176] ShiningPanda PATH update uses wrong separator on Windows

          Chris Withers added a comment -

          C Girard: What suggests to you that this code is running on the master not the slave? How would the code be moved to execute on the slave?

          Chris Withers added a comment - C Girard: What suggests to you that this code is running on the master not the slave? How would the code be moved to execute on the slave?

          Chris Withers added a comment -

          Chris Withers added a comment - Latest diggings on this are here: https://groups.google.com/forum/#!topic/jenkinsci-dev/VgCqabekqPM

          C Girard added a comment -

          I think the analysis from the prior comment is the same reasoning I had for assuming this was running on the master. The path separator is either coming from the "File" class or from the "platform" instance which appears to only come from Platform.current(). Note that "platform" is a private class member, so this rules out subclasses having overridden this or manually setting prior to copy construction.

          In either case, getting a colon back would indicate the code is running on a linux machine. In our setup, we have a linux master and windows slave, so it seems like the code is running on the master.

          C Girard added a comment - I think the analysis from the prior comment is the same reasoning I had for assuming this was running on the master. The path separator is either coming from the "File" class or from the "platform" instance which appears to only come from Platform.current(). Note that "platform" is a private class member, so this rules out subclasses having overridden this or manually setting prior to copy construction. In either case, getting a colon back would indicate the code is running on a linux machine. In our setup, we have a linux master and windows slave, so it seems like the code is running on the master.

          C Girard added a comment -

          I just realized I hadn't considered that the instance might have been transferred from the slave. Java is not my "first language", so I hadn't realized that serialization is somewhat built in. EnvVars extends TreeMap which implements Serializable, so that very well could be happening here.

          In any case, it seems that wherever the code is actually being run, the EnvVars instance being used originated on the master.

          C Girard added a comment - I just realized I hadn't considered that the instance might have been transferred from the slave. Java is not my "first language", so I hadn't realized that serialization is somewhat built in. EnvVars extends TreeMap which implements Serializable, so that very well could be happening here. In any case, it seems that wherever the code is actually being run, the EnvVars instance being used originated on the master.

          For a solution that uses Python (which we know we've already got installed) and doesn't need sed, the following worked for me. No guarantees.

          REM Fix for Jenkins bug #16176 (Wrong PATH separator used on Windows)
          for /f "delims=" %%a in ('%PYTHON_EXE% -c "import os; print(os.environ['PATH'].replace(':c:', ';c:').replace(':C:', ';C:'))"') do @set "PATH=%%a"

          Doug Goldstein added a comment - For a solution that uses Python (which we know we've already got installed) and doesn't need sed, the following worked for me. No guarantees. REM Fix for Jenkins bug #16176 (Wrong PATH separator used on Windows) for /f "delims=" %%a in ('%PYTHON_EXE% -c "import os; print(os.environ ['PATH'] .replace(':c:', ';c:').replace(':C:', ';C:'))"') do @set "PATH=%%a"

          David Haney added a comment -

          For both of the workarounds involving for, it appears that these only work if the build environment is Shell. In the XShell environment, the for /f will be replaced by for \f, resulting in the for call failing. It appears that the plugin is interpreting the /f as a path and converting it to use Windows separators.

          David Haney added a comment - For both of the workarounds involving for , it appears that these only work if the build environment is Shell . In the XShell environment, the for /f will be replaced by for \f , resulting in the for call failing. It appears that the plugin is interpreting the /f as a path and converting it to use Windows separators.

          Colin Bennett added a comment -

          Weird – I am seeing an incorrect path in the converse way – there are semicolons getting inserted into the Linux slave's PATH. I have a Windows master and Linux slave. The PATH variable is set like this

          /usr/lib/jvm/java-7-openjdk-amd64/bin:/var/jenkins/workspace/testproject;/var/jenkins/shiningpanda/jobs/ef9d92b6/virtualenvs/4a2c09ff/bin;/var/jenkins/bin:/var/jenkins/bin/cmake-3.5.0-rc2-Linux-x86_64/bin:/usr/local/bin:/usr/bin:/bin

          Colin Bennett added a comment - Weird – I am seeing an incorrect path in the converse way – there are semicolons getting inserted into the Linux slave's PATH. I have a Windows master and Linux slave. The PATH variable is set like this /usr/lib/jvm/java-7-openjdk-amd64/bin:/var/jenkins/workspace/testproject;/var/jenkins/shiningpanda/jobs/ef9d92b6/virtualenvs/4a2c09ff/bin;/var/jenkins/bin:/var/jenkins/bin/cmake-3.5.0-rc2-Linux-x86_64/bin:/usr/local/bin:/usr/bin:/bin

          Colin Bennett added a comment -

          Look at this very similar bug JENKINS-14515, especially this code change that uses toComputer(): 8b09f8bbff45...6f337351a745:

          -                EnvVars vars = new EnvVars();
          +                EnvVars vars = node.toComputer().getEnvironment();
          

          Colin Bennett added a comment - Look at this very similar bug JENKINS-14515 , especially this code change that uses toComputer() : 8b09f8bbff45...6f337351a745 : - EnvVars vars = new EnvVars(); + EnvVars vars = node.toComputer().getEnvironment();

          Colin Bennett added a comment -

          Here's a possible way to use Python itself for a cross-platform PATH workaround hack. Call this script with the name of the command to run with a corrected PATH:

          python -c "import os; import sys; import subprocess; env=os.environ.copy(); badsep = {':': ';', ';': ':'}[os.pathsep]; env['PATH'] = env['PATH'].replace(badsep, os.pathsep); subprocess.call(sys.argv[1:], env=env)" python --version
          # Replace "python --version" with any other command to run
          

          This can be run on either Windows cmd.exe or a bash shell on Windows or Linux.

          Colin Bennett added a comment - Here's a possible way to use Python itself for a cross-platform PATH workaround hack. Call this script with the name of the command to run with a corrected PATH: python -c "import os; import sys; import subprocess; env=os.environ.copy(); badsep = {':': ';', ';': ':'}[os.pathsep]; env['PATH'] = env['PATH'].replace(badsep, os.pathsep); subprocess.call(sys.argv[1:], env=env)" python --version # Replace "python --version" with any other command to run This can be run on either Windows cmd.exe or a bash shell on Windows or Linux.

          Paul Londino added a comment - - edited

          I have encountered this bug as well. It's not trivial to troubleshoot unless you stumble upon this issue, and it's not an uncommon problem to want to test Python code across Windows and Linux, which necessitates a mismatch in the OS between the master and the slave.

          Paul Londino added a comment - - edited I have encountered this bug as well. It's not trivial to troubleshoot unless you stumble upon this issue, and it's not an uncommon problem to want to test Python code across Windows and Linux, which necessitates a mismatch in the OS between the master and the slave.

            Unassigned Unassigned
            dbacher Dave Bacher
            Votes:
            5 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: