Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37116

Add support for Jenkins Pipeline to the shiningpanda-plugin

      As a user of Jenkins and the ShiningPanda plugin
      I want to be able to define build steps from the ShiningPanda plugin in a Jenkinsfile
      So that builds in my pipelines can use virtualenv

          [JENKINS-37116] Add support for Jenkins Pipeline to the shiningpanda-plugin

          +1 for getting this working.

          In the mean time, using the workflowlibs plugin https://github.com/jenkinsci/workflow-cps-global-lib-plugin I have working virtual environments. I'm using vex here to run commands https://pypi.python.org/pypi/vex , as activating the virtualenv doesn't stay active across sh invocations in pipeline scripts.

          It's a bit hackish, I just got it going, but maybe this will be useful for someone else.

          % cat workflowLibs/src/com/testing/jenkins/python_utils.groovy
          package com.testing.jenkins;
          
          def create_virtualenv(){
              withEnv(["PATH+VEX=~/.local/bin"]){
                  sh 'pwd > pwd.current'
                  env.WORKSPACE = readFile('pwd.current').trim()
          
                  sh "virtualenv ${env.WORKSPACE}/.virtualenv"
              }
          }
          
          def create_virtualenv_py3(){
              withEnv(["PATH+VEX=~/.local/bin"]){
                  sh 'pwd > pwd.current'
                  env.WORKSPACE = readFile('pwd.current').trim()
          
                  sh "virtualenv --python=python3 ${env.WORKSPACE}/.virtualenv"
              }
          }
          
          def run_cmd(pycmd){
              withEnv(["PATH+VEX=~/.local/bin"]){
                  sh "vex --path=${env.WORKSPACE}/.virtualenv ${pycmd}"
              }
          }
          
          

          In the pipeline script:
          def vex = new com.testing.jenkins.python_utils()
          vex.create_virtualenv_py3()
          vex.run_cmd("python --version")
          vex.run_cmd("pip install -r requirements.txt")
          vex.run_cmd("some_script.py")

          matthew snyder added a comment - +1 for getting this working. In the mean time, using the workflowlibs plugin https://github.com/jenkinsci/workflow-cps-global-lib-plugin I have working virtual environments. I'm using vex here to run commands https://pypi.python.org/pypi/vex , as activating the virtualenv doesn't stay active across sh invocations in pipeline scripts. It's a bit hackish, I just got it going, but maybe this will be useful for someone else. % cat workflowLibs/src/com/testing/jenkins/python_utils.groovy package com.testing.jenkins; def create_virtualenv(){ withEnv([ "PATH+VEX=~/.local/bin" ]){ sh 'pwd > pwd.current' env.WORKSPACE = readFile( 'pwd.current' ).trim() sh "virtualenv ${env.WORKSPACE}/.virtualenv" } } def create_virtualenv_py3(){ withEnv([ "PATH+VEX=~/.local/bin" ]){ sh 'pwd > pwd.current' env.WORKSPACE = readFile( 'pwd.current' ).trim() sh "virtualenv --python=python3 ${env.WORKSPACE}/.virtualenv" } } def run_cmd(pycmd){ withEnv([ "PATH+VEX=~/.local/bin" ]){ sh "vex --path=${env.WORKSPACE}/.virtualenv ${pycmd}" } } In the pipeline script: def vex = new com.testing.jenkins.python_utils() vex.create_virtualenv_py3() vex.run_cmd("python --version") vex.run_cmd("pip install -r requirements.txt") vex.run_cmd("some_script.py")

          Byron Dover added a comment -

          Thank you, msnyder!

          Byron Dover added a comment - Thank you, msnyder !

          +1 to this. Would love to be able to run "native" pipeline commands to build inside venv.

          Omer van Kloeten added a comment - +1 to this. Would love to be able to run "native" pipeline commands to build inside venv.

          Sorin Sbarnea added a comment -

          I don't know if this helps others but because of the multiple bugs related to shining panda, including the jobs executed on parallel, we decided to replace it with shell script and manage the virtualenv management ourselves. 

          Sorin Sbarnea added a comment - I don't know if this helps others but because of the multiple bugs related to shining panda, including the jobs executed on parallel, we decided to replace it with shell script and manage the virtualenv management ourselves. 

          Sorin Sbarnea added a comment -

          While trying to find a better way to use virtualenvs inside (bigger) pipelines, I found encountered this https://codegists.com/code/jenkins-pipeline-dsl/ and https://codegists.com/snippet/groovy/build-jobgroovy_joebew42_groovy which seems to use two interesting syntaxes. Still I was not able to identify which plugins do expose these new commands.

          Sorin Sbarnea added a comment - While trying to find a better way to use virtualenvs inside (bigger) pipelines, I found encountered this https://codegists.com/code/jenkins-pipeline-dsl/  and https://codegists.com/snippet/groovy/build-jobgroovy_joebew42_groovy  which seems to use two interesting syntaxes. Still I was not able to identify which plugins do expose these new commands.

          Colin Starner added a comment -

          I created a plugin which exposes new Pipeline DSL steps geared towards running commands within a specified virtualenv. I personally am not affiliated with the ShiningPanda project, but this plugin will utilize ShiningPanda Python installations if specified. 

          Perhaps this is not a perfect solution, but hopefully some will find it useful.

          https://plugins.jenkins.io/pyenv-pipeline

          Colin Starner added a comment - I created a plugin which exposes new Pipeline DSL steps geared towards running commands within a specified virtualenv. I personally am not affiliated with the ShiningPanda project, but this plugin will utilize ShiningPanda Python installations if specified.  Perhaps this is not a perfect solution, but hopefully some will find it useful. https://plugins.jenkins.io/pyenv-pipeline

          Sorin Sbarnea added a comment -

          Thanks cstarner, this is clearly going in the right direction, I really like the API for using it.

          Still there are few essential questions that do need to be addressed before we could be able to test it in production:

          • are these virtualenvs unique per build (not only job)? they need to be.
          • can we safely run builds in parallel?
          • are these virtualenvs stored in a safe-path? safe paths means one shorter than ~40 chars because otherwise we risk having broken tools due to linux shebang limitations, or even socket file-path limitations for some cases. Using user home directory is not very safe as is likely to be already too deep. Using workspace folder is a sure recipe for failures, they are too deep  and we have no way to control their depth, also workspace directories are renamed by jenkins in some cases (like parallel builds), another source of broken virtualenvs.
          • Because virtualenvs cannot be stored in the workspace, we need a way to recycle (remove) them when the workspace is cleaned. That's not an optional nice-to-have feature as orphan virtualenvs can fill the disk in just a couple of days if your servers are under high load.

          I know that you may not be so happy to see so many concerns, but I hope that considering them will help all of us finding a reliable solution, one that does not need maintenance once we start using it.

          My initial work around this issue ended up with this very simple groovy function: https://github.com/pycontribs/powertape/blob/master/vars/venv.groovy – which does not address the problems I listed above, this being the reason why this is not used in production yet.

          Sorin Sbarnea added a comment - Thanks cstarner , this is clearly going in the right direction, I really like the API for using it. Still there are few essential questions that do need to be addressed before we could be able to test it in production: are these virtualenvs unique per build (not only job)? they need to be. can we safely run builds in parallel? are these virtualenvs stored in a safe-path? safe paths means one shorter than ~40 chars because otherwise we risk having broken tools due to linux shebang limitations, or even socket file-path limitations for some cases. Using user home directory is not very safe as is likely to be already too deep. Using workspace folder is a sure recipe for failures, they are too deep  and we have no way to control their depth, also workspace directories are renamed by jenkins in some cases (like parallel builds), another source of broken virtualenvs. Because virtualenvs cannot be stored in the workspace, we need a way to recycle (remove) them when the workspace is cleaned. That's not an optional nice-to-have feature as orphan virtualenvs can fill the disk in just a couple of days if your servers are under high load. I know that you may not be so happy to see so many concerns, but I hope that considering them will help all of us finding a reliable solution, one that does not need maintenance once we start using it. My initial work around this issue ended up with this very simple groovy function: https://github.com/pycontribs/powertape/blob/master/vars/venv.groovy – which does not address the problems I listed above, this being the reason why this is not used in production yet.

          Colin Starner added a comment - - edited
          • The virtualenvs are, by default, unique per build. They are simply generated in the current Workspace directory, (which themselves are unique per build by default). Unless someone goes out of their way to share Workspace directories between builds for some reason, these should remain unique.
          • The pysh and pybat steps (the ones that actually execute commands), use DurableTasks to do the heavy lifting. This is the same basis for the built-in sh and bat steps; as such they are safe to use in parallel. (This also means that any options that can be passed to sh and bat will work for pysh and pybat i.e. returnStdout)
          • These last two concerns have not been addressed, because of the way that virtualenvs were dumped into workspaces. I was unaware of these issues, and had not ran into them personally. 

          These seem relatively simple to workaround. The last concern can be solved with the creation of another Step (although it's a little less clean). If you wouldn't mind opening some issues on the Github page, I will begin to address them

          Colin Starner added a comment - - edited The virtualenvs are, by default, unique per build. They are simply generated in the current Workspace directory, (which themselves are unique per build by default). Unless someone goes out of their way to share Workspace directories between builds for some reason, these should remain unique. The pysh and pybat steps (the ones that actually execute commands), use DurableTasks to do the heavy lifting. This is the same basis for the built-in sh and bat steps; as such they are safe to use in parallel. (This also means that any options that can be passed to sh and bat will work for pysh and pybat i.e. returnStdout) These last two concerns have not been addressed, because of the way that virtualenvs were dumped into workspaces. I was unaware of these issues, and had not ran into them personally.  These seem relatively simple to workaround. The last concern can be solved with the creation of another Step (although it's a little less clean). If you wouldn't mind opening some issues on the Github page, I will begin to address them

            Unassigned Unassigned
            davidparsson David Pärsson
            Votes:
            13 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated: