Thanks cstarner, this is clearly going in the right direction, I really like the API for using it.
Still there are few essential questions that do need to be addressed before we could be able to test it in production:
- are these virtualenvs unique per build (not only job)? they need to be.
- can we safely run builds in parallel?
- are these virtualenvs stored in a safe-path? safe paths means one shorter than ~40 chars because otherwise we risk having broken tools due to linux shebang limitations, or even socket file-path limitations for some cases. Using user home directory is not very safe as is likely to be already too deep. Using workspace folder is a sure recipe for failures, they are too deep and we have no way to control their depth, also workspace directories are renamed by jenkins in some cases (like parallel builds), another source of broken virtualenvs.
- Because virtualenvs cannot be stored in the workspace, we need a way to recycle (remove) them when the workspace is cleaned. That's not an optional nice-to-have feature as orphan virtualenvs can fill the disk in just a couple of days if your servers are under high load.
I know that you may not be so happy to see so many concerns, but I hope that considering them will help all of us finding a reliable solution, one that does not need maintenance once we start using it.
My initial work around this issue ended up with this very simple groovy function: https://github.com/pycontribs/powertape/blob/master/vars/venv.groovy – which does not address the problems I listed above, this being the reason why this is not used in production yet.
+1 for getting this working.
In the mean time, using the workflowlibs plugin https://github.com/jenkinsci/workflow-cps-global-lib-plugin I have working virtual environments. I'm using vex here to run commands https://pypi.python.org/pypi/vex , as activating the virtualenv doesn't stay active across sh invocations in pipeline scripts.
It's a bit hackish, I just got it going, but maybe this will be useful for someone else.
In the pipeline script:
def vex = new com.testing.jenkins.python_utils()
vex.create_virtualenv_py3()
vex.run_cmd("python --version")
vex.run_cmd("pip install -r requirements.txt")
vex.run_cmd("some_script.py")