This tutorial gives a rapid primer of Python Virtual Environments and how to use them in your effectively in your Python programming endeavours.
By the end you’ll know how to use Virtual Environments in Python 3 and 2, how to install modules, freezing and replaying environments, whether or not to activate an environment, how to package environments for reuse, how to create wheels of installed modules and how you can make use of the awesome pip-tools module for effective management.
Finally, I provide some context on other ways of managing virtual environments, outside of the python core pip package.
Introduction
When working on Python, or teaching Python to others, an area that I am an advocate for is the use of Virtual Environments for projects. Whilst many online training resources skip or leave such context until later on, personally, I feel that it is an area that is best covered at the earliest opportunity. From a simplistic viewpoint, it provides a convenient environment for individuals to experiment with the plethora of modules available in the python ecosystem, and, mitigates the need for root privileges when installing modules on Linux based systems. There are also a variety of other benefits that this post will aim to cover.
Creating Virtual Environments
In Both Python 2 and Python 3, virtual environments are available, however the creation of which varies between the two major languages and in version 3, there is both a preferred and legacy method. Given the future of Python 3 this post emphasises more on this version of Python, however, much of which in the post will apply to both versions.
Creating A Virtual Environment in Python 3 (Preferred Approach)
$ python3 -mvenv python3venv
Breaking this down, the -m option on the command line, tells the interpreter to run the venv module that is provided as part of the core Python3 distribution. The second parameter of ‘python3venv’ is the name of the virtual environment. The execution of this command should give you a virtual environment in the directory of python3venv.
If you’re inquisitive, you can understand more on what is happening behind the scenes by reviewing the actual venv module. A convenient way of identifying the source location, is through the REPL shell -
$ python3 Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import venv >>> print(venv) <module 'venv' from '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/venv/__init__.py'>
With the code here being at ‘/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/venv/__init__.py’
If you jump to the bottom of the code in question, it has an execution block at the footer of the file allowing it to be executed as a script -
if __name__ == '__main__': rc = 1 try: main() rc = 0 except Exception as e: print('Error: %s' % e, file=sys.stderr) sys.exit(rc)
Thus, executing the code directly, executes the main function that provides a list of available arguments to the user -
$ python3 -mvenv usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] [--upgrade] [--without-pip] [--prompt PROMPT] ENV_DIR [ENV_DIR ...] venv: error: the following arguments are required: ENV_DIR
Creating A Virtual Environment in Python 3.6 and Below (Deprecated Approach)
Python Version 3.6 and below, have an executable called pyvenv which is essentially a binary wrapper for the venv module. If you execute the binary as it is, you can see similarities in the usage syntax, with, an added deprecation warning in certain versions -
$ pyvenv WARNING: the pyenv script is deprecated in favour of `python3.6 -m venv` usage: venv [-h] [--system-site-packages] [--symlinks | --copies] [--clear] [--upgrade] [--without-pip] [--prompt PROMPT] ENV_DIR [ENV_DIR ...] venv: error: the following arguments are required: ENV_DIR
Running the executable through the strings command, shows a similar entry point with an implicit import of the venv module -
$ strings /Library/Frameworks/Python.framework/Versions/3.6/bin/pyvenv #!/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 if __name__ == '__main__': import sys import pathlib executable = pathlib.Path(sys.executable or 'python3').name print('WARNING: the pyenv script is deprecated in favour of ' f'`{executable} -m venv`', file=sys.stderr) rc = 1 try: import venv venv.main() rc = 0 except Exception as e: print('Error: %s' % e, file=sys.stderr) sys.exit(rc)
Finally, executing the binary gives the same result as the preferred approach, with the added warning shown in the snippet -
$ pyvenv python3venv WARNING: the pyenv script is deprecated in favour of `python3.6 -m venv`
Creating A Virtual Environment in Python 2.x
On Python 2.x, the virtualenv binary can be used to create the Virtual Environment, on my Mac, this is available as part of the Python 2.7 installation. In other operating systems your milage may vary and subsequently, virtualenv may need to be installed if it’s not already available.
Check for a virtualenv binary -
$ which virtualenv /Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenv
Create a Python 2.x virtualenv. The syntax is similar to Python 3 with the target virtualenv directory. The Python 2 equivalent however gives a lot more in terms of output -
$ virtualenv python2venv New python executable in /Users/james/python2venv/bin/python Installing setuptools, pip, wheel...done.
Inspecting the Virtual Environments
In both cases, a similar structure is created -
$ ls python3venv python2venv/ python2venv/: bin include lib
python3venv: bin include lib pyvenv.cfg
The main difference being the pyvenv.cfg file that is visible in the root of the directory. If you’re interested in knowing more about this file PEP-0405 has a lot of context.
Activating the Virtual Environment
A convenient way of using the virtual environment is through activation, however, it is not an implicit requirement (something we’ll look into further). When you activate the virtual environment, the user’s shell prompt (PS1) is updated and subsequently, the command path gives higher priority to the virtual environment. If we check the PATH before activation, it may look something like this -
If we now execute either python or pip, owing to the PATH precedence, the binary from the virtual environment take priority
(python3venv) $ which python /Users/james/python3venv/bin/python (python3venv) $ which pip /Users/james/python3venv/bin/pip
The Python environment, is automatically configured to search the environments paths for module initialisation, we can verify this with the REPL environment, note the addition of /Users/james/python3venv/lib/python3.6/site-packages to the sys.path list -
(python3venv) $ python Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> print(sys.path) ['', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python36.zip', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6', '/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/lib-dynload', '/Users/james/python3venv/lib/python3.6/site-packages']
Installing packages through pip
With the virtual environment activated, the use of pip will result in requested packages being installed direct to the virtual environment. For example, if we install the awesome datetime module called arrow -
(python3venv) $ pip install arrow Collecting arrow Downloading https://files.pythonhosted.org/packages/f4/7f/0360628ba40bb93c10cd89cd289b6a8e9ea87b2db884b8edf32c80ee1c73/arrow-0.13.1-py2.py3-none-any.whl Collecting python-dateutil (from arrow) Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB) 100% |████████████████████████████████| 235kB 3.6MB/s Collecting six>=1.5 (from python-dateutil->arrow) Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl Installing collected packages: six, python-dateutil, arrow Successfully installed arrow-0.13.1 python-dateutil-2.8.0 six-1.12.0 You are using pip version 9.0.1, however version 19.0.3 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
It has installed both arrow, and a dependent module of six and python-dateutil. We can again confirm, that it is actually installed in the virtual environment, through the REPL interface -
(python3venv) $ python Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import arrow >>> print(arrow) <module 'arrow' from '/Users/james/python3venv/lib/python3.6/site-packages/arrow/__init__.py'> >>> ^D
Using pip to upgrade pip
The eagle eyed amongst you, may have noticed the warning that pip provided above, where a new version is available. In the virtual environment, I typically recommend that people upgrade pip as a post action to creating the venv, prior to installing any modules but showed the approach above as an example of the warning that you may see. We can upgrade the environment as follows -
We have a significant number of modules, post Flask/Arrow installation. The convenient command of pip freeze, allows us to capture a snapshot of the virtual environment, in this example I’m using tee, to also send the output to a text file called requirements.txt
Earlier on, I created a Python 2 virtual environment, let’s use this, to replay the virtual environment configuration. Firstly, we deactivate the existing venv
(python3venv) $ deactivate $
Then we activate the Python 2 virtual environment -
$ source python2venv/bin/activate (python2venv) $
Upgrade pip, in most cases you should be able to use ‘pip install –upgrade pip’, however, I encountered an issue and had to perform the following workaround -
$ (python2venv) curl https://bootstrap.pypa.io/get-pip.py | python % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1659k 100 1659k 0 0 2375k 0 --:--:-- --:--:-- --:--:-- 2377k DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. Collecting pip Using cached https://files.pythonhosted.org/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl Installing collected packages: pip Found existing installation: pip 9.0.1 Uninstalling pip-9.0.1: Successfully uninstalled pip-9.0.1 Successfully installed pip-19.0.3
We can now replay the environment in our Python 3 setup, in the Python 2 environment, using the requirements.txt captured using pip freeze -
$ (python2venv) pip install -r /tmp/requirements.txt DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. Collecting arrow==0.13.1 (from -r /tmp/requirements.txt (line 1)) Using cached https://files.pythonhosted.org/packages/f4/7f/0360628ba40bb93c10cd89cd289b6a8e9ea87b2db884b8edf32c80ee1c73/arrow-0.13.1-py2.py3-none-any.whl Collecting Click==7.0 (from -r /tmp/requirements.txt (line 2)) Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl Collecting Flask==1.0.2 (from -r /tmp/requirements.txt (line 3)) Using cached https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl Collecting itsdangerous==1.1.0 (from -r /tmp/requirements.txt (line 4)) Using cached https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl Collecting Jinja2==2.10 (from -r /tmp/requirements.txt (line 5)) Using cached https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl Collecting MarkupSafe==1.1.1 (from -r /tmp/requirements.txt (line 6)) Downloading https://files.pythonhosted.org/packages/6d/d2/0ccd2c0e2cd93b35e765d9b3205cd6602e6b202b522fc7997531353715b3/MarkupSafe-1.1.1-cp27-cp27m-macosx_10_6_intel.whl Collecting python-dateutil==2.8.0 (from -r /tmp/requirements.txt (line 7)) Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl Collecting six==1.12.0 (from -r /tmp/requirements.txt (line 8)) Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl Collecting Werkzeug==0.14.1 (from -r /tmp/requirements.txt (line 9)) Using cached https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl Collecting backports.functools-lru-cache>=1.2.1; python_version == "2.7" (from arrow==0.13.1->-r /tmp/requirements.txt (line 1)) Downloading https://files.pythonhosted.org/packages/03/8e/2424c0e65c4a066e28f539364deee49b6451f8fcd4f718fefa50cc3dcf48/backports.functools_lru_cache-1.5-py2.py3-none-any.whl Installing collected packages: six, python-dateutil, backports.functools-lru-cache, arrow, Click, MarkupSafe, Jinja2, itsdangerous, Werkzeug, Flask Successfully installed Click-7.0 Flask-1.0.2 Jinja2-2.10 MarkupSafe-1.1.1 Werkzeug-0.14.1 arrow-0.13.1 backports.functools-lru-cache-1.5 itsdangerous-1.1.0 python-dateutil-2.8.0 six-1.12.0
Pip freeze, will confirm that the Python 2 environment has the same versions installed as the Python 3 environment and is almost identical to what we saw previously. The version 2 equivalent also required a dependency of ‘backports.functools-lru-cache==1.5’ hence an additional module -
(python2venv) $ pip freeze DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. arrow==0.13.1 backports.functools-lru-cache==1.5 Click==7.0 Flask==1.0.2 itsdangerous==1.1.0 Jinja2==2.10 MarkupSafe==1.1.1 python-dateutil==2.8.0 six==1.12.0 Werkzeug==0.14.1
Whilst I don’t necessarily recommend switching virtual environments between Python versions, the example was chosen to show the versitiliy of the pip freeze command and the ability, to replay an environment, albeit into the same version of a virtual environment, or another version whether major, or minor.
Using a Virtual Environment, without activation
Earlier, I made reference that it’s convenient, to activate a virtual environment but, in some cases, it’s also convenient not to activate it. For example, you may wish to call a script or application direct.
The python binary, in the virtual environment can be called directly and existing search paths are honoured. For example, if we call the binary direct without any virtualenv, we can still import the modules we’ve installed without error -
(python2venv) $ deactivate $ python3venv/bin/python Python 3.6.4 (v3.6.4:d48ecebad5, Dec 18 2017, 21:07:28) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import arrow >>> print(arrow) <module 'arrow' from '/Users/james/python3venv/lib/python3.6/site-packages/arrow/__init__.py'> >>> ^D
$ python2venv/bin/python Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 12:01:12) [GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import arrow >>> print(arrow) <module 'arrow' from '/Users/james/python2venv/lib/python2.7/site-packages/arrow/__init__.pyc'> >>> ^D
Our scripts can also use the #! directive, making direct reference to the python binary, if we create a quick script -
Making it highly convenient for a script or application, to run with the specific requirements that are needed.
Packaging and re-use
With a virtual environment configured, it’s possible to package an entire virtual environment, and use it as an execution base providing that the target installation has the same version of Python that was used to create the virtual environment and that the virtual environment is placed in the same location. This is very convenient, when working in environments that have standardised builds. Essentially, the whole environment can be packaged and deployed to another system.
To quickly demonstrate, take a dump of the virtual env
$ tar cf python3venv.tar python3venv
Destroy the venv -
$ rm -rf python3venv
Extract the dump -
$ tar xf python3venv.tar
Test a dependent script -
$ ./test.py 2019-03-03T21:26:47.082092+00:00
Using pip-tools to manage Virtual Environments
Personally, I am a big fan of pip-tools, it provides a great way, of managing and maintaining virtual environments and their dependencies. From the examples above, we saw that the installation of both arrow and flask, resulted in a number of dependent modules. If for example we now remove both arrow and flask, the dependencies, remain in the virtual environment -
(python3venv) $ pip uninstall arrow flask Uninstalling arrow-0.13.1: Would remove: /Users/james/python3venv/lib/python3.6/site-packages/arrow-0.13.1.dist-info/* /Users/james/python3venv/lib/python3.6/site-packages/arrow/* Proceed (y/n)? y Successfully uninstalled arrow-0.13.1 Uninstalling Flask-1.0.2: Would remove: /Users/james/python3venv/bin/flask /Users/james/python3venv/lib/python3.6/site-packages/Flask-1.0.2.dist-info/* /Users/james/python3venv/lib/python3.6/site-packages/flask/* Proceed (y/n)? y Successfully uninstalled Flask-1.0.2
We can now use an input file to track requirements. In this case, I’m going to create a requirements.in file, and I will implicitly put the versions of arrow and flask that I am interested in
Should however we wish to remove a component, for example Flask, we can update our requirements.in. When we repeat the process with pip-sync, it will remove Flask as well as Flask’s dependencies -
Using your Virtual Environment with a Specific PyPi Repository
In some circumstances, you may have a requirement to use a specific PyPi repository for installation. Should this be the case, the creation of a file called pip.conf, in the root of the Virtual Environment is enough to satisfy this dependency. An example pip.conf looks like the following, change hostname.spurin.com as required -
Virtual Environments are a great way of isolating, specific versions of modules. Over time you may have a project, that has dependencies on a specific version whilst another, may benefit from newer functionality.
A particular module where this isolation is effective in Ansible and in the course that I created, Mastering Ansible, my preferred approach for installing Ansible is via a Virtual Environment. You can be specific with the version of Ansible that you install, can choose whether or not you leverage Python 2 or Python 3 (historically, Python 2 is a more reliable choice with Ansible) and you can have multiple versions of Ansible on the same system. The following diagram from the course demonstrates the effectiveness, of Virtual Environments -
Using Multiple Virtual Environments
Creating Wheels from your Virtual Environment
Wheel can be used to capture Python modules into zip like files that can be conveniently used for installations. They even handle complicated Python packages that require compilation (C extensions). Using such an approach, allows you to facilitate installations from one machine that has a compiler to another that may not.
This post, covers a lot of the use cases for how I personally use virtual environment and the traditional approach for setup and use. There are as to be expected with a topic such as this, a multitude of a ways of managing and maintaining your own virtual environments.
A popular module at the moment is Kenneth Reitz pipenv. It is an excellent module and as per his other endeavours, it has been met with significant praise. There is however a flip side to it’s use and both the article by Chris Warrick and the subsequent discussion in the hacker news thread provide another viewpoint. I recommend both trying the module for yourself and reading the counter arguments.
Using Multiple Virtual Environments
