Python Virtual Environments

Don't attempt any serious Python programming without first creating a virtual environment. Here's how to do this, simply.

Exploring a new programming language is a little like dating. You can go out for the software equivalent of a casual coffee or a movie "as friends" from time to time, and as long as you don't have any expectations, nobody will get hurt. But if you start making plans for the future—that is, if you expect the little programs you've been writing to still be working next month, not to mention at your best friend's wedding next September—you're going to have to make some kind of long-term commitment. In Python, the first step of going steady is to start creating your programs inside a virtual environment.

I did not follow this policy when I started writing code in Python full-time. I worked on a project for about six months, and had working code both on my machine and on a network server. Then I was moved on to another project, in Java. A few months after that I was asked to revisit the Python project, and in both the network version and my local version I got runtime errors. To make matters worse, they were different runtime errors. The cause?, you ask. Both computers had been updated, and in different ways. The fix? I should have wrapped my project up in a virtual environment, which protects the application from changes on the machine it runs on.

These steps assume you are either working on a Mac or some flavor of Linux. I'm sure that the technique on a Windows machine is not very different.

Installing Python with pyenv

This section helps resolve the following problem: You want to program in a more recent version of Python than is currently on your system. For example, python3 --version returns "3.4," while you want to program in 3.6. And you're smart enough to know that you can't upgrade the entire system because there's too much of a chance that existing system-wide applications will be broken if you do. You know that what you want is to have a specific version of Python just for use in a virtual environment.

Some Background—and a Warning

(Even though the examples below come from a Mac, they are also applicable to Linux—and probably Windows as well.)

It's fairly normal to have multiple versions of Python on your system.

$ python --version
Python 2.7.14

$ which python
/usr/local/bin/python

$ python3 --version
Python 3.6.5

$ which python3
/usr/local/bin/python3

Often they'll be linked to a different location that hints at how these versions were installed.

$ ls -al /usr/local/bin/python*
lrwxr-xr-x  1 jkurlandski  admin  38 Apr 10  2018 /usr/local/bin/python -> ../Cellar/python@2/2.7.14_3/bin/python
lrwxr-xr-x  1 jkurlandski  admin  39 Apr 10  2018 /usr/local/bin/python2 -> ../Cellar/python@2/2.7.14_3/bin/python2
lrwxr-xr-x  1 jkurlandski  admin  41 Apr 10  2018 /usr/local/bin/python2.7 -> ../Cellar/python@2/2.7.14_3/bin/python2.7
lrwxr-xr-x  1 jkurlandski  admin  34 Apr 11  2018 /usr/local/bin/python3 -> ../Cellar/python/3.6.5/bin/python3
lrwxr-xr-x  1 jkurlandski  admin  36 Apr 11  2018 /usr/local/bin/python3.6 -> ../Cellar/python/3.6.5/bin/python3.6
(...)

As the paths containing the word Cellar indicate, this is a Mac machine and these versions of Python were all installed with the Homebrew package. A Linux box is also likely to have multiple versions on Python, though they'll have been installed by some other means.

Other versions of Python are probably on your computer, though in a different location.

$ ls -al /usr/bin/py*
lrwxr-xr-x  1 root  wheel     75 Oct  9  2019 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x  1 root  wheel     75 Oct  9  2019 /usr/bin/python2 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
lrwxr-xr-x  1 root  wheel     75 Oct  9  2019 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
(...)

These are called system versions of Python, and they are likely to have been installed before you bought your computer. You don't want to change these because for all you know there may be some application or service on your computer that depends on that specific version—or even that specific distribution—of Python.

Takeaways:

Do not change system-wide versions of Python, either with Homebrew on a Mac or wget or some other command on a Linux machine.
Do not download and install a new version of Python from https://www.python.org/downloads/.
The easiest way to get a particular version of Python—to be used for a virtual environment—is to use pyenv.

Install pyenv

Thus far, I have worked with pyenv only on my Mac. So I can't document how to install it on a Linux machine. A quick google leaves me with the impression that it's pretty simple.

On a Mac, use Homebrew.

$ which pyenv

$ brew install pyenv
(…)

$ which pyenv
/usr/local/bin/pyenv

$ pyenv --version
pyenv 1.2.13

Get the Python Version You Want

Now that pyenv is installed, you can check out your options. pyenv install --list displays all the versions of Python available; it's better to follow the command with a grep.

$ pyenv install --list | grep " 3\."
  3.0.1
  3.1.0
  3.1.1
  3.1.2
  3.1.3
  3.1.4
(...)
  3.7-dev
  3.7.1
  3.7.2
  3.7.3
  3.7.4
  3.8-dev
  3.9-dev

Here's how to install 3.7.4.

$ pyenv install 3.7.4
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.4.tar.xz...
-> https://www.python.org/ftp/python/3.7.4/Python-3.7.4.tar.xz
Installing Python-3.7.4...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.7.4 to /Users/jkurlandski/.pyenv/versions/3.7.4

$ /Users/jkurlandski/.pyenv/versions/3.7.4/bin/python --version
Python 3.7.4

$ /Users/jkurlandski/.pyenv/versions/3.7.4/bin/python3 --version
Python 3.7.4

Creating the Environment with venv

There are many different ways of setting up the virtual environment for a Python project. I take what I think is the simplest route. Specifically, this means that each project gets its own virtual environment, even if their environments are identical. It also means that the name of every project's virtual environment is the same: for mine, I have chosen the name "venv."

You could argue that it's confusing to use Python's venv module to create a virtual environment named "venv." My explanation is that I began using "venv" back when I was creating virtual environments with the virtualenv library. Perhaps one day I will start giving the virtual environments a different name.

In any case, in this section I'm going to show you how to create a project-specific virtual environment with the native venv module. The virtual environment will use a version of Python that is not part of the system (as described previously).

Note that the venv module is available only from Python 3.3 on. If you need a previous version of Python for your environment, you'll have to use virtualenv. I have some notes on how to do this in the appendix of this page.

Here, finally, are the steps for creating your environment:

Locate on your machine the version of Python you want to use.
Navigate to the directory where you want to do your programming.
Use the venv command with the path to your Python version.

$ pwd
/path/to/my/python/project

$ ls

(directory is empty)

$ /Users/jkurlandski/.pyenv/versions/3.7.4/bin/python3 -m venv venv
(...)

$ ls
venv

$ python --version
Python 2.7.14

$ venv/bin/python --version
Python 3.7.4

Notice that the Python version in the venv directory differs from the system version.

At this point you can start building your project. Create your sub-project folders in the same directory as the venv directory.

Directory structure of a virtualenv Python project

For example, here's a screenshot of an IDE showing the directory structure of a Python project with a virtual environment. The project is called PythonCode, and it has a single module called NER. The venv folder sits at the same level as NER.

The NER project will of course be stored in your repository; the venv folder will typically be "ignored."

Activating the Environment

In essence, activating a virtual environment means replacing system executables with local versions. In practice, it means changing the paths of the command-line shell you are in.

$ which python
/usr/local/bin/python

$ python --version
Python 2.7.10

$ ls
NER         venv

$ source venv/bin/activate

(venv) $ which python
/Users/jkurlandski/workspace/DeepLearning/venv/bin/python

(venv) $ python --version
Python 3.7.4

Note how the command prompt changes after the activation. More importantly, notice how activating the environment changes both the location of the Python executable being used and the version of Python being used.

Occasionally you might want to deactivate a virtual environment—though you can always close the terminal, or shut down the computer, without doing so.

(venv) $ deactivate

$ which python
/usr/local/bin/python

Again, notice how the command prompt changes.

Update Pip and Other Tools

Ensure pip, setuptools, and wheel are up to date.

From the Python documentation:

While pip alone is sufficient to install from pre-built binary archives, up to date copies of the setuptools and wheel projects are useful to ensure you can also install from source archives.

Most of the time you can install project requirements without the wheel package, but sometimes the process will throw warnings and take longer. Also, the Python documentation quoted above would have you perform this step before you create the virtual environment, but in my experience these tools are not made part of the virtual environment, so you would have to repeat them again here.

$ python -m pip install --upgrade pip setuptools wheel
 Successfully installed pip-20.1.1 setuptools-47.1.1 wheel-0.34.2

When to Activate the Environment

You'll need to remember to activate your virtual environment whenever you:

build from the command line
run the code from the command line
install a new project dependency from the command line

Installing Project Dependencies

The single most important thing to remember is to make sure your environment is activated when you install project dependencies. Otherwise the installation will be to your computer's system-wide environment, rather than to the Python project's, thus defeating the whole purpose of using a virtual environment.

In the listing below I activate the virtual environment, then use pip to install the numpy package.

$ ls
venv

$ source venv/bin/activate

(venv) $ pip install numpy
Collecting numpy
  Downloading numpy-1.12.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (4.4MB)
    100% |████████████████████████████████| 4.4MB 279kB/s 
Installing collected packages: numpy
Successfully installed numpy-1.12.0

Freezing Project Dependencies

The pip freeze command is a useful way to manage project environments and their dependencies across programmers and computers.

Traditionally, a project's dependencies are stored in the file requirements.txt.

$ source venv/bin/activate

(venv) $ ls
NER venv

(venv) $ pip freeze > requirements.txt

(venv) $ ls
NER         requirements.txt    venv

(venv) $ cat requirements.txt 
appdirs==1.4.3
numpy==1.12.0
packaging==16.8
pyparsing==2.2.0
six==1.10.0

At this point I could commit the project with the requirements.txt file into a git or Subversion repository. Then my colleagues could: (1) create virtual environments on their machines; (2) get my sourcecode from the repository; and (3) install the project dependencies listed in requirements.txt.

Installing Dependencies from requirements.txt

Here is how to install project dependencies from the requirements.txt file.

$ ls
NER         requirements.txt    venv

$ source venv/bin/activate

(venv) $ pip install -r requirements.txt 
Requirement already satisfied: appdirs==1.4.3 in ./venv/lib/python3.4/site-packages (from -r requirements.txt (line 1))
Collecting numpy==1.12.0 (from -r requirements.txt (line 2))
  Using cached numpy-1.12.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Requirement already satisfied: packaging==16.8 in ./venv/lib/python3.4/site-packages (from -r requirements.txt (line 3))
Requirement already satisfied: pyparsing==2.2.0 in ./venv/lib/python3.4/site-packages (from -r requirements.txt (line 4))
Requirement already satisfied: six==1.10.0 in ./venv/lib/python3.4/site-packages (from -r requirements.txt (line 5))
Installing collected packages: numpy
Successfully installed numpy-1.12.0

(venv) $

IDEs: Eclipse and PyCharm

Some IDE's offer the ability to create virtual environments: PyCharm's support is helpful, while in Eclipse I found the procedure more complicated than working from the command line.

With either IDE, you'll want to set it up to use your environment when building and running your code. Once set up, you won't have to worry about activating your virtual environment—as long as you're working within the IDE. Keep in mind, though, that you'll need to activate it whenever you move to the command line.

PyCharm

Idea's PyCharm offers you a very pleasant place to work. There's a free Community Edition available if you want to try it out; this edition is good enough for most Python tasks. You may find that the additional functionality of the pay version (the "Professional Edition") is worth the price. I did.

You can use PyCharm to set up your virtual environment, by which I mean tell it to create the environment itself. Once you've done that, you can load all your project's requirements from within the IDE.

However, I tend to set things up this way:

Create the virtual environment from the command line, as described above.
Activate the virtual environment and use pip to install components I know I'll be needing—again, from the command line.
Open PyCharm, point it toward my new project, and then configure the IDE's Python interpreter.

To configure the interpreter within PyCharm:

Open the PyCharm Preferences menu. (How to do this depends on the operating system.)
In the search box, type "project interpreter." This should bring up the Project Interpreter settings for the current project.
- To the right of the Project Interpreter dropdown window is the dropdown control button (marked by an arrow in the figure below). This is where you can select your Python version if you're not using a virtual environment. On the other hand, if you are—for example, if you've set one up as described above—it should appear in the dropdown list.
- At the far right is a [...] button (also marked with an arrow). This is where you can create a new virtual environment.
- At the bottom of the window is a "+" sign. Select this to install a new project dependency for your environment.
- To the right of that is an up arrow. Click on this to upgrade an existing project dependency. (Project dependencies that are eligible for upgrading are marked as such in the list. In the figure below, none of the dependencies are marked in this way.)

External Resources

Here are some links where you can find additional information about Python virtual environments:

Mac users should start here: Installing Python 3 on Mac OS X
Matthew Broberg's The right and wrong way to set Python 3 as default on a Mac provides a good introduction to pyenv, though I don't endorse his approach in every detail.
Logan Jones on the Real Python website provides Managing Multiple Python Versions With pyenv. This is useful but makes using pyenv appear more complicated than it needs to be.
https://www.reddit.com/r/learnpython/comments/4hsudz/pyvenv_vs_virtualenv/ is a good starting point to learn more about Python's venv module.

Appendix

In this appendix I'm dumping some old notes on virtual environments and working with multiple versions of Python which are a little outdated but in certain cases may still be useful.

virtualenv

virtualenv is a library which can be used to create Python virtual environments for versions of Python prior to 3.3, which is the version providing the venv module. In other words, if you need to create a virtual environment for a pre-3.3 version of Python, use virtualenv; otherwise, use the native venv module.

Install virtualenv

You'll need to decide up-front whether you want to work in Python 2.x or 3.x, or both. You'll need to have already installed the corresponding version of both Python and pip. (The latter is a utility program for installing Python-based programs and modules.)

$ python --version
Python 2.7.12

$ pip --version
pip 8.1.2 from /home/jerry/.local/lib/python2.7/site-packages (python 2.7)

I also have a 3.4 pair on my computer.

$ /usr/local/bin/python3 --version
Python 3.4.3

$ /usr/local/bin/pip3 --version
pip 8.1.2 from /usr/local/lib/python3.4/site-packages (python 3.4)

Fortunately the same version of virtualenv can be used to set up an environment for either version of Python. Use your default version of pip to download and install virtualenv.

$ sudo pip install virtualenv
[...]

Create your Environment

Navigate to the directory where you want to do your programming. If you have a single version of Python on your computer, you can use the simple virtualenv venv command to create an environment called "venv". If you have multiple versions of Python installed, you should specify the version you want to use with the -p <path to Python version> flag.

$ pwd
/path/to/my/python/project

$ virtualenv -p /usr/local/bin/python3 venv
Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4'
New python executable in /Users/jkurlandski/workspace/PythonCode/venv/bin/python3.4
Also creating executable in /Users/jkurlandski/workspace/PythonCode/venv/bin/python
Installing setuptools, pip, wheel...done.

$ ls
venv

A Separate Python Distribution on CentOS

Looking back on the problem described below, I realize that I might have been able to avoid all the trouble described here with pyenv. I'm not sure whether it wasn't available, or didn't work, or I simply didn't know of its existence—in any case, I installed a private, virtual-environment-only version of Python on a CentOS machine with the steps below. I present these notes here with the hope that they may be of some use to somebody, somewhere.

In a Nutshell

Start by going to https://www.python.org/ftp/python and deciding which Python distribution you want.

You'll want to modify the steps given below for your own circumstances. My steps make two assumptions:

You want to download Python version 3.6.4.
The place you want to install this version of Python is /home/user/Software.

$ pwd
/home/user/Software

$ wget  https://www.python.org/ftp/python/3.6.4/Python-3.6.4.tar.xz

$ unxz Python-3.6.4.tar.xz

$ tar xvf Python-3.6.4.tar

$ cd Python-3.6.4

At this point all the files have been downloaded and we need to build.

$ ./configure --enable-shared \
--prefix=/home/user/Software/Python-3.6.4/mybuild \
LDFLAGS="-Wl,--rpath=/home/user/Software/Python-3.6.4/mybuild/lib"

$ make

$ make install

The configure options are explained in the "Troubleshooting" section below.

Now we're ready to create our virtual environment out of the local, non-system version of Python that we've just built. My habit is to put the virtual environment into a directory called "venv" located at the root of my software project. How you manage your virtual environments doesn't matter—what's important about the next command is that you provide the path to your Python executable with the -p option.

$ virtualenv -p /home/user/Software/Python-3.6.4/mybuild/bin/python3.6 venv

Troubleshooting

Operating System

I performed the steps given above on Ubuntu 14. I think the Mac equivalent of wget for this purpose would be curl, but I admit I haven't verified this.

The `--enable-sharedEdit` Option

My first attempt at building Python did not include --enable-sharedEdit in the configuration. Later, when I tried to run the pyinstaller program from within my virtual environment, I got the following error.

    OSError: Python library not found: libpython3.5m.so.1.0, libpython3.5mu.so.1.0, libpython3.5.so.1.0
    This would mean your Python installation doesn't come with proper library files.
    This usually happens by missing development package, or unsuitable build parameters of Python installation.

    * On Debian/Ubuntu, you would need to install Python development packages
      * apt-get install python3-dev
      * apt-get install python-dev
    * If you're building Python by yourself, please rebuild your Python with `--enable-shared` (or, `--enable-framework` on Darwin)

The fix was to add '--enable-shared' to the Python build config.

The `LDFLAGS` Option

Building Python locally without the LDFLAGS option in the configuration resulted in the same error message whenever any program related to Python was run--python, pip, virtualenv, pyinstaller, and so on.

$ /home/user/Software/Python-3.6.4/mybuild/bin/python3 --version
/home/user/Software/Python-3.6.4/mybuild/bin/python3: error while loading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

$ virtualenv -p /home/user/Software/Python-3.6.4/mybuild/bin/python3.6 venv
Running virtualenv with interpreter /home/user/Software/Python-3.6.4/mybuild/bin/python3.6
/home/user/Software/Python-3.6.4/mybuild/bin/python3.6: error while loading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

The fix was to add the LDFLAGS to the Python build config as already shown.

Close to the Machine