Python Modules and Virtual Environments

Users are free to compile and install Python modules in their own home directories on Pegasus. Most Python modules can be installed with the --user flag using PIP, easy_install, or the setup.py file provided by the package. If you need a specific version of a Python module, we suggest using PIP with a direct link or downloading, extracting, and installing using setup.py. If you need to maintain multiple versions, see Python Virtual Environments (below).

The --user flag will install Python 2.7 modules here:   ~/.local/lib/python2.7/site-packages
Note the default location ~/.local is a hidden directory.

To specify a different location, use --prefix=$HOME/local/python2mods (or another path).
The above prefix flag example will install Python 2.7 modules here:   ~/local/python2mods/lib/python2.7/site-packages

Loading and Switching Python Modules

Confirm Python is loaded:

[username@pegasus ~]$ module list
Currently Loaded Modulefiles:
  1) perl/5.18.1             3) gcc/4.4.7(default)
  2) python/2.7.3(default)   4) share-rpms65

Switch Python modules:

[username@pegasus ~]$ module switch python/3.3.1
$ module list
Currently Loaded Modulefiles:
  1) perl/5.18.1          3) share-rpms65
  2) gcc/4.4.7(default)   4) python/3.3.1

Installing Python Modules with Package Managers

Install using PIP with --user:

[username@pegasus ~]$ pip install --user munkres
or install a specific version:
[username@pegasus ~]$ pip install --user munkres==1.0.7

Install using easy_install with --user:

[username@pegasus ~]$ easy_install --user munkres

Installing Downloaded Python Modules

Install with PIP with --user:

[username@pegasus ~]$ pip install --user https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz
Downloading/unpacking https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz
  Downloading munkres-1.0.7.tar.gz
  Running setup.py egg_info for package from https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz
    
Cleaning up...

Install using setup.py with --user:

[username@pegasus ~]$ wget https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz --no-check-certificate
[username@pegasus ~]$ tar xvzf munkres-1.0.7.tar.gz
[username@pegasus ~]$ cd munkres-1.0.7
[username@pegasus munkres-1.0.7]$ python setup.py --user install

Checking Module Versions

Launch Python and confirm module installation:

[username@pegasus ~]$ python
...
>>> import munkres
>>> print munkres.__version__
1.0.7
>>> CTRL-D (to exit Python)

Users can create their own Python virtual environments to maintain different module versions for different projects. Virtualenv is available on Pegasus for Python 2.7.3. By default, virtualenv does not include packages that are installed globally. To give a virtual environment access to the global site packages, use the --system-site-packages flag.

Creating Virtual Environments

These example directories do not need to be named exactly as shown.

Create a project folder, cd to the new folder (optional), and create a virtualenv:
[username@pegasus ~]$ mkdir ~/python2
[username@pegasus ~]$ cd ~/python2
[username@pegasus python2]$ virtualenv ~/python2/test1
PYTHONHOME is set.  You *must* activate the virtualenv before using it
New python executable in test1/bin/python
Installing setuptools, pip...done.
Create a virtualenv with access to global packages:
[username@pegasus python2]$ virutalenv --system-site-packages test2

Activating Virtual Environments

Activate the virtual environment with the source command and relative or absolute path/to/env/bin/activate. The environment name will precede the prompt.

[username@pegasus ~]$ source ~/python2/test1/bin/activate
(test1)[username@pegasus ~]$ which python
~/python2/test1/bin/python

Installing Python modules in Virtual Environments

Once the virtual environment is active, install Python modules normally with PIP, easy_install, or setup.py. Any package installed normally will be placed into that virtual environment folder and isolated from the global Python installation. Note that using --user or --prefix=... flags during module installation will place modules in those specified directories, NOT your currently active Python virtual environment.

(test1)[username@pegasus ~]$ pip install munkres

Deactivating Virtual Environments
(test1)[username@pegasus ~]$ deactivate
[username@pegasus ~]$ 

Comparing two Python Virtual Environments

PIP can be used to save a list of all packages and versions in the current environment (use freeze). Compare using sdiff to see which packages are different.

List the current environment, deactivate, then list the global Python environment:
(test1)[username@pegasus ~]$ pip freeze > test1.txt
(test1)[username@pegasus ~]$ deactivate
[username@pegasus ~]$ pip freeze > p2.txt
Compare the two outputs using sdiff:
[username@pegasus ~]$ sdiff p2.txt test1.txt
...
matplotlib==1.2.1                                             <
misopy==0.5.0                                                 | munkres==1.0.7
...
[username@pegasus ~]$ 

As seen above, the test1 environment has munkres installed (and no other global Python packages).

Recreating Python Virtual Environments

To recreate a Python virtual environment, use the r flag and the the saved list:

(test2)[username@pegasus ~]$ pip install -r test1.txt
Installing collected packages: munkres
  Running setup.py install for munkres
...
Successfully installed munkres
Cleaning up...
(test2)[username@pegasus ~]$ 

Python virtual environment wrapper

Users can install virtualenvwrapper in their own home directories to facilitate working with Python virtual environments. Once installed and configured, virtualenvwrapper can be used to create new virtual environments and to switch between your virtual environments (switching will deactivate the current environment). Virtualenvwrapper reads existing environments located in the WORKON_HOME directory.

Install a local copy of virtualenv with --user:

Recall that --user installs Python 2.7 modules in ~/.local/lib/python2.7/site-packages
To specify a different location, use --prefix=$HOME/local/python2mods (or another path).

[username@pegasus ~]$ pip install --user virtualenvwrapper
or
[username@pegasus ~]$ easy_install --user --always-copy virtualenvwrapper
Set virtual environment home directory and source:

WORKON_HOME should be the parent directory of your existing Python virtual environments (or another directory of your choosing). New Python virtual environments created with virtualenv will be stored according to this path. Set source to virtualenvwrapper.sh in the same location specified during installation.

[username@pegasus ~]$ export WORKON_HOME=$HOME/python2
[username@pegasus ~]$ source ~/.local/bin/virtualenvwrapper.sh
Create a virtual environment using virtualenvwrapper:

This will also activate the newly-created virtual environment.

[username@pegasus ~]$ mkvirtualenv test3
PYTHONHOME is set.  You *must* activate the virtualenv before using it
New python executable in test3/bin/python
Installing setuptools, pip...done.
(test3)[username@pegasus ~]$ 
Activate or switch to a virtual environment:
(test3)[username@pegasus ~]$ workon test1
(test1)[username@pegasus ~]$ 
Deactivate the virtual environment:
(test1)[username@pegasus ~]$ deactivate
[username@pegasus ~]$