Local software installation guide

Pegasus users are free to compile and install software in their own home directories by following source code installation instructions. We suggest keeping downloaded source files separate from compiled files and downloaded binaries. In general, software installation involves:

  1. downloading and extracting files
  2. configuring the installation and compiling the software (compile only)
  3. updating PATH and creating symbolic links

Downloading and extracting files

If necessary, create software directories under your home directory:

$ mkdir ~/local ~/src

We suggest keeping your compiled software separate from any downloaded files. Consider keeping downloaded binaries (pre-compiled software) separate from source files if you will be installing many different programs. These directories do not need to be named exactly as shown above – just remember which is for what.

Navigate to the src folder and download files:

$ cd ~/src
$ wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz
$ wget http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/36.0/linux-x86_64/en-GB/firefox-36.0.tar.bz2

Some programs require configuration and compilation (like autoconf). Other programs are pre-compiled and simply need to be extracted (like Firefox). Read and follow all instructions provided for each program. Pre-compiled software can be extracted and immediately moved into your local software directory. We suggest maintaining subdirectories with application names and version numbers, as shown below.

Extraction flags:
tar.gz – xvzf
eXtract, Verbose, filter through gZip, using File


tar.bz2 – xvjf
filter through bzip2 (J)

Extract source code and change directory to the recently extracted folder:

$ tar xvzf autoconf-2.69.tar.gz
$ cd autoconf-2.69

Source code, proceed to Configuring installation and compiling software.

Extract pre-compiled software and move to local software directory:

$ tar xvjf firefox-36.0.tar.bz2
$ mv firefox $HOME/local/firefox/36

The newly-extracted Firefox should now be located in ~/local/firefox/36/firefox

Pre-compied binaries, skip to Updating PATH and creating symbolic links.

Configuring installation and compiling software

We suggest using subdirectories with application names and version numbers, as shown below. There may be other configuration settings specific to your software.

Configure with local folder prefix (full path):

$ ./configure --prefix=$HOME/local/autoconf/2.69

Make and install the software:

$ make
$ make install

If there are dependencies or conflicts, investigate the error output and try to resolve each error individually (install missing dependencies, check for specific flags suggested by software authors, check your local variables).

Updating PATH and creating symbolic links

PATH  directories are searched in order. To ensure your compiled or downloaded software is found and used first, prepend the software executable location (usually in software/bin or software directories) to your PATH  environment variable. Remember to add :$PATH to preserve existing environment variables.

Prepend software location to your PATH environment variable:

$ export PATH=$HOME/local/autoconf/2.69/bin:$PATH

Confirm by checking which software:

$ which autoconf
~/local/autoconf/2.69/bin/autoconf

Check software version (this may be software-dependent):

$ autoconf --version
autoconf (GNU Autoconf) 2.69
...

Symbolic links

To maintain multiple different versions of a program, use soft symbolic links to differentiate between the installation locations. Make sure the link and the directory names are distinct (example below). If you have been maintaining subdirectories with application names and version numbers, symlinks are not likely to conflict with other files or directories.

Create a distinctly-named symlink that points to your local software:

$ ln -s ~/local/firefox/36/firefox ~/local/firefox36

Append the local location to your PATH environment variable:

$ export PATH=$PATH:$HOME/local

Confirm both cluster copy and recently installed software:

$ which firefox
/usr/bin/firefox
$ firefox --version
Mozilla Firefox 17.0.10

$ which firefox36
~/local/firefox36
$ firefox36 --version
Mozilla Firefox 36.0

Reminder – to launch Firefox, connect to Pegasus via ssh with X11 forwarding enabled.

Persistent PATH

To persist additions to your PATH variable, edit the appropriate profile configuration file in your home directory. For bash on Pegasus, this is .bash_profile.

Update PATH in shell configuration (bash):

$ echo 'export PATH=$HOME/local/autoconf/2.69/bin:$PATH' >> ~/.bash_profile
$ echo 'export PATH=$PATH:$HOME/local' >> ~/.bash_profile

both in one command:

$ echo -e 'export PATH=$HOME/local/autoconf/2.69/bin:$PATH\nexport PATH=$PATH:$HOME/local' >> ~/.bash_profile

or edit file directly:

$ vim ~/.bash_profile
...
PATH=$PATH:$HOME/bin
PATH=$HOME/local/autoconf/2.69/bin:$PATH
PATH=$PATH:$HOME/local
...:wq

Reload shell configurations (bash) and check PATH:

$ source ~/.bash_profile
$ echo $PATH
/nethome/username/local/autoconf/2.69/bin:/share/opt/python/2.7.3/bin: ... :/share/sys65/root/sbin:/nethome/username/bin:/nethome/username/local

Python module installation

Users are free to compile and install Python modules in their own home directories. Most Python modules can be installed with the --user flag using pip, easy_install, or the setup.py file provided by the package. If you need a specific version of a Python module, we suggest downloading, extracting, and installing using setup.py. If you need to maintain multiple versions, see Python virtual environments below.

The --user flag will install Python 2.7 modules here:   ~/.local/lib/python2.7/site-packages

To specify a different location, instead use --prefix=$HOME/local/python2mods (or another directory). This example will install Python 2.7 modules here:   ~/local/python2mods/lib/python2.7/site-packages

Confirm Python is loaded:

$ module list
Currently Loaded Modulefiles:
  1) perl/5.18.1             3) gcc/4.4.7(default)
  2) python/2.7.3(default)   4) share-rpms65

Switching Python modules:

$ module unload python
$ module list
Currently Loaded Modulefiles:
  1) perl/5.18.1          2) gcc/4.4.7(default)   3) share-rpms65
$ module load python/3.3.1
$ module list
Currently Loaded Modulefiles:
  1) perl/5.18.1          3) share-rpms65
  2) gcc/4.4.7(default)   4) python/3.3.1

PIP

Install using pip with --user:

$ pip install --user munkres

Easy Install

Install using easy_install with --user:

$ easy_install --user munkres

Downloading, extracting, and installing

Download, extract, and cd to recently extracted folder:

$ wget https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz --no-check-certificate
$ tar xvzf munkres-1.0.7.tar.gz
$ cd munkres-1.0.7.tar.gz

Install using setup.py with –user:

$ python setup.py install --user

Checking installation

Launch Python and confirm module installation:

$ python
...
>>> import munkres
>>> print munkres.__version__
1.0.7
>>> CTRL-D (to exit Python)

Python virtual environments

Users can create their own Python virtual environments to maintain different module versions for different projects. Virtualenv is available on Pegasus for Python 2.7.3. By default, virtualenv does not include packages that are installed globally. To give a virtual environment access to the global site packages, use --system-site-packages flag. These directories do not need to be named exactly as shown.

Create a project folder, cd to the new folder, and create a virtualenv:

$ mkdir ~/python2
$ cd ~/python2
$ virtualenv test1
PYTHONHOME is set.  You *must* activate the virtualenv before using it
New python executable in test1/bin/python
Installing setuptools, pip...done.

Create a virtualenv with access to global packages:

$ virutalenv --system-site-packages test2

Activate the virtual environment:

$ source ~/python2/test1/bin/activate
(test1) $ which python
~/python2/test1/bin/python

While the virtual environment is active (noted by the name in parentheses), any package you install normally will be placed into that virtual environment folder and isolated from the global Python installation. Note that using --user or --prefix= flags during module installation will place modules in those specified directories, NOT your currently active Python virtual environment.

Install Python modules normally with pip, easy_install, or setup.py:

(test1) $ pip install munkres
or
(test1) $ wget https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz --no-check-certificate
(test1) $ tar xvzf munkres-1.0.7.tar.gz
(test1) $ cd munkres-1.0.7.tar.gz
(test1) $ python setup.py install

Deactivate the virtual environment:

(test1) $ deactivate
$ 

Compare two Python environments

Pip can be used to save a list of all packages and versions in the current environment (use freeze). Compare using sdiff to see which packages are different.

List the current environment, deactivate, then list the global Python environment:

(test1) $ pip freeze > test1.txt
(test1) $ deactivate
$ pip freeze > p2.txt

Compare the two outputs using sdiff:

$ sdiff p2.txt test1.txt
...
matplotlib==1.2.1                                             <
misopy==0.5.0                                                 | munkres==1.0.7
...
$ 

As seen above, the test1 environment has munkres installed (and no other global Python packages).

To recreate in a Python virtual environment, use the the saved list:
(test2) $ pip install -r test1.txt
Installing collected packages: munkres
  Running setup.py install for munkres
...
Successfully installed munkres
Cleaning up...
(test2) $ 

Python virtual environment wrapper

Users can install virtualenvwrapper in their own home directories to facilitate working with Python virtual environments. Once installed and configured, virtualenvwrapper can be used to create new virtual environments and to switch between your virtual environments (switching will deactivate the current environment). Virtualenvwrapper will read existing environment folders located in the WORKON_HOME directory. Set source to the local virtualenvwrapper.sh location as shown below.

Install a local copy of virtualenv with --user (and --always-copy for easy_install):

$ pip install --user virtualenvwrapper
or
$ easy_install --user --always-copy virtualenvwrapper

Set virtual environment home directory and source:

$ export WORKON_HOME=$HOME/python2
$ source ~/.local/bin/virtualenvwrapper.sh

Create a virtual environment using virtualenvwrapper (also activates):

$ mkvirtualenv test3
PYTHONHOME is set.  You *must* activate the virtualenv before using it
New python executable in test3/bin/python
Installing setuptools, pip...done.
(test3) $ 

Activate or switch to a virtual environment:

(test3) $ workon test1
(test1) $ 

Deactivate the virtual environment:

(test1) $ deactivate
$