Local software installation guide
Pegasus users are free to compile and install software in their own home directories by following source code installation instructions. We suggest keeping downloaded source files separate from compiled files and downloaded binaries. In general, software installation involves:
- downloading and extracting files
- configuring the installation and compiling the software (compile only)
- updating PATH and creating symbolic links
Downloading and extracting files
If necessary, create software directories under your home directory:
$ mkdir ~/local ~/src
We suggest keeping your compiled software separate from any downloaded files. Consider keeping downloaded binaries (pre-compiled software) separate from source files if you will be installing many different programs. These directories do not need to be named exactly as shown above – just remember which is for what.
Navigate to the src folder and download files:
$ cd ~/src $ wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz $ wget http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/36.0/linux-x86_64/en-GB/firefox-36.0.tar.bz2
Some programs require configuration and compilation (like autoconf). Other programs are pre-compiled and simply need to be extracted (like Firefox). Read and follow all instructions provided for each program. Pre-compiled software can be extracted and immediately moved into your local software directory. We suggest maintaining subdirectories with application names and version numbers, as shown below.
tar.gz – xvzf
eXtract, Verbose, filter through gZip, using File
tar.bz2 – xvjf
filter through bzip2 (J)
Extract source code and change directory to the recently extracted folder:
$ tar xvzf autoconf-2.69.tar.gz $ cd autoconf-2.69
Source code, proceed to Configuring installation and compiling software.
Extract pre-compiled software and move to local software directory:
$ tar xvjf firefox-36.0.tar.bz2 $ mv firefox $HOME/local/firefox/36
The newly-extracted Firefox should now be located in
Pre-compied binaries, skip to Updating PATH and creating symbolic links.
Configuring installation and compiling software
We suggest using subdirectories with application names and version numbers, as shown below. There may be other configuration settings specific to your software.
Configure with local folder prefix (full path):
$ ./configure --prefix=$HOME/local/autoconf/2.69
Make and install the software:
$ make $ make install
If there are dependencies or conflicts, investigate the error output and try to resolve each error individually (install missing dependencies, check for specific flags suggested by software authors, check your local variables).
Updating PATH and creating symbolic links
PATH directories are searched in order. To ensure your compiled or downloaded software is found and used first, prepend the software executable location (usually in software/bin or software directories) to your PATH environment variable. Remember to add :$PATH to preserve existing environment variables.
Prepend software location to your PATH environment variable:
$ export PATH=$HOME/local/autoconf/2.69/bin:$PATH
Confirm by checking which software:
$ which autoconf ~/local/autoconf/2.69/bin/autoconf
Check software version (this may be software-dependent):
$ autoconf --version autoconf (GNU Autoconf) 2.69 ...
To maintain multiple different versions of a program, use soft symbolic links to differentiate between the installation locations. Make sure the link and the directory names are distinct (example below). If you have been maintaining subdirectories with application names and version numbers, symlinks are not likely to conflict with other files or directories.
Create a distinctly-named symlink that points to your local software:
$ ln -s ~/local/firefox/36/firefox ~/local/firefox36
Append the local location to your PATH environment variable:
$ export PATH=$PATH:$HOME/local
Confirm both cluster copy and recently installed software:
$ which firefox /usr/bin/firefox $ firefox --version Mozilla Firefox 17.0.10 $ which firefox36 ~/local/firefox36 $ firefox36 --version Mozilla Firefox 36.0
Reminder – to launch Firefox, connect to Pegasus via ssh with X11 forwarding enabled.
To persist additions to your PATH variable, edit the appropriate profile configuration file in your home directory. For bash on Pegasus, this is
Update PATH in shell configuration (bash):
$ echo 'export PATH=$HOME/local/autoconf/2.69/bin:$PATH' >> ~/.bash_profile $ echo 'export PATH=$PATH:$HOME/local' >> ~/.bash_profile
both in one command:
$ echo -e 'export PATH=$HOME/local/autoconf/2.69/bin:$PATH\nexport PATH=$PATH:$HOME/local' >> ~/.bash_profile
or edit file directly:
$ vim ~/.bash_profile ... PATH=$PATH:$HOME/bin PATH=$HOME/local/autoconf/2.69/bin:$PATH PATH=$PATH:$HOME/local ...:wq
Reload shell configurations (bash) and check PATH:
$ source ~/.bash_profile $ echo $PATH /nethome/username/local/autoconf/2.69/bin:/share/opt/python/2.7.3/bin: ... :/share/sys65/root/sbin:/nethome/username/bin:/nethome/username/local
Python module installation
Users are free to compile and install Python modules in their own home directories. Most Python modules can be installed with the
--user flag using pip, easy_install, or the setup.py file provided by the package. If you need a specific version of a Python module, we suggest downloading, extracting, and installing using setup.py. If you need to maintain multiple versions, see Python virtual environments below.
--user flag will install Python 2.7 modules here:
To specify a different location, instead use
--prefix=$HOME/local/python2mods (or another directory). This example will install Python 2.7 modules here:
Confirm Python is loaded:
$ module list Currently Loaded Modulefiles: 1) perl/5.18.1 3) gcc/4.4.7(default) 2) python/2.7.3(default) 4) share-rpms65
Switching Python modules:
$ module unload python $ module list Currently Loaded Modulefiles: 1) perl/5.18.1 2) gcc/4.4.7(default) 3) share-rpms65 $ module load python/3.3.1 $ module list Currently Loaded Modulefiles: 1) perl/5.18.1 3) share-rpms65 2) gcc/4.4.7(default) 4) python/3.3.1
Install using pip with
$ pip install --user munkres
Install using easy_install with
$ easy_install --user munkres
Downloading, extracting, and installing
Download, extract, and cd to recently extracted folder:
$ wget https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz --no-check-certificate $ tar xvzf munkres-1.0.7.tar.gz $ cd munkres-1.0.7.tar.gz
Install using setup.py with –user:
$ python setup.py install --user
Launch Python and confirm module installation:
$ python ... >>> import munkres >>> print munkres.__version__ 1.0.7 >>> CTRL-D (to exit Python)
Python virtual environments
Users can create their own Python virtual environments to maintain different module versions for different projects.
Virtualenv is available on Pegasus for Python 2.7.3. By default,
virtualenv does not include packages that are installed globally. To give a virtual environment access to the global site packages, use
--system-site-packages flag. These directories do not need to be named exactly as shown.
Create a project folder, cd to the new folder, and create a
$ mkdir ~/python2 $ cd ~/python2 $ virtualenv test1 PYTHONHOME is set. You *must* activate the virtualenv before using it New python executable in test1/bin/python Installing setuptools, pip...done.
virtualenv with access to global packages:
$ virutalenv --system-site-packages test2
Activate the virtual environment:
$ source ~/python2/test1/bin/activate (test1) $ which python ~/python2/test1/bin/python
While the virtual environment is active (noted by the name in parentheses), any package you install normally will be placed into that virtual environment folder and isolated from the global Python installation. Note that using
--prefix= flags during module installation will place modules in those specified directories, NOT your currently active Python virtual environment.
Install Python modules normally with pip, easy_install, or setup.py:
(test1) $ pip install munkres or (test1) $ wget https://pypi.python.org/packages/source/m/munkres/munkres-1.0.7.tar.gz --no-check-certificate (test1) $ tar xvzf munkres-1.0.7.tar.gz (test1) $ cd munkres-1.0.7.tar.gz (test1) $ python setup.py install
Deactivate the virtual environment:
(test1) $ deactivate $
Compare two Python environments
Pip can be used to save a list of all packages and versions in the current environment (use
freeze). Compare using
sdiff to see which packages are different.
List the current environment, deactivate, then list the global Python environment:
(test1) $ pip freeze > test1.txt (test1) $ deactivate $ pip freeze > p2.txt
Compare the two outputs using
$ sdiff p2.txt test1.txt ... matplotlib==1.2.1 < misopy==0.5.0 | munkres==1.0.7 ... $
As seen above, the test1 environment has
munkres installed (and no other global Python packages).
To recreate in a Python virtual environment, use the the saved list:
(test2) $ pip install -r test1.txt Installing collected packages: munkres Running setup.py install for munkres ... Successfully installed munkres Cleaning up... (test2) $
Python virtual environment wrapper
Users can install
virtualenvwrapper in their own home directories to facilitate working with Python virtual environments. Once installed and configured,
virtualenvwrapper can be used to create new virtual environments and to switch between your virtual environments (switching will deactivate the current environment).
Virtualenvwrapper will read existing environment folders located in the WORKON_HOME directory. Set source to the local
virtualenvwrapper.sh location as shown below.
Install a local copy of
--always-copy for easy_install):
$ pip install --user virtualenvwrapper or $ easy_install --user --always-copy virtualenvwrapper
Set virtual environment home directory and source:
$ export WORKON_HOME=$HOME/python2 $ source ~/.local/bin/virtualenvwrapper.sh
Create a virtual environment using
virtualenvwrapper (also activates):
$ mkvirtualenv test3 PYTHONHOME is set. You *must* activate the virtualenv before using it New python executable in test3/bin/python Installing setuptools, pip...done. (test3) $
Activate or switch to a virtual environment:
(test3) $ workon test1 (test1) $
Deactivate the virtual environment:
(test1) $ deactivate $