Programming Environment

Compilers

Several compiler suites are available on Pegasus. Table 11 summarizes the installed compilers.

Table 11: Summary of available compilers on the Pegasus cluster

The MPI compilers are wrapper scripts which invoke the Portland Group, Intel or GNU compilers depending on the module chosen. For example if you wanted to use the Intel compilers, you must load the intel module.

$ which ifort
/usr/bin/which: no ifort in (/share/apps/...
$ module load intel
$ which ifort
/share/apps/intel/Compiler/11.1/069/bin/intel64/ifort
$ which icc
/share/apps/intel/Compiler/11.1/069/bin/intel64/icc

If you want to use the Portland compilers, you must load the pgi module.

[pdavila@u04 ~]$ which pgcc
/usr/bin/which: no pgcc in (/share/apps/...
$ module load pgi
$ which pgcc
/share/apps/pgi/10.3/linux86-64/10.3/bin/pgcc

See module section below for more information.

Modules

The Modules Package is installed on the Pegasus cluster which allows users to quickly switch between different programming environments. The Modules package sets up appropriate environment variables such as PATH, MANPATH, LD_LIBRARY_PATH etc. depending on the module chosen. Table 12 lists commonly used modules commands.

Table 12: Summary of modules package usage

The currently available modules as listed by the module avail command is shown below:

[user@Pegasus ~]$ module avail
------------- /share/apps/modules/Modules/versions ----------------
3.2.7

------------- /share/apps/modules/Modules/3.2.7/modulefiles -------
R                  fsl                mihg               openmpi/intel
atlas              gcc/4.4.3          modules            paraview
blast              gdb                mpich2/gcc         pgi
casava             gromacs            mpich2/intel       pvm
clustalw           gromacs-pbs        mpich2/pgi         python-2.6.4
cvs                gsl/gcc            netcdf-3.6.3/gcc   valgrind
eclipse            gsl/intel          netcdf-3.6.3/intel visit
emboss             impute             netcdf-4.0.1/gcc   xmgrace
ensembl            intel              netcdf-4.0.1/intel yasara
erange             java               null
fftw               lsf                openeye
freesurfer         maq                openmpi/gcc

------------ /nethome/sagebase/modulefiles ------------------------
sage

[user@Pegasus ~]$

Typically, modules are loaded as a part of the login by placing module commands in the
.bash_profile/.profile (for bash/sh/ksh users) or .login for (csh/tcsh users).

Assume your Fortran 90/C source program is stored in file ‘program.f/program.c’. These files
are compiled with the command:

Portland Group Fortran:
  pgf90 program.f -o program

Intel Fortran:
  ifort program.f -o program

Portland Group C:
  pgcc   program.c -o program

Intel Group C:
  icc   program.c -o program

The ‘-o program’ option instructs the compiler to name the executable program ‘program’. If the ‘-o’ option is not used, the executable program is called ‘a.out’ by default. If your source program is spread over multiple files, say for example that a subroutine in the file ‘program.f’ calls a subroutine/function that is declared in the file ‘program2.f/program2.c’. Then one can compile in two steps:

Portland Group Fortran:
  pgf90 -c program2.f
  pgf90 program.f program2.o -o program

Intel Fortran:
  ifort -c program2.f
  ifort  program.f program2.o -o program

Portland Group C:
  pgcc -c program2.c
  pgcc program.c program2.o -o program

Intel C:
  icc -c program2.c
  icc program.c program2.o -o program

The ‘-c’ option instructs the compiler to create an object file (program2.o) instead of an executable file. This object file is used at a later step of the compilation. In the last step of the compilation, all object files (or one source file with other object files) are usually linked together to create the executable file. By default, the compiler links these object files together with several libraries that are present on the system and that contain implementations of the language standard (for example Fortran intrinsics like abs, sin, mod, …).

When your code consists of many source files, the venerable Unix make utility can automate the maintenance, update, compilation, and regeneration of object and executable files. The make utility requires by default a file called ‘Makefile’ in which you specify your compiler options, source/object files, and rules for compilation. After you have created the Makefile, you can build your program with the command ‘make’ or ‘gmake’ (GNU make). The /nethome/examples directory contains an example Makefile for GNU make (which we recommend), that can be used to build an example Fortran program. The sample Makefile provides a template and can be edited to suit
your needs Writing Makefiles is not always easy but can simplify your life considerably. Contact user support for help when needed.

Compiling OpenMP Programs

Most options for compiling sequential programs can also be used for compiling parallel OpenMP (or threaded) programs. For example:

Portland Fortran:
  pgf90 -mp program.f

Intel Fortran:
  ifort -openmp program.f

Portland C:
  pgcc -mp program.c

Intel C:
  icc -openmp program.c

Compiling MPI Programs

Two open source MPI (Message Passing Interface) libraries MPICH3/OpenMPI are installed. A
module corresponding to the compiler you wish to use must be loaded in order to set up the
correct environment. MPI Libraries are specific to compiler and to networking protocol.
For example, to use MPICH3 library compiled with the Intel compiler over the Ethernet
networking protocol, which is the only protocol available, the command would be

module load intel openmpi/intel

Once the module is loaded the following commands should be used to compile programs that use MPI code:

Fortran:
  mpif90 program.f

C:
  mpicc  program.c

The ‘mpi…’ commands are wrapper shell scripts that invoke the chosen compiler. Flags are passed by mpif90 script to the back-end compiler, so any of the compiler options can be used on the mpi shell script. By default, the mpi.. scripts pass the proper MPI header files to the compilers. It is therefore not necessary to specify the directory (normally with the ‘-I’ option) where the MPI header file is located. The MPI compilers also link in the MPI libraries by default (and linking with something like ‘-L… -lmpi’ is not needed).

Frequently used Compiler Options

The different compilers accept many options e.g. for debugging, optimization for code size or performance.
Use ‘man pgf90′ or man ifort to get an overview of the options for the Fortran compilers and
‘man pgcc/pgCC’ or ‘man icc’ for the C/C++ compilers for detailed information on the available options.

For basic performance and run-time optimization a good starting point is to compile your code with the options

Portland Group:
	-fastsse 

Intel:
	-O3

Debugging Tools

Portland Linux Cluster Debugger (PGDBG)

You can use PGDBG to debug MPI-parallel and OpenMP thread-parallel programs on Pegasus. PGDBG complements the powerful MPI and OpenMP parallel performance
analysis profiler, PGPROF. Some features of PGDBG:

Control Threads and Processes Separately – The PGDBG cluster debugger provides the ability to separately debug and control OpenMP threads, pthreads, and MPI processes on your Linux cluster. You can Step, Next, Break, Halt, Wait or Continue on threads or processes individually or collectively as a group.

Powerful GUI Windows – PGDBG provides a comprehensive set of GUI windows at your disposal to assist you in this difficult task. Using a single window, you now have precise control over each node of your cluster or each thread of your multiple-CPU SMP node. The Main GUI window displays source code for your F77, F95, C or C++ program and it has buttons for one-touch commands such as Run, Break, Quit, Print, or Stack traceback.

Buttons on the Main window also allow you to select and control individual or collective threads and processes as well as the commands to access them. Input and output is displayed in the Main window or in a separate I/O Window. A Disassembly window shows you how the currently executing high-level source code has been compiled into assembly language. The assembly is interleaved with the source code as well. Assembly language Stepping is enabled as well as breakpoint indicators. Separate Active Process and Active Thread summary windows can be spawned that give you a current status on each active thread or process.

A Register window can be used to display register values in a variety of formats – integer, hex, floating point, etc. Custom Display windows can be used to follow values for important variables or memory regions in your executing program. A separate I/O Window can receive the stdout and stderr from your debug session.

Intel Debuggers

The Intel Debugger is a component of the Intel compilers. It is a full-featured symbolic source code application debugger that helps programmers locate run-time programming errors (i.e., bugs) in their code. It provides extensive debugging support for the following languages: C, C++ and Fortran (including Fortran 90). It also provides a choice of command-line or graphical user interface (GUI). The command-line interface provides DBX-like (default) or GDB (GNU Debugger)-like (optional) command syntax. The GUI supports all Intel Debugger command-line functionality. The main window includes the basic debugging and convenience features, optional windows, various pop-up menus and dialog boxes, and a command-entry prompt. Intel Debugger works with the following compilers Intel C/C++, Fortran GCC, G++ and G77 compilers.

Basic Profiling

gprof is a simple text-based utility that provides procedural-level profiling of serial and parallel codes. This helps users to identify how much time is being spent in subroutines and functions. xprofiler generates a graphical display of the performance, and provides application profiling at the source statement level. Both gprof and xprofiler are very simple to use:

• Compile the code with the -pg option, in addition to optimization flags. If you use xprofiler, using -g in
  addition to the -pg option will offer profiling at source line level, however the -g will degrade the
  performance and is incompatible with some optimization flags (e.g. inlining).
• Run the parallel code as usual.
• Each process will write an additional file to disk named gmon.out.pid
• Process the output with gprof or xprofiler where exec_file is the name of the compiled executable:
• gprof exec_file gmon.out.pid

Portability Tips

The following table 13 may help you in porting your code (esp. makefiles) from IBM to LINUX and/or to change compilers on the Linux cluster. Always check options carefully before using it.

Table 13: Summary of compiler option on IBM and Linux Platforms