Several compiler suites are available on Pegasus. Table 11 summarizes the installed compilers.
Table 11: Summary of available compilers on the Pegasus cluster
|Compiler||Portland Group -7.1.5||Intel – 10.1||MPI||GNU|
The MPI compilers are wrapper scripts which invoke the Portland Group, Intel or GNU compilers depending on the module chosen. For example if you wanted to use the Intel compilers, you must load the intel module.
$ which ifort /usr/bin/which: no ifort in (/share/apps/... $ module load intel $ which ifort /share/apps/intel/Compiler/11.1/069/bin/intel64/ifort $ which icc /share/apps/intel/Compiler/11.1/069/bin/intel64/icc
If you want to use the Portland compilers, you must load the pgi module.
[pdavila@u04 ~]$ which pgcc /usr/bin/which: no pgcc in (/share/apps/... $ module load pgi $ which pgcc /share/apps/pgi/10.3/linux86-64/10.3/bin/pgcc
See module section below for more information.
The Modules Package is installed on the Pegasus cluster which allows users to quickly switch between different programming environments. The Modules package sets up appropriate environment variables such as PATH, MANPATH, LD_LIBRARY_PATH etc. depending on the module chosen. Table 12 lists commonly used modules commands.
Table 12: Summary of modules package usage
|module avail||module avail||lists all available modules|
|module load package||module load ifort||loads a module e.g., the ifort package|
|module list||module list||list modules currently loaded|
|module switch old new||module switch ifort pgi||replace old module with new|
|module purge||module purge||restores original setting by unloading all modules|
The currently available modules as listed by the module avail command is shown below:
[user@Pegasus ~]$ module avail ------------- /share/apps/modules/Modules/versions ---------------- 3.2.7 ------------- /share/apps/modules/Modules/3.2.7/modulefiles ------- R fsl mihg openmpi/intel atlas gcc/4.4.3 modules paraview blast gdb mpich2/gcc pgi casava gromacs mpich2/intel pvm clustalw gromacs-pbs mpich2/pgi python-2.6.4 cvs gsl/gcc netcdf-3.6.3/gcc valgrind eclipse gsl/intel netcdf-3.6.3/intel visit emboss impute netcdf-4.0.1/gcc xmgrace ensembl intel netcdf-4.0.1/intel yasara erange java null fftw lsf openeye freesurfer maq openmpi/gcc ------------ /nethome/sagebase/modulefiles ------------------------ sage [user@Pegasus ~]$
Typically, modules are loaded as a part of the login by placing module commands in the
.bash_profile/.profile (for bash/sh/ksh users) or .login for (csh/tcsh users).
Assume your Fortran 90/C source program is stored in file ‘program.f/program.c’. These files
are compiled with the command:
Portland Group Fortran: pgf90 program.f -o program Intel Fortran: ifort program.f -o program Portland Group C: pgcc program.c -o program Intel Group C: icc program.c -o program
The ‘-o program’ option instructs the compiler to name the executable program ‘program’. If the ‘-o’ option is not used, the executable program is called ‘a.out’ by default. If your source program is spread over multiple files, say for example that a subroutine in the file ‘program.f’ calls a subroutine/function that is declared in the file ‘program2.f/program2.c’. Then one can compile in two steps:
Portland Group Fortran: pgf90 -c program2.f pgf90 program.f program2.o -o program Intel Fortran: ifort -c program2.f ifort program.f program2.o -o program Portland Group C: pgcc -c program2.c pgcc program.c program2.o -o program Intel C: icc -c program2.c icc program.c program2.o -o program
The ‘-c’ option instructs the compiler to create an object file (program2.o) instead of an executable file. This object file is used at a later step of the compilation. In the last step of the compilation, all object files (or one source file with other object files) are usually linked together to create the executable file. By default, the compiler links these object files together with several libraries that are present on the system and that contain implementations of the language standard (for example Fortran intrinsics like abs, sin, mod, …).
When your code consists of many source files, the venerable Unix make utility can automate the maintenance, update, compilation, and regeneration of object and executable files. The make utility requires by default a file called ‘Makefile’ in which you specify your compiler options, source/object files, and rules for compilation. After you have created the Makefile, you can build your program with the command ‘make’ or ‘gmake’ (GNU make). The /nethome/examples directory contains an example Makefile for GNU make (which we recommend), that can be used to build an example Fortran program. The sample Makefile provides a template and can be edited to suit
your needs Writing Makefiles is not always easy but can simplify your life considerably. Contact user support for help when needed.
Most options for compiling sequential programs can also be used for compiling parallel OpenMP (or threaded) programs. For example:
Portland Fortran: pgf90 -mp program.f Intel Fortran: ifort -openmp program.f Portland C: pgcc -mp program.c Intel C: icc -openmp program.c
Two open source MPI (Message Passing Interface) libraries MPICH3/OpenMPI are installed. A
module corresponding to the compiler you wish to use must be loaded in order to set up the
correct environment. MPI Libraries are specific to compiler and to networking protocol.
For example, to use MPICH3 library compiled with the Intel compiler over the Ethernet
networking protocol, which is the only protocol available, the command would be
module load intel openmpi/intel
Once the module is loaded the following commands should be used to compile programs that use MPI code:
Fortran: mpif90 program.f C: mpicc program.c
The ‘mpi…’ commands are wrapper shell scripts that invoke the chosen compiler. Flags are passed by mpif90 script to the back-end compiler, so any of the compiler options can be used on the mpi shell script. By default, the mpi.. scripts pass the proper MPI header files to the compilers. It is therefore not necessary to specify the directory (normally with the ‘-I’ option) where the MPI header file is located. The MPI compilers also link in the MPI libraries by default (and linking with something like ‘-L… -lmpi’ is not needed).
The different compilers accept many options e.g. for debugging, optimization for code size or performance.
Use ‘man pgf90′ or man ifort to get an overview of the options for the Fortran compilers and
‘man pgcc/pgCC’ or ‘man icc’ for the C/C++ compilers for detailed information on the available options.
For basic performance and run-time optimization a good starting point is to compile your code with the options
Portland Group: -fastsse Intel: -O3
You can use PGDBG to debug MPI-parallel and OpenMP thread-parallel programs on Pegasus. PGDBG complements the powerful MPI and OpenMP parallel performance
analysis profiler, PGPROF. Some features of PGDBG:
Control Threads and Processes Separately – The PGDBG cluster debugger provides the ability to separately debug and control OpenMP threads, pthreads, and MPI processes on your Linux cluster. You can Step, Next, Break, Halt, Wait or Continue on threads or processes individually or collectively as a group.
Powerful GUI Windows – PGDBG provides a comprehensive set of GUI windows at your disposal to assist you in this difficult task. Using a single window, you now have precise control over each node of your cluster or each thread of your multiple-CPU SMP node. The Main GUI window displays source code for your F77, F95, C or C++ program and it has buttons for one-touch commands such as Run, Break, Quit, Print, or Stack traceback.
Buttons on the Main window also allow you to select and control individual or collective threads and processes as well as the commands to access them. Input and output is displayed in the Main window or in a separate I/O Window. A Disassembly window shows you how the currently executing high-level source code has been compiled into assembly language. The assembly is interleaved with the source code as well. Assembly language Stepping is enabled as well as breakpoint indicators. Separate Active Process and Active Thread summary windows can be spawned that give you a current status on each active thread or process.
A Register window can be used to display register values in a variety of formats – integer, hex, floating point, etc. Custom Display windows can be used to follow values for important variables or memory regions in your executing program. A separate I/O Window can receive the stdout and stderr from your debug session.
The Intel Debugger is a component of the Intel compilers. It is a full-featured symbolic source code application debugger that helps programmers locate run-time programming errors (i.e., bugs) in their code. It provides extensive debugging support for the following languages: C, C++ and Fortran (including Fortran 90). It also provides a choice of command-line or graphical user interface (GUI). The command-line interface provides DBX-like (default) or GDB (GNU Debugger)-like (optional) command syntax. The GUI supports all Intel Debugger command-line functionality. The main window includes the basic debugging and convenience features, optional windows, various pop-up menus and dialog boxes, and a command-entry prompt. Intel Debugger works with the following compilers Intel C/C++, Fortran GCC, G++ and G77 compilers.
gprof is a simple text-based utility that provides procedural-level profiling of serial and parallel codes. This helps users to identify how much time is being spent in subroutines and functions. xprofiler generates a graphical display of the performance, and provides application profiling at the source statement level. Both gprof and xprofiler are very simple to use:
- Compile the code with the -pg option, in addition to optimization flags. If you use xprofiler, using -g in
addition to the -pg option will offer profiling at source line level, however the -g will degrade the
performance and is incompatible with some optimization flags (e.g. inlining).
- Run the parallel code as usual.
- Each process will write an additional file to disk named gmon.out.pid
- Process the output with gprof or xprofiler where exec_file is the name of the compiled executable:
- gprof exec_file gmon.out.pid
The following table 13 may help you in porting your code (esp. makefiles) from IBM to LINUX and/or to change compilers on the Linux cluster. Always check options carefully before using it.
Table 13: Summary of compiler option on IBM and Linux Platforms
list compiler options
|man xlf||-help||-help||man g77|
Fortran source file format
|-qfixed/qfree||-fixed/free||-M fixed/free||-ffixed-form/free-form||fixed/free format|
Generate 64-bit executables
Generate 32-bit executables
|-q32 (default)||change path setting||-tp k8-32||-m32|
Standard optimization options
|-O[1,2,3]||-O[1,2,3]||-O[1,2,3]||-O[1,2,3]||same options, different meanings (!)|
|-qarch=auto -qtune=auto -qcache=auto||-xW||-tp p7||-march=em64t|
|-O4 -qhot -ipa||-fast -ipo -static||-fastsse||-ffast-math -funroll-all-loops -malign-double|
Local variable Placement in Fortran
|-qsave/nosave||-save/auto||-Msave/nosave||-fstatic||off/on the stack|
Options for rounding(not equivalent)
|-qstrict||-mp||-mp1||-pc XX||see “man pgf90″ for -pc option|
Runtime array bounds checking
C preprocessor directives