Using MATLAB on Pegasus Cluster


Outline

I. Interactive Mode

a) Graphical Interface Mode
b) Graphical Interactive Mode with no graphical desktop window
c) Non-Graphical Interactive Mode

II. Batch Processing
III.Parallel Computations

a) Single-node parallel Matlab jobs on a single node (up to 16 cpu-s)
b) Multi-node parallel Matlab jobs (up to 32 cpu-s)
c) Notes on Matlab cluster configurations

I. Interactive Mode

There are several ways to run MATLAB commands/jobs interactively, with or without graphical interface.

a) Graphical Interface Mode

If you want to run the MATLAB using the graphical interface,
you need an X Server on your computer.

Windows users will need to install an X Server and Terminal emulator. We recommend Xming and Putty.

Linux/Unix/OS X users, can just forward X11 from their terminal when opening an ssh connection.

 xhost +
 ssh -X username@pegasus.ccs.miami.edu

Then, use the command below to launch matlab on one of the interactive compute nodes.

 module load matlab
 bsub -q interactive -Is -XF matlab

If you have more than one project and are required to specify the project name (PROJECT_ID) during the job submission, your submission should look as following:

 module load matlab
 bsub -q interactive -P PROJECT_ID -Is -XF matlab

The bsub command above will bring up the MATLAB graphical desktop on your computer.
You can then run MATLAB commands or scripts in the MATLAB command window.
The results will be shown in the MATLAB command window and the figure/plot will be displayed
in new graphical windows on your computer. See examples below.

>> x = rand(1,100);
>> plot(x);
>>
>> x = [0: pi/10: pi];
>> y = sin(x);
>> z = cos(x);
>> figure;
>> plot(x, y);
>> hold(‘on’);
>> plot(x, z, ‘--’);

b) Graphical Interactive Mode with no graphical desktop window

Running MATLAB in a full graphical mode may get slow depending on the network load. Running it with “-nodesktop” option will use your current terminal window (in Linux/Unix) as a desktop, while allowing you still to use graphics for figures and editor:

module load matlab
bsub -q interactive -P PROJECT_ID -Is -XF matlab -nodesktop

c) Non-Graphical interactive Mode

If your MATLAB commands/jobs do not need to show graphics such as figures and plots, or to use a built-in script editor, you may run the MATLAB in the non-graphical interactive mode.

Open a regular ssh connection to Pegasus.

module load matlab
bsub -q interactive -Is  matlab -P PROJECT_ID -nodisplay

If you are not requires to specify a project name (PROJECT_ID), remove the “-P PROJECT_ID” from the above line.
This will bring up the MALAB command window like the following

                            < M A T L A B (R) >
                  Copyright 1984-2013 The MathWorks, Inc.
                    R2013a (8.1.0.604) 64-bit (glnxa64)
                             February 15, 2013

No window system found.  Java option 'Desktop' ignored.

To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
>> 

To exit, type “exit” or “quit”. Again, remember to import the prepared LSF configuration file
mentioned above if you want to use MATLAB parallel computing.

II. Batch Processing

For the off-line non-interactive computations, you submit the MATLAB script to the LSF scheduler as outlined in the User’s Guide using the bsub command. See example below for submission a single-processor job.

bsub < example.job 
cat example.job
#BSUB -J example
#BSUB -q general
#BSUB -P PROJECT_ID
#BSUB -n 1
#BSUB -o example.o%J
#BSUB -e example.e%J
matlab -nodisplay -r my_script

In this example “my_script” corresponds to “my_script.m” in the current working directory.

After the job is finished, the results will be saved in the output file named “example.o######” where “######” is a Job ID number assigned by LSF when you submit your job.

III. Parallel Computations

MATLAB has software products to enable parallel computations for multi-core computers as well as for multiple-node computer clusters. The latter case scenario requires a job scheduler, such as LSF on Pegasus cluster.

The MATLAB product for the parallel processing that uses the cores of the same node is "Distributed Computing Toolbox/DCT" (also appears in MATLAB documentation under the name of "Parallel Computing Toolbox"). Licensed MATLAB software product for a computer cluster is called "Distributed Computing Engine/DCE" (also appears in documentation as "MATLAB Distributed Computing Server").

a) Single-node parallel MATLAB jobs on a single node (up to 16 cpu-s)

For a single-node parallel jobs, MATLAB Distributed Computing Toolbox (licensed software) is used. It has a build-in default MATLAB cluster profile 'local', from which the pool of MatlabWorkers could be reserved for computations. The default number of MatlabWorkers is 12, but you could specify up to 16 on a single Pegasus node. Please refer to MATLAB documentation on the ways to adapt your script for multi-processor calculations. One of the parallel tools in MATLAB is the "parfor" loop replacing the regular "for" loop, and in the example is given below:

%==============================================================
% dct_example.m
% Distributed Computing Toolbox (DCT)
% Example: Print datestamp within a parallel "parfor" loop
%==============================================================
%% Create a parallel pool of MatlabWorkers on the current working node:
parpool('local',16);
% The test loop size
N = 40;       
tstart = tic();
parfor(ix=1:N)
  ixstamp = sprintf('Iteration %d at %s\n', ix, datestr(now));
  disp(ixstamp);
  pause(1);
end
cputime=toc(tstart);
toctime= sprintf('Time used is %d seconds', cputime);
disp(toctime)
%% delete current parallel pool:
delete(gcp)  

b) Multi-node parallel MATLAB jobs (up to 32 cpu-s)

For running multi-processor MATLAB jobs that involve more than a single node, MATLAB Distributed Computer Engine (licensed software) is used, with currently 32 licenses available on Pegasus. The parallel LSF MATLAB cluster needs to be configured. After loading the matlab module, you could import the default LSF parallel configuration as following:

matlab -nodisplay -r "parallel.importProfile('/share/opt/MATLAB/etc/LSF1.settings');exit";reset

This command only needs to be run once. It imports the cluster profile named ‘LSF1′ that is configured to use up to 32 MatlabWorkers and to submit MATLAB jobs to the ‘general’ pegasus queue. This profile does not have a project name (PROJECT_ID) associated with the job, and you may need to coordinate the project name for the LSF job submission. This could be done by running the following script (only once!) during your matlab session:

%% conf_lsf1_project_id.m
%% Verify that LSF1 profile exists, and indicate the current default profile:
[allProfiles,defaultProfile] = parallel.clusterProfiles()
%% Define the current cluster object using LSF1 profile
myCluster=parcluster('LSF1')
%% View current submit arguments:
get(myCluster,'SubmitArguments')
%% Set new submit arguments, change PROJECT_ID below to your current valid project:
set(myCluster,'SubmitArguments','-q general -P PROJECT_ID')
%% Save the cluster profile:
saveProfile(myCluster)
%% Set the 'LSF1' to be used as a default cluster profile instead of a 'local'
parallel.defaultClusterProfile('LSF1');
%% Verify the current profiles and the default:
[allProfiles,defaultProfile] = parallel.clusterProfiles()

The above script also reviews your current settings of the cluster profiles. You could now use the cluster profile for distributed calculations on up to 32 cpu-s, for example, to create a pool of MatlabWorkers for a "parfor" loop:

%=========================================================
% dce_example.m
% Distributed Computing Engine (DCE) 
% Example: Print datestamp within a parallel "parfor" loop
%=========================================================
myCluster=parcluster('LSF1')
% Maximum number of MatlabWorkers is 32 (number of MATLAB DCE Licenses)
parpool(myCluster,32);
% The test loop size
N = 40;  
tstart = tic();
parfor(ix=1:N)
  ixstamp = sprintf('Iteration %d at %s\n', ix, datestr(now));
  disp(ixstamp);
  pause(1);
end
cputime=toc(tstart);
toctime= sprintf('Time used is %d seconds', cputime);
disp(toctime)
delete(gcp)

Please see MATLAB documentation on more ways to parallelize your code.

There may be other people running Distributed Computing Engine and thus using several licenses. Please check the license count as following (all in a single line):

/share/opt/MATLAB/R2013a/etc/lmstat -S MLM -c /share/opt/MATLAB/R2013a/licenses/network.lic

Find the information about numbers of licenses used for the "Users of MATLAB_Distrib_Comp_Engine", "Users of MATLAB", and "Users of Distrib_Computing_Toolbox".

c) Note on Matlab cluster configurations

After you imported the new cluster profile, it will remain in your available cluster profiles, validated using parallel.clusterProfiles() function. You could create and change profiles, and save them using SaveProfile and SaveAsProfile methods on a cluster object. In the above examples, "myCluster" is the cluster object.
Also, you could create, import, export, delete, and modify the profiles through the Cluster Profile Manager accessible via MATLAB menu in a graphical interface. It is accessed from the "HOME" tab in the GUI desktop window under "ENVIRONMENT" section, ->"Parallel"->"Manage Cluster Profiles"

Cluster Profile Manager

MATLAB Cluster Profile Manager

You could also create your own LSF configuration from the Cluster Profile Manager. Choose "Add"->"Custom"->"3RD PARTY CLUSTER PROFILE"->"LSF" as shown below:

Cluster Profile Manager: new LSF cluster

Matlab Cluster Profile Manager


… and configure to your needs:
New LSF cluster in Matlab

Configure a new LSF cluster in MATLAB