Compute Systems

The University of Miami maintains one of the largest centralized academic cyber infrastructures in the country with numerous assets.  Since 2007, the core has grown from zero advanced computing cyberinfrastructure to a regional high-performance computing environment that currently supports more than 1,700 users, 240 TFlops of computational power, and more than 3 Petabytes of disk storage.

The University’s latest acquisition, TRITON, is rated one of the Top 5 Academic Institution Supercomputers in U.S. for 2019, and is UM’s first GPU-accelerated high performance computing (HPC) system, representing a completely new approach to computational and data science for all University campuses. Built using IBM Power Systems AC922 servers, this system was designed to maximize data movement between the IBM POWER9 CPU and attached accelerators like GPUs. The University’s first Supercomputer “Pegasus”, an IBM IDataPlex system, was ranked at number 389 on the November 2012 Top 500 Supercomputer Sites list.

Triton Specs

• IBM Power9/Nvidia Volta – 6 Racks
• IBM declustered storage – 2 Racks
• 96 IBM Power 9 servers
• 30TB RAM (256/node)
• 1.2 Petaflop Double Precision
• 240 Tflop Deep Learning
• 64 bit scalar
• 100 GB/sec Storage
• 150 TB shared flash storage
• 400 TB shared home
• 2 @ 1.99 TB ssd local storage

At present, CCS maintains several clusters and application servers.

Data Storage

CCS offers an integrated storage environment for both structured (relational) and unstructured (flat file) data. These systems are specifically tuned for CCS’ data type and application requirements, whether they are serial access or highly parallelized. Each investigator or group has access to its own area and can present his or her data through a service-oriented architecture (SOA) model. Researchers can share their data via access control lists (ACLs), which ensure data integrity and security while allowing flexibility for collaboration.

CCS offers structured data services through the most common relational database formats, including: Oracle, MySQL, and PostgreSQL. Investigators and project teams can access their space through SOA and utilize their resources with the support of an integrated backend infrastructure.

The CCS flat file storage environment is built as a multi-tier solution combining high-speed storage with dense high capacity storage in a tiered architecture, all supported by IBM’s GPFS.  Our HPC/Global tier (700TB) is available on all compute nodes.  This storage is designed for massively parallel work and has been clocked at 157,000 IOP/sec and over 20 GB/sec bandwidth.

Our standard tier of storage (2.8 PB) is designed for general-purpose data storage, analysis, and presentation of data to collaborators both within and without the University of Miami.  All tier 2 storage is available from all systems including our visualization cluster.  Several data management tools are available for tier 2 storage including public presentation, long-term archive, deduplication, encryption, and HSM.

Our archival tier of storage (2.5 PB) leverages several platforms for keeping critical data safe.  By using a combination of tape and disk technologies, we are able to reduce restore times significantly while still ensuring data integrity.

Advanced Computing Core Expertise

The Advanced Computing Team has in-depth experience in various scientific research areas with extensive experience in parallelizing or distributing codes written in Fortran, C, Java, Perl, Python and R. The Team is active in contributing to Open Source software efforts including: R, Python, the Linux Kernel, Torque, Maui, XFS and GFS. The Team also specializes in scheduling software (LSF) to optimize the efficiency of the HPC systems and adapt codes to the CCS environment. The Advanced Computing Program has expertise in parallelizing code using both MPI and OpenMP depending on the programming paradigm. CCS has contributed several parallelization efforts back to the community in projects such as R, WRF, and HYCOM.

The Program specializes in implementing and porting open source codes to CCS’ environment and often contributes changes back to the community. CCS currently supports more than 300 applications and optimized libraries on its computing environment. The Advanced Computing Team are experts in implementing and designing solutions in the three different variants of Unix.