### Basin Scale Ocean modeling using HYCOM

*Z. Garraffo and A. Srinivasan*

We use the HYbrid Coordinate Ocean Model extensively for high-resolution simulations. A sample HYCOM application is 60 year multi-decadal North Atlantic simulations. The computations are done at a resolution of 9 km at the equator, 7 km at mid latitudes, and 4 km in the Arctic, and with 32 vertical layers. The model is being forced with NCEP fields from 1948 through 1979, and with ECMWF fields from 1979 to the present. The interannual simulations would allow assessing circulation variability at interannual, decadal and longer time scales. With a 7 km (mid-latitude) grid, these simulations have sufficient resolution to provide a baseline depiction of the coastal regions, and can be used as boundary conditions for nesting of coastal Atlantic areas.

These simulations use over 80 million grid points and a very small time step and hence are computationally very demanding. The simulations are typically run on 256 or more procesors. The model is domain decomposed and parallelized using a SPMD approach in which the region is divided in smaller sub-domains or tiles, each processor dealing with one tile (Figure 8).

Figure 8: *Left Panel*:Sea Surface height from a HYCOM North Atlantic Simulation. Click here for model animation. *Right Panel*: Decomposition of the domain in tiles, for parallel computations.

### Ocean response to Hurricanes

*G. R. Halliwell*

A regional example of Hybrid Coordinate Ocean Model is the Gulf of Mexico (GoM-HYCOM) configuration used for studying ocean response to hurricanes in a region with complex oceanographic features (Loop Current and eddies). This work is necessary to improve ocean model performance, which can potentially lead to improved hurricane intensity forecasts by coupled hurricane forecast models.

Figure 9: Sea Surface Temperature (SST) in the Gulf of Mexico before and after hurricane Ivan

Comparison of Sea Surface Temperature (SST) before and after hurricane Ivan between a GoMHYCOM simulation and infrared satellite measurements. The two cold spots are regions where the supply of heat from the ocean to the storm is reduced, which acts to reduce storm intensity. They form where two cyclonic, cold-core eddies (blue spots, top panel) exist when the storm hits. The upper ocean warm layer is very thin over these features, allowing the storm to efficiently mix deeper cold water up to the surface.

### Coastal Ocean Modeling

*V. Kourafalou and H. Kang*

Dr. Villy Kourafalou and her group are actively involved in numerical modeling studies of circulation and transport processes in coastal and shelf seas. Their research topics include wind-driven circulation, river plume dynamics and interactions of coastal flows with large scale currents, with an emphasis on topographically complex marginal seas. Applications on ecosystem dynamics and development of operational oceanographic networks and forecasts are under way. Current research projects (http://coastalmodeling.rsmas.miami.edu) include hydrodynamic models in the Intra-America Seas, the Gulf of Mexico, South Florida and the Aegean Sea. A multi-nested, downscaling approach has been developed to connect local to regional and global models. The modeling studies are closely linked to observational studies for model forcing and validation. Recent research activities have focused on the connectivity among remote coastal regions and the impacts of the related physical processes on ecosystem dynamics. Implications on socioeconomic and management issues are being established through collaboration with the National Oceanic and Atmospheric Administration and the South Florida Water Management District.

An example project is the high-resoultion modleing of the Florida Keys with several applications:

- Eddies in the Straits of Florida
- Tortugas Eddy
- High Resolution Currents
- Sea Level Changes Due to Hurricanes
- Fish Recruitment in the Florida Keys

### Understanding the Role of the Physical Environment on Marine Organisms in Tropical Ecosystems

*L. M. Cherubin*

Physicists tend to study the ocean as a feature empty of life and conversely, it could be said that biologists consider it as empty of water – exploring its contents but eschewing the effects of currents and waves. That is to say, their individual concerns rarely cross paths. This is understandable, owing to the immense complexities of both fields in and of themselves. But as scientists, we must pool our resources and collaborate across fields of study, sharing information in order to shed light on issues that will influence all of us. This is why I have decided to dedicate my research to the open field where the physical environment of marine ecosystems meets with its living inhabitants. In an era of biodiversity collapse and climate change, both on land and in the aquatic world, understanding the interactions of marines organisms with the dynamics of their physical environment becomes necessary. In my projects I have chosen to focus on endangered species of coral and fish such as Elkhorn coral and Nassau grouper. I’m particularly interested in the transport of eggs and larvae through motions of oceanic waters. While I’m applying my knowledge and my understanding of the ocean physics to the transport of marine organisms, Claire Paris, a biophysical modeler in the division of Applied Marine Physcis at the University of Miami, and my main partner in this collaboration, studies the behavior (swimming, orientation capabilities, development) of marine organisms in their moving and changing environment. So far, our research has focused on tropical species in the Caribbean where reef ecosystems are degrading under growing threats… Our long term goal is to understand the response of the marine organisms to a physical environment which carries threats, such as carbon dioxide, warmer temperatures, pathogens, terrestrial sediments and pollutants. Transport itself might become a threat if it changes too much from the usual or past patterns as a result of climate change …

I use numerical model that simulate the ocean circulation such as MICOM, HYCOM and ROMS in computationally demanding configurations to study these interaction in a more realistic framework (Figure 11)

Figure 11: Figure panel shows an example ocean model simulation for the Bahamas region performed as a part of the Bahamas Biocomplexity project. Six-hour interval snapshots of temperature and current vectors in the Bahamas model reveal the influence of the tide on the Bahamas shallow banks (for instance south and west of Andros…).

### Unstructured Grid Ocean Modeling

*Mohamed Iskandarani*

Dr. Iskandarani has been working on the development of unstructured grids ocean models. The geometric flexibility inherent in these grids improves the representation of complex coastline in computer models, enables multiscale simulation via variable resolution computational grids, and mitigates the cost of ocean modelling by localizing resolution in dynamically active regions. The result has been a series of models that differ primarily in their representation of the vertical solution structure. These include two and three-dimensional models, with either an isopycnal or terrain-following coordinate system, that solve the hydrostatic Navier-Stokes equations; a non-hydrostatic three-dimensional is currently under development.

The original shallow water solver is based on a spectral element discretization of the flow region. The spectral element paradigm combines the geometric flexibility of low-order finite element methods with the high convergence rates of spectral methods. Its numerical attributes include very low numerical dissipation, and excellent phase propagation properties; furthermore, the dense local computational kernels at the element level, and the sparse inter-element communication patterns lead to extremely good scalability on parallel computers. A sample application is shown in Figure 12 where the model was applied as a 1½ inverted layer model to simulate the abyssal flow in the indian ocean, and the dispersal of 3He plume emanating from line sources along the south-western ridges and north-eastern ridges bounding the domain. The unstructured grid was critical in enabling the economical representation of the flow through gaps at 30 S and 25 S in the 90 East ridge. The flow exchange between the eastern and western Central Indian basin occurs through zonal jets aligned along the gaps, and lend the 3He plume its south-eastern bending character in line with observations reported in Srinivasan et al. (2004). In additional to our spectral element shallow water solver, a high-order finite volume solver, with a nonlinear Flux Corrected Transport-based limiter has also been developed. This model is a precursor to a multi-layer version capable of handling vanishing layer thickness and isopycnals intersecting topography. This model is targeted at applications where water mass conservation during long integration time is critical, or where baroclinic dynamics need to be represented with a small number of vertical degrees of freedom.

The three-dimensional spectral element model uses a terrain-following coordinate system with a relatively low order spectral element element representation in the vertical (Iskandarani et al., 2003). This model has been retrofitted with a Discontinuous Galerkin Method (DGM) to improve the model performance for advection dominated flow, and to stagger the tracer and vertical velocity with respect to the velocity collocation points. DGM, unlike the traditional finite element methods, does not require function continuity at inter-element boundaries, and hence is relatively insensitive to unresolved large gradients in the solution; furthermore numerical noise is damped selectively and remains localized. The improvements in model capabilities have been documented in Iskandarani et al. (2005).

An example of DGM’s capabilities in shown in Figure 13, which shows the mixing of a two-density fluid adjusting under the force of gravity. The simulation was performed with a non-hydrostatic, vorticitiystreamfunction- based spectral element model. The model relies on the traditional (continuous) spectral element method to discretize vorticity and streamfunction, and on DGM for the discretization of density. The DGM was critical in enhancing the model’s stability by damping the numerical noise created by sharp density gradients; a continuous formulation would have required 100 fold increase in the resolution. These simulations are part of an ongoing effort to improve our understanding of oceanic mixing and our ability to parametrize it in large scale ocean models. Further improvements in algorithms and computing capabilities will allow us to push these simulation to higher and more realistic values of the Reynolds and Peclet numbers.

Figure 12: Upper left: Elemental partition of the abyssal basin below the 3000 m isobath. Each element contains a 7*7 local grid for the velocity interpolation. Upper right: 3He distribution after 800 years of integration. Lower left: velocity vector, and lower right interface displacement.

Figure 13: Simulation of a two-density fluid mixing using a spectral element model solving the vorticity streamfunction equations. The Reynolds number based on the gravity wave speed and channel height is 10, 000, while the Peclet number (ratio of viscous to diffusion coefficient) is 7.

### 3D Nonhydrostatic Spectral Element Modelling of Ocean Mixing

*Tamay ÖOzgökmen*

Özgökmen has been collaborating with Paul F. Fischer (http://www-unix.mcs.anl.gov/fischer/) in Mathematics and Computer Science Division of Argonne National Laboratory using Nek5000, the nonhydrostatic spectral element model developed by Fischer. Nek5000 integrates the unsteady incompressible Navier-Stokes equations within Boussinesq approximation. Nek5000 handles general three-dimensional flow configurations, supports a broad range of boundary conditions for hydrodynamics, and accommodates multiple-species transport.

For problems having smooth solutions, the spectral element method achieves exponential convergence with N, despite having only Cº continuity (which is advantageous for parallelism). The convection operator exhibits minimal numerical dissipation and dispersion, which is important for high-Reynolds number applications. Efficient solution of the Navier-Stokes equations in complex domains depends on the availability of fast solvers for sparse linear systems. Nek5000 uses as a preconditioner the additive overlapping Schwarz method introduced by Dryja and Widlund (1987) and developed in the spectral element context by Fischer (1997) and Fischer et al. (2000). The key components of the overlapping Schwarz implementation are fast local solvers that exploit the tensor-product form, and a parallel coarse-grid solver that scales to 1000s of processors (Tufo and Fischer, 2001). The overlapping Schwarz method has provided a significant reduction in work over previous multilevel solvers (Fischer, 1997). In terms of parallel performance, Nek5000 achieved 376 GFLOPS on ASCI-Red (TFLOP machine based on 8186 Pentium-3 processors), an effort, that was recognized with the Gordon Bell Prize (Tufo and Fischer, 1999).

The primary objective of the on-going research effort is to conduct direct numerical simulations (DNS) and large eddy simulations (LES) bottom gravity currents in stratified environment and over complex topography ( Özgökmen et al., 2004a,b; 2005; Figure 14) These results have been subsequently utilized to develop parameterizations of gravity current mixing in HYCOM (Chang et al., 2005; Xu et al., 2005).

### Data Assimilation in HYCOM

*A. Mariano, T. M. Chin, and A. Srinivasan*

The primary computational research areas are:

- (i) Implementation of the Reduced-Order Information Filter (ROIF) for assimilation of ocean data into high resolution simulations of coastal and ocean circulation by the HYbrid Coordinate Ocean Model (HYCOM).
- (ii) Development of particle filters for both forward and inverse prediction problems in both the Eulerian and Lagrangian framework.

The ROIF is an approximate extended Kalman filter in which the large covariance matrix is implemented as a sparsely banded *information matrix*, which is essentially the inverse of the covariance matrix. Probabilistically, the truncated information matrix is a Gaussian Markov Random Field (MRF) approximation of the Kalman covariances. An advantage of this assimilation scheme above seems to be in the MRF representation of the horizontal covariance, as the small-scale features can be reconstructed better than the more traditional/common Kalman filter approximation schemes (e.g., using a truncated EOF expansion or a small ensemble of Monte Carlo simulations). This advantage of the ROIF method seems to originate from the large number of the variability/error modes that MRF can represent efficiently. We have demonstrated the reliability of the ROIF for the assimilation of altimetric sea surface height

Figure 14: (Left panel) Distribution of salinity anomaly in a 3D nonhydrostatic numerical simulation for a flow over complex bottom topography. Details can be found in Özgökmen et al. (2004b) and a sample animation is available from: http://www.rsmas.miami.edu/personal/tamay/3D/bp14.gif. (Right panel) Snapshot of salinity anomaly for a flow reaching neutral buoyancy when the ambient fluid is stratified by stable linear temperature distribution ( Özgökmen et al., 2005). A sample animation is available from: http://www.rsmas.miami.edu/personal/tamay/3D/ts21-S.gif. These computations have been carried out with 4 million grid points.

Figure 15: Snapshots of Sea Surface Height from the truth run and the twin experiments using ROIF. The satellite altimeter tracks are shown on the top left. The ROIF assimilation reproduces some of the meso-scale features in the truth runet al., 1999; 2002) and for drifters ( Özgökmen et al, 2003). In comparison to other available assimilation technology, ROIF appears advantageous in accurate and timely reconstruction of meso-scale features that is relevant to operational requirements for high resolution estimates of the velocity fields in coastal and ocean environments.