Massively-Parallel Computational Fluid Dynamics

Started: 1st October 2013
Research Team: James Hawkes
Investigators: Simon Cox, Stephen Turnock, Alexander Phillips

Examples of CFD applications. Clockwise from top-left: aircraft simulation; internal combustion engine; pedestrian wind environment; flow through a cerebral aneurysm; severe weather prediction; nuclear reactor core coolant pump. See references.

Computational Fluid Dynamics (CFD) is a numerical method for modelling fluid flows and heat transfer - and is used in many industries. It can be used to model dynamics around aircraft, ships and land vehicles; and also has uses in engine design, architecture, weather forecasting, medicine, computer-generated imagery (CGI) and much more. To harness the full power of CFD, it is necessary to utilise the full power of modern supercomputers. This project aims to improve the scalabilty of existing CFD codes so that more complex problems can be tackled efficiently.

This project, from the Fluid Structure Interactions group, is sponsored by MARIN - a maritime research institute from the Netherlands. Their existing CFD code, ReFRESCO, focuses on tackling fluid flows around ships and offshore structures. ReFRESCO already has many features that make it a specialist solver in this regard - such as cavitation models, sliding interfaces (for modelling propellers), and automatic mesh refinement. The computational cost of incorporating all these models into a single high-accuracy simulation is prohibitively high, and the goal of this project is to improve the efficiency of the software to enable such complex simulations.

Large CFD simulations currently encounter a communication bottleneck when the problem is split across many nodes in a large supercomputer. This quickly limits the amount of resources that can be used efficiently and makes it impossible to use brute computing power to arrive at an accurate solution in a timely manner.

In this project, a scalability study will be performed, which compares execution time versus number of cores, from 1 to 1024, on Iridis4. Profiling tools will be used to look inside ReFRESCO, assessing which parts of the code are the least efficient in a massively parallel environment. There are three particularly expensive parts of a CFD code, they are:

the conversion of the physical geometry (discretized mesh) into a set of simultaneous equations;
the iterative solution of those equations;
and the calculation of derivatives, which are required to create the set of equations.

It is predicted that the iterative solver will be the least scalable, as it has the largest quantity of parallel communication. Therefore, the remainder (and largest part) of the project will likely focus on developing a new solver for CFD codes which is aimed at high scalabilty.

The intention is to create a solver which mixes a tried-and-tested Algebraic Multigrid Method with a more unusual Chaotic/Asynchronous Iterative Method. This aims to take the advantages of multigrid methods, and compensate the disadvantages with chaotic communication. The Chaotic Iterative Method should also implicitly make the code heterogeneous - capable of using the Xeon Phi accelerators on Iridis4 and other accelerated clusters such as Emerald (GPU accelerators). See "Asynchronous parallel solvers for linear systems arising in computational engineering" (Jimack and Walkley, 2011) for more information.

ReFRESCO uses the scientific suite PETSc to solve the system of equations, thus this is where the development from this project will be applied. There are a wide variety of CFD programs, in many industries, that also use PETSc (or could) so the developments from this project could have an impact on CFD in all industries - not just maritime.

Image sources: cfd4aircraft.com, ec.europa.eu, flometrics.com, htf.fl, hko.gov.uk, aij.or.jp

Computational Modelling Group

Massively-Parallel Computational Fluid Dynamics

Categories