2 Jun 2020

PUBLICATION NOTICE: Parallel File I/O for Geometric Models: Formats and Methods

Abstract: Processing large amounts of data in a High-Performance Computing (HPC) environment can be throttled quickly without an efficient method for utilizing disk I/O. The Geometry Engine component of the Virtual Environment for Sensor Performance Assessment (VESPA) uses MPI-IO to load the geometric data in parallel and avoid creating a bottleneck on disk I/O interactions. This parallel I/O method requires formatting the data into specific binary file formats so each MPI process of the parallel program can determine where to read or write data without colliding with other MPI processes. Addressing the collision problem resulted in the development of two binary file formats, the Mesh Binary file (.mb) and the Scene Chunk Pack file (.scp). The Mesh Binary file contains the essential data required to recreate the landscape and vegetation geometry used by the Geometry Engine. The Scene Chunk Pack file is used to write the partitioned geometry to disk, so the ray casting engine can reload the distributed geometry without repeating the partitioning process. Both of these files together support reading and writing for the partitioning phase and the ray casting phase of the Geometry Engine. This report discusses these formats in detail and outlines how the Geometry Engine reads and writes these files in parallel on HPC.