Background
The fusion of low-cost unmanned aerial systems (UAS) with advanced photogrammetric techniques has revolutionized 3D terrain reconstruction, enabling the automated creation of detailed models. Concurrently, the advent of 3D Gaussian Splatting (3DGS) has introduced a paradigm shift in 3D data representation, offering visually realistic renditions distinct from traditional polygon-based models. Our research builds upon this foundation, aiming to integrate Gaussian Splatting into interactive simulations for immersive virtual environments. We address challenges such as collision detection by adopting a hybrid approach, combining Gaussian Splatting with photogrammetry-derived meshes. Through comprehensive experimentation covering varying terrain sizes and Gaussian densities, we evaluate scalability, performance, and limitations. Our findings contribute to advancing the use of advanced computer graphics techniques for enhanced 3D terrain visualization and simulation.
Objectives
ICT’s research in 3D Gaussian Splatting aims to demonstrate its significant potential for both visualization and simulation, offering promising avenues for rendering detailed urban environments with high fidelity. The key factor contributing to the impressive performance boost in Gaussian splatting compared to neural radiance field techniques is the visibility-aware sorting method. This method allows for direct updating of the frame buffer with alpha blended values, eliminating the need for per-pixel alpha blending through depth testing. As a result, rendering performance is significantly improved, allowing for real-time rendering at over a hundred frames per second. The benefits of this approach extend beyond rendering, as it also speeds up training times by enabling a fast backpropagation step through reverse sorted traversal of the Gaussians. Additionally, increasing the number of Gaussians that receive gradients has little to no impact on performance, making it an efficient choice for training. However, when addressing large-scale areas, the 3DGS model reveals the necessity for further research into optimized data formats, large scale reconstructions, and visualization techniques.
Results
In order to facilitate the training of expansive scenes, we have implemented a tiling methodology that partitions the scenes into smaller sections and trains them using the corresponding subset of images. The outcome is then merged into a single point cloud or divided into multiple manageable point clouds that can be streamed seamlessly. The LumaAI plugin boasts a competitive Gaussian splatting renderer for Unreal Engine. However, it has a two million particle count restriction of Niagara that limits its potential. To address this, we introduced a partitioning technique that divides Gaussian point clouds into smaller point clouds. These can be instantiated as individual Niagara systems and streamed on demand, thus optimizing performance and resource usage. Taking it a step further, we integrated simulation capabilities into the renderer provided by the LumaAI plugin. To communicate simulation data with the Niagara particle system, we introduced Gaussian collider actors, along with a scene-specific mapping of renderers to colliders. This minimizes the number of GPU calls to push data into the VRAM. With these techniques, we achieved a first-person interactive Gaussian splatting environment at playable frame rates with high fidelity.
Next Steps
Our next steps involve developing more efficient streaming mechanisms, integrating with Unreal Engine’s chaos destruction engine, and implementing Gaussian-based collisions without the need for photogrammetric meshes. These advancements will help increase performance and simplify the workflow for creating large-scale 3D terrain reconstruction using 3D Gaussian splatting for visualization and simulation.
Published academic research papers are available here. For more information Contact Us