#5: Dynamic Gaussian Splats - Taking Gaussian Splats research deeper
Faster, greener and virtual real-time - revolutionising visual experience one Splat at a time
Research at the CoSTAR National Lab into the challenges and possibilities of Gaussian Splatting continues. We are particularly interested in a number of foundational R&D challenges:
Dynamic Gaussian Splats
Traditional 3D Gaussian splatting methods excel at reconstructing static scenes, capturing intricate details and providing stunning visualisations. However, when applied to fully dynamic scenes—those with moving objects or changing scenes—these models fall short. Each frame must be trained independently, requiring painstaking manual animation to create a coherent sequence. This workflow is both time-consuming and inefficient.
Dynamic Gaussian splatting aims to address this problem by enabling the reconstruction of time-variant scenes directly from video inputs. Instead of training for each frame individually, a dynamic model could ingest an entire video sequence, learning to represent motion and temporal changes seamlessly. The goal is to render a fully animated video from a chosen camera view without manual intervention.
Such advancements could revolutionise industries reliant on 3D rendering, making it faster and more accessible to produce high-quality animations, simulations, and interactive experiences.
Squashing the Splat: Compression in Gaussian Splatting
As Gaussian Splatting methods grow in prominence, a critical bottleneck has emerged: the massive storage and memory demands associated with these methods. By addressing the dual challenges of maintaining fidelity and reducing storage demands, these advancements can make Gaussian Splatting far more practical for real-world applications.
Drawing on previous research at the University of Surrey, we applied a compression model to scenes from Ulysses the Snail. The results demonstrate significant reduction in file size and GPU load whilst only requiring a minimal trade-off in visual quality (example below).
Moving forward, work by the CoSTAR National Lab partners will extend these compression strategies. Our focus will be on preserving the fidelity of subjects while ensuring efficient memory use.
Green Splats
As the demand for 3D rendering grows across industries the environmental impact of rendering techniques comes into sharp focus. Traditional mesh-based rendering techniques have benefited from decades of optimisation. Our research will focus on reducing the carbon footprint of Gaussian Splats - critical for mitigating the environmental impact of this technology, particularly for complex, high-volume scenes.
Getting Real: Using Gaussian Splats in real time engines
Our research will focus on a number of key challenges for the wider adoption of Gaussian Splats in real-time engines, including:
Rendering speeds. Splats are traditionally rendered using rasterization which is slow. Emerging ray tracing technologies offer significant promise.
3DGS to Mesh. Converting splats to mesh forms allows the full range of existing tools and techniques that support mesh editing and manipulation to be harnessed. While Gaussians are an explicit representation, they do not have a defined surface or any sense of interconnectedness. This makes constructing a mesh from a Gaussian model a challenging task.
Relighting. To make a 3D world believable, it is critical that lighting is consistent among all objects in the scene. For Gaussian Splats, this requires approximating and removing the illumination present at the time of capture and identifying the reflectance of the surface to allow relighting under novel lighting conditions.