In early 2024, the CoSTAR National Lab prototyping team embarked on a research project into the field of Gaussian Splatting: pushing a 3D technology into the fourth dimension. This is the story that starts with Nine Inch Nails and ends with an animated odyssey of a Giant African Snail called, Ulysses and that demonstrates the promise and the potential of dynamic Splats.
The Prototyping Team
The CoSTAR National Lab prototyping team is a small multidisciplinary team of researchers, producers, creative technologists, and developers from across the CoSTAR National Lab’s core partners: NFTS, University of Surrey, Abertay University and Royal Holloway, University of London.
Lead by Miles Bernie (Co-Head of Innovation), the team for this project were: Neil Smith (Senior Technician), Destiny Lawrence (Technician Advanced Production), Johnny Johnson (Senior Creative Technologist), Cristen Caine (R&D Producer), Dr Violeta Menéndez González (Senior Research Software Engineer), Umar Farooq (Research Fellow), Jamie Buttenshaw (Research Fellow), Naman Merchant (Lecturer in Game Technology and Mathematics), Alessandro Lollo, Gemma Campbell (Lead Creative Producer), Talia Finlayson (Creative Technologist) and Laura Bell (Creative Technologist).
Our ultimate goal is to connect deep research themes with practical industry use cases. We use an Agile sprint-based innovation methodology: ideate, build, test, review, course correct. Repeat!
What is Gaussian Splatting?
Gaussian Splatting is an emergent 3D capture technology, similar to photogrammetry, but instead of meshes, captures points of light and plots them as mathematical models “that can vary in width, height, and orientation, offering a flexible way to model the density and appearance of different parts of the scene… with properties like position, size, orientation, and colour.”1
In short, Splats make it easy to capture an object or a scene, using a simple camera or smartphone, and to render a faithful representation of that capture in 3 dimensions using an ordinary computer with a good quality GPU (or, via an app like Scaniverse, on a smartphone).
We have seen growing excitement about Gaussian splatting since the first research paper on the topic was published in 2023, with some believing that it “promises a time not too far off where we will no longer be using photography and videos as the dominant imaging medium and be able to routinely and with minimal effort be able to document our lives, business, and society in a hyper realistic way, similar to how we experience everyday life." 2
Whether this is an industry-changing development or not we’ll have to wait and see, but we’re sure we are going to see a lot more creative and technical investigation of Splatting in 2025 and beyond.
Where are we going with this?
We set out to better understand the technology in its current form and its possibilities across time and space: from 3D to 4D Splats.
To establish best practice in capturing Gaussian Splats, with support from the National Film and Television School (NFTS), we undertook a wide range of test captures, from small objects to large scenes, using cameras from smartphones to DSLRs, and analysed the results. Our early research was successful in painting a picture of how Splatting works, how to do it in varying conditions, and how the creative industries might make use it, quickly focusing in on how splats can be changed, ordered, and manipulated over time to create movement, or the illusion of movement. These included:
Stop frame animation: transforming the creative process;
Real-time, immersive environments: tackling challenges lighting and editing splats in game engines;
Digital humans: the use of machine learning/computer vision for more efficient capture and reconstruction.
Virtual Production: Integrating Splats into workflows and testing compute efficiencies
Our initial approach to understanding movement in Gaussian Splats was to think of it as a continuum: at one end is stop-frame animation, where still captures are assembled frame-by-frame to create the illusion of movement, and at the other is full-motion dynamic Gaussian Splat capture, also known as 4DGS. We decided to focus our efforts on three different points along that process:
We are convinced that this prototyping research validates our belief that static and dynamic Gaussian splatting is ready for development and application across the creative industries. It’s a fast-moving field: new software plugins, research papers, and commercial and creative applications were unfolding during our prototyping cycle as fast as we could stay informed.
The opportunities for industry are tangible, however there are significant challenges ahead in terms of processing power, file size and storage, lighting, and workflow.
Upcoming research across CoSTAR National Lab will investigate these challenges, and we are planning to share more insights as this work develops in the following months
You’ll be able to dive deeper into the technical R&D in each of the areas discussed in this post in a series of future updates.