The JPEG of 3D: How Gaussian Splatting and AI Agents are Revolutionising Interactive Digital Twins

In a recent Gemini Call presentation, Michael Setton from Twin4Green unveiled how cutting-edge Gaussian splatting technology combined with AI agents is transforming digital twins from static models into photorealistic, interactive environments accessible through any web browser.

Beyond the Fragmented Landscape of Digital Twins

Michael opened his presentation with a challenge familiar to anyone working in the digital twin space: the term means vastly different things to different people. “If you ask somebody to define what they wanted out of a digital twin, you would ask 30 people and you would get 30 different responses,” he observed, drawing from his experience in IoT at Vodafone.

This fragmentation represents a fundamental challenge in digital twin adoption. From CAD/CAM simulations to building information models, from medical scans to IoT sensor networks, the digital twin landscape has evolved into a complex ecosystem of disparate technologies and expectations. Twin4Green’s solution, aims to bridge these divides through a revolutionary approach combining gaming technology, photorealistic capture, and AI-driven interaction.

The Gaming Engine Revolution

Rather than building from scratch, Twin4Green recognised that the gaming industry had already solved many of the technical challenges facing digital twins. “I’ve always been jealous of gamers because they always had the most powerful machines,” Setton admitted, highlighting how NVIDIA’s success was built on gaming’s insatiable demand for GPU power.

This isn’t just about borrowing computing power—it’s about leveraging decades of development in real-time rendering, physics simulation, and user interaction. Major projects are already demonstrating this approach’s potential:

• The European ILIAD project created diverse applications for a digital twin of the ocean using Unreal Engine
• SAS Institute built a factory digital twin for Georgia Pacific, simulating autonomous guided vehicles and optimising their collaboration with human workers
• Their tagline captures the essence: “A digital twin makes reality work better”

Enter Gaussian Splatting: The Breakthrough Technology

At the heart of Twin4Green’s innovation lies Gaussian splatting—a technology Michael describes as “the JPEG but for three dimensions.” Invented just two years ago at INRIA in France, this technique represents a paradigm shift in how we capture and render 3D environments.

Understanding the Technology

Gaussian splatting works through two key concepts:

1. Gaussians: Instead of defining every single point in 3D space, the technology uses rugby ball-shaped mathematical functions that describe how light and colour are distributed. Each Gaussian contains:
   • Position (where it exists in 3D space)
   • Shape (how it’s stretched—from sphere to elongated ellipsoid)
   • Colour (RGB values)
   • Opacity (contrast and luminosity)
2. Splatting: The projection of these 3D Gaussians onto your 2D screen, creating photorealistic views from any angle

This approach is remarkably efficient. Rather than storing millions of individual points, Gaussian splatting creates compact representations that can be rendered in real-time within web browsers—no special hardware required beyond standard devices.

Practical Implementation

The capture process has been democratised to the point where anyone with an iPad or Insta 360 camera can create professional-quality digital twins:

• Capture: Take approximately 2,000 images for a high-resolution room scan, using three passes at different heights (ceiling level, middle, floor level)
• Processing: Upload to cloud-based GPU processing (typically a few hours)
• Result: A photorealistic, navigable 3D environment accessible through any web browser

During the demonstration, Michael showed a Gaussian splat of Digital Catapult’s London facility, where users could seamlessly navigate the space and interact with IoT devices placed within the environment.

The Convergence: IoT Meets Photorealism

What sets them apart is its bidirectional capability—not just visualising sensor data but enabling control of physical systems through the digital twin. As Michael explained, “You want to not just see the data from these sensors… you also want to be able to act upon it.”

The platform demonstrates this through practical examples:

• Click on a water leak sensor in the 3D environment to view its status
• Identify an issue and immediately trigger a shut-off valve
• View both real-time and historical data within spatial context
• Drag and drop new sensors into the environment for instant integration

This spatial intelligence transforms how organisations interact with their IoT infrastructure, moving from abstract dashboards to intuitive, location-based interfaces that non-technical users can navigate effortlessly.

AI Agents: The Intelligence Layer

Twin4Green employs AI agents across three critical areas:

1. Device Integration: Automating the complex process of connecting diverse IoT protocols (Wi-Fi, LTE, LoRa) into a unified system
2. User Experience: Creating intuitive interfaces that adapt to user needs and technical capabilities
3. Time to Value: Accelerating deployment from months to days through intelligent automation

These agents handle the complexity behind the scenes, allowing users to focus on solving business problems rather than wrestling with technical integration.

Industry Applications and Future Potential

The technology’s applications span multiple sectors, with particular value in:

Industry 4.0

• Carbon footprint reduction through spatial energy monitoring
• Supply chain optimisation with visual logistics tracking
• Operational efficiency improvements via intuitive facility management

Outdoor and Large-Scale Environments

• Smart city deployments where spatial context is crucial
• Maritime navigation and port management
• Construction sites requiring real-time coordination
• Defence applications needing accurate spatial intelligence

Meta’s recent announcement of Horizon Hyperscape validates this approach, bringing Gaussian splatting to consumer applications and accelerating standardisation through the glTF 3D standard.

Scaling for the Future

Twin4Green’s roadmap addresses critical challenges for widespread adoption:

Protocol Integration: Supporting diverse communication standards without requiring standardisation Security: Addressing heightened concerns following incidents like the JLR cyberattack Accessibility: Ensuring the technology works on standard mobile phones, not just high-end VR equipment Partnerships: Actively seeking collaborators to accelerate 3D digital twin adoption

Key Takeaways for Digital Twin Practitioners

Michael Setton’s presentation offers several crucial insights for organisations considering next-generation digital twins:

1. Leverage Existing Technology: Don’t reinvent the wheel—gaming engines have solved many fundamental challenges
2. Photorealism Matters: Users engage more effectively with environments that look and feel real
3. Simplicity Drives Adoption: If non-technical users can’t navigate your digital twin, its value remains locked away
4. Think Bidirectionally: True digital twins don’t just visualise—they enable action
5. Start with Capture: Modern tools make creating photorealistic environments surprisingly accessible

The Path Forward

As digital twin technology evolves, the convergence of photorealistic rendering, IoT integration, and AI intelligence represents more than incremental improvement—it’s a fundamental shift in how we interact with digital representations of our physical world.

Twin4Green’s approach demonstrates that the future of digital twins isn’t about choosing between different technologies or standards, but rather integrating the best of what’s available—from gaming engines to AI agents, from IoT sensors to photorealistic capture—into cohesive platforms that solve real business problems.

For organisations looking to move beyond fragmented digital twin implementations, the message is clear: the technology now exists to create truly interactive, photorealistic digital twins that work in any browser, integrate with existing systems, and provide immediate value to users regardless of their technical expertise.

The Digital Twin Hub’s weekly Gemini Calls continue every Tuesday at 10:30am BST, showcasing breakthrough technologies shaping the future of digital twins. All sessions are recorded and available to community members.

Ready to explore how photorealistic, interactive digital twins can transform your operations? Join the Digital Twin Hub community to connect with innovators pioneering these next-generation solutions.