VGGT
https://github.com/facebookresearch/vggt
While drafting my Repo Roundup post this week, I was intrigued by the VGGT project that came out of Facebook Research. Rather than review a few projects, I wanted to spend some time diving deeper into this one to understand it better. It is very much still in research project mode, but as a relative layperson in this area, I think I can see where it's headed from a practical use perspective.
What is it?
This project is demo code from an award-winning paper titled "VGGT: Visual Geometry Grounded Transformer." It essentially takes one or many images (or video) of a subject and reconstructs it as a 3D scene. Behind the scenes, it uses a neural network to predict cameras, point maps, depth maps, etc., as opposed to more simplistic solutions that more or less attempt to stitch scenes together or actual 3d scanning hardware. It is capable of doing these calculations in less than a second, so very close to real-time.
Why is this important?
Previously, if you wanted to create a 3D model of something, you would need to use a 3D scanner (like a Leica BLK360) or something like the LiDAR scanner on an iPhone. While this method works well and can produce excellent results, the high-end hardware is expensive and the process is time-consuming. While there is still work to be done with VGGT, it is just a matter of time until we will be able to generate real 3D models on consumer-grade hardware using just 2D input sources in real-time!
Practical Uses
Anywhere 3D models are needed, this could be applied. I would wager we'll see this or similar technology introduced on our mobile phones within 12-24 months. Other practical applications of this type of technology could be found in:
- 3D reconstruction and mapping
- Computer vision tasks
- AR/VR from 2D images
- Autonomous systems
- Content creation & media
- Photogrammetry and surveying
Demo
I tested it using just a single photo I took of a LEGO figurine that was near my desk. While the 3D model still has a number of artifacts, I think with either additional images or some post-processing, it would generate a relatively good 3D model. It outputs the results as a glb file that I imported into the Threejs sandbox.
Input Image:
Output Model: (a glb file I loaded in the threejs sandbox)
Nicholas Maloney is a Co-Founder of The Gnar Company, where he leverages over two decades of software industry experience to transform complex ideas into foundational digital products. He specializes in building scalable software solutions, implementing AI-driven applications, and leading high-performing development teams. A veteran engineer and Certified Scrum Master, Nick is dedicated to creating elegant, impactful solutions that solve gnarly problems and drive business growth.
Before co-founding The Gnar Company in 2016, Nick served as Lead Engineer at MeYou Health and was a Senior Software Engineer at Terrible Labs, where he built digital products for a range of clients from startups to large enterprises. His career also includes technical roles at Massachusetts General Hospital's MIND Informatics and a four-year tenure as a Web Architect at Bentley University. Nick holds a Bachelor of Science in Computer Information Systems from Bentley University.
