GCP – Streaming augmented reality with Google Cloud

Every year at CES, people from around the world experience the latest and greatest that consumer tech has to offer. In 2021, CES will be in an all-digital format for the first time. 

So how can a virtual show like CES create immersive experiences for attendees tuning in remotely? That’s an interesting challenge for the cloud, and of course, every challenge presents an opportunity. 

Google Cloud and 5G help enterprises deliver new experiences

Earlier in 2020, we announced our enterprise telecommunications strategy to deliver workloads to the network edge on Google Cloud, and during our SearchOn event last October, we announced how cloud streaming technology can power augmented reality (AR) in consumer search results. 

Now, we’re merging the best of both worlds: Technology built for consumer search can take advantage of our enterprise edge capabilities. In light of the COVID-19 pandemic, this past year accelerated our support for enhanced consumer experiences across the board in novel ways. For example, we attempted to address questions such as how potential buyers can make a purchasing decision when they can’t see the product up close. This question becomes even more critical when considering a large purchase such as a new car. 

That’s exactly what Fiat Chrysler Automobiles (FCA) and Google Cloud are working together to solve. As part of FCA’s Virtual Showroom CES event, you can experience the new innovative 2021 Jeep Wrangler 4xe by scanning a QR code with your phone. You can then see an Augmented Reality (AR) model of the Wrangler right in front of you—conveniently in your own driveway or in any open space. Check out what the car looks like from any angle, in different colors, and even step inside to see the interior with incredible details.

“As we continue our journey towards becoming a customer-centric mobility company, FCA is adopting emerging technologies that enable us to accelerate and deliver at the speed of our customers’ expectations,” said Mamatha Chamarthi, Chief Information Officer, FCA – North America and Asia Pacific. “Through our collaborative partnership with Google, we are able to expand our efforts to provide an immersive customer experience.” 

Harnessing the power of edge with 5G

In order to create a mixed-reality experience with a 3D car model, computer-aided design (CAD)-based data sources that represent a 3D vehicle with highly detailed geometry, depth, texture, and lighting were used. High-fidelity models, such as cars with full interiors, often mean large files (GBs in size). Traditionally, depending on your connection, this can result in long waiting times as assets are downloaded onto your phone. In addition, while mobile phones are more powerful than the Apollo Guidance Computer, they are no match for the power we have in the cloud. We still want to bring these high end experiences to everyone, regardless of their respective device or geographical location. 

We solve this problem by rendering the model in Google Cloud, then streaming it to the devices. 

Specifically, the Cloud AR tech uses a combination of edge computing and AR technology to offload the computing power needed to display large 3D files, rendered by Unreal Engine, and stream them down to AR-enabled devices using Google’s Scene Viewer. Using powerful rendering servers with gaming console grade GPUs, memory, and processors located geographically near the user, we’re able to deliver a powerful but low friction, low latency experience. This rendering hardware allows us to load models with tens of millions of triangles and textures up to 4k, allowing the content we serve to be orders of magnitude larger than what’s served on mobile devices (i.e., on-device rendered assets). Doing so leverages high-speed 5G connectivity and streams directly from Google Cloud’s distributed edge, delivering a rich, photorealist immersive experience. Customers like FCA benefit from Google’s years of investment and expertise in streaming technology (have you tried playing Cyberpunk2077 on Stadia yet?). With the expansion of 5G networks, not only will streaming enable the experience for anyone anywhere, but it will also cut the wait time of downloading large assets required for detailed AR/VR experiences, ultimately providing instant gratification.

Applications and experiences are at the core of a winning edge proposition

We’re working to make these capabilities available to all enterprise customers to enable innovative use cases such as leveraging AR to help design teams collaborate, technicians perform machine diagnostics, creating next-generation, live video experiences for sports events, enabling new customer experiences across many industries, and supporting our customers in their digital transformation. Stay tuned!

To see the Jeep Wrangler 4xe come to life whether you’re attending CES or not, scan the QR code below, or check out the FCA CES website. Depending on your OS, device, and network strength, you will see either a photorealistic, cloud-streamed AR model or an on-device 3D car model, both of which can then be placed in your physical environment.