Passthrough camera access is causing quite a stir in the XR world these days. We know where Meta, Apple, and Pico stand, but the big question that remains is: what will Google do with Android XR? After having a chat with folks at Google, I can reveal that they plan to offer something much like what you’d find on a smartphone. Curious for more details? Stick around!
### The Camera Access Dilemma
Let me fill you in if you’re scratching your head. Our latest standalone VR headsets are more like MR headsets, displaying an RGB passthrough vision of our surroundings via front cameras. This technology fuels delightful MR applications such as Cubism, Starship Home, and Pencil, to name a few.
The system taps into camera-captured frames for the passthrough view. As developers, we’d love to get our hands on these frames to apply some AI and computer vision algorithms, thereby amplifying the user’s reality in impressive ways. I’ve mentioned before how critical camera access is for unleashing the true potential of mixed reality. With it, apps can fully grasp the context of the surrounding environment. For instance, I once used a clever workaround on Quest to develop an AI+MR app that aids in home interior design. Without camera access, that experiment wouldn’t have gotten off the ground.
Here’s where things get sticky: privacy concerns. Imagine a rogue developer gaining camera access; they could secretly snapshot the user’s environment, run images through AI detection, and scrape sensitive personal info, like national IDs or bank cards lying on the table, all unbeknownst to the user. There’s also the risk of capturing personal images for dubious purposes.
It’s a tightrope walk between protecting user privacy and unlocking the boundless opportunities of mixed reality.
### The Behavior of the XR Companies
In the beginning, unrestricted camera access was no big deal. If you’ve been with me from the start, you might recall the experiments we did at NTW around 2019 with the Vive Focus, testing things like diminished reality and Aruco marker tracking.
But as mixed reality gained traction, companies took a caution-first approach, halting camera frame access due to privacy worries. Meta, Pico, HTC, and Apple all enforced these restrictions.
This stance solidified until the XR developers’ community began advocating for camera access. Notable voices like Cix Liv, Michael Gschwandtner, and myself started pushing for a transparent camera access feature—one transparent to users and requiring their consent. After all, your phone can grant camera access with a simple permission request, so why not XR devices?
Thanks to this push, some XR companies, like Meta, have shown signs of relenting, with Meta announcing a Passthrough API to roll out soon. But what about Google and Android XR?
### Android XR to Treat the Headset Like a Phone
Android powers most of the world’s smartphones, and if you’re developing an Android app, you can request camera stream access with user permission. Once given, you specify the camera ID (like 0 for the back camera) to capture frames for your app.
Google aims to make Android XR highly compatible with existing Android apps, offering a similar setup on its new OS. After some extensive emails with Google, I confirmed that the XR platform will allow developers to access camera frames with user permission, leveraging familiar systems like Camera2 and CameraX for world-facing and selfie cameras, which show either the environment or the user’s avatar, respectively.
Developers can thus use their go-to tools, like CameraX, to manage camera streams on XR devices, paving the way for functions like ML analysis. However, for selfie cameras, the stream is an avatar projection, similar to Apple’s Vision Pro. This design replicates how smartphones access what they call ‘front’ and ‘back’ cameras and aligns XR behavior accordingly.
### A Little Caveat About Android XR
Keep in mind, Android XR is still in its preview stage, and no headsets operating on it have officially launched. This means things could shift before the final release. While unlikely, it’s smart to be aware that changes could occur.
### The Opening Up of Camera Access
As Google and Meta open their gates to camera access, it’s likely others will follow, setting the stage for 2025 to be a landmark year in unlocking mixed reality potential. I can hardly wait to see the ingenious creations this will inspire in the developer community!
(Header image courtesy of Samsung)
Disclaimer: This blog contains adverts and affiliate links to keep going. When you click an affiliate link, I might earn a small commission, which would make me quite happy. You can read my full disclosure if you’re interested.
