Passthrough camera access is currently sparking a lot of conversation within the XR community. While we know where companies like Meta, Apple, and Pico stand, speculation is rife about Google’s intentions with Android XR. After chatting directly with Google, it seems they’re leaning towards a solution similar to that on their phones. Keep reading to find out more about their approach!
Navigating the Camera Access Challenge
If you’re not sure what all the fuss is about, let’s take a quick detour. Recent standalone VR headsets are now more like MR headsets, offering users an RGB passthrough view of their surroundings via front-facing cameras. This capability has made cool mixed reality applications like Cubism, Starship Home, and Pencil possible.
The operating system displays this passthrough view by processing the frames from these cameras. Developers, like myself, are eager to access those frames, too, to incorporate AI and computer vision into apps, enhancing user experience by making the applications context-aware. For instance, using a workaround on Quest, I managed to prototype an AI+MR app to help with home interior design, which was a feat only possible with camera access.
However, this great technology comes with privacy concerns. There is a risk that an unethical developer could use unrestricted camera access to invasively collect private information, such as details from documents or credit cards lying around, or even inappropriate images of people.
It’s a tricky balancing act: ensuring user privacy while unlocking mixed reality’s potential.
How XR Companies Are Tackling This
In the beginning, there was carefree camera access for developers. If you’ve been following my journey, you might remember the experiments with camera textures we did on the Vive Focus back in 2019 — things like diminished reality and Aruco marker tracking.
Once mixed reality gained traction, companies grew cautious and began restricting access due to privacy concerns. Meta, Pico, HTC, and eventually Apple all stopped allowing developers to access camera frames.
That was the norm until the XR developer community started pushing back, advocating for controlled camera access. Some of us, like Cix Liv, Michael Gschwandtner, and myself, were quite vocal, arguing that developers should have clear and transparent access to cameras similar to the straightforward permission model on smartphones. As a result, companies started opening up, with Meta pledging a “Passthrough API” debuting this year. But what is Google’s game plan with Android XR?
Android XR’s Camera Approach
Android powers most phones globally. For app developers, requesting camera access is as simple as securing user permission. Once granted, developers specify which camera they need and can start capturing frames.
Google intends for Android XR to support Android apps as seamlessly as possible. After corresponding with a Google representative, she confirmed this approach:
Just like any Android app, XR developers can use camera frames with user permission. The developers need to know that they can request access to the main world-facing camera or the selfie camera through standard Android APIs.
Android XR aims to mirror the setup on smartphones. When accessing the “rear camera,” developers get a view of the world in front of the user; accessing the “selfie camera” gives an image stream of the user’s avatar, constructed from tracking data provided by inward-facing cameras.
Staying in Tune with Android’s Standards
Google is keen on ensuring Android apps operate effortlessly on Android XR devices. This uniform approach covers apps requiring camera use. It’s a savvy move that maintains consistency between phone and headset permissions, a concept I wholeheartedly support.
You might be curious about direct access to raw camera data. Unfortunately, the answer isn’t positive at the moment:
Currently, Google isn’t offering ways for applications to access non-standard sensor data.
This stance suggests non-standard camera feeds won’t be available just yet, but there’s hope for future access, particularly for enterprise users.
For those developing with Unity, the challenge remains that Android Camera2 and CameraX are native classes, which Unity doesn’t natively support. However, developers might use the WebcamTexture class or create a JNI bridge to leverage CameraX functionalities within Unity.
A Pre-Release Caution for Android XR
Keep in mind, Android XR is in its preview phase, and no official headset release has occurred. All of these details might change prior to the official launch, so stay aware of potential updates.
Broadening Camera Access: The Road Ahead
With companies like Google and Meta starting to open camera access, it’s likely other XR companies will soon follow suit. By 2025, we could be on the brink of new mixed reality innovations. I’m eager to see the creative projects the developer community will unveil!
(Header image attributed to Samsung)
(Note: This blog contains advertisements and affiliate links. Click on an affiliate link to support this blog, as I earn a small commission on purchases. Full disclosure can be found here.)
Don’t forget to share this post with fellow innovators!
