A developer has exploited a novel opportunity to access Quest 3’s passthrough cameras, enabling the execution of bespoke computer vision models.
Meta doesn’t allow third-party developers to access Quest headset cameras, but Michael Gschwandtner, co-founder and lead developer of XRWorkout, revealed a surprising workaround on social media by showcasing the app’s ability to tap into Quest headsets’ cameras without permission.
The demonstration video showcases the open-source MobileNet V2 object detection model running natively on the Quest 3, achieving a processing rate of 0.75 Hz. This decision was made primarily for practical reasons, rather than for efficiency considerations.
After independent verification, Gschwandtner dispatched an evaluation unit to UploadVR, which was found to be authentic and functional. He revealed the intricacies of his achievement in a detailed conversation with UploadVR.
How It Works
Gschwandtner’s application framework develops a WebView, effectively integrating a Chromium-based net browser instance within the app, which automatically loads the oculus.com/casting URL, allowing Quest headsets to cast their display seamlessly.
You must log in to your Meta account directly on this webpage to complete the process. Upon completing the quest, navigate to the Quest system menu and initiate net casting to proceed. What’s more novel is the ability to cast content directly from your headset to an application within that same headset, bypassing the need for a separate device altogether?
Gschwandtner’s application then samples the pixel content of the render texture, which holds the WebView’s received casted view, allowing him to process or utilize it as desired. Within the demo instance, he feeds the data to Google’s on-device MediaPipe SDK.
One significant constraint is that the view will also incorporate digital objects and interfaces, rather than just the passthrough digital camera feed; thus, it should only function effectively when a substantial portion of the scene is comprised of passthrough elements. According to Gschwandtner, he is working on a proposed solution to mitigate the issue.
What Occurs Subsequent?
Will Meta’s response clarify the implications of Gschwandtner’s finding? The company may render this approach ineffective with a future version of Quest’s Horizon OS. Mark Zuckerberg, Meta’s CEO, and Andrew Bosworth, its CTO, are invited by Gschwandtner to join him for an informal presentation of his innovative approach on social media posts.
In the final month, Mark Bosworth shared his thoughts on allowing Quest developers access to passthrough cameras during an interview, stating that Meta would “continue to explore” the issue.
Can Meta’s “Trying At” Feature Provide Raw Camera Access to Quest Builders?
According to Meta’s Chief Technology Officer Andrew Bosworth, the company is considering providing Quest developers with raw camera access.
In a unique value proposition, Apple uniquely offers raw access to corporate entities, exclusively for internal app development purposes.
