Quest’s extremely anticipated “Passthrough Digital camera API” is now obtainable for all builders to experiment with, although they can not but embrace it in retailer app builds.
The brand new functionality was introduced at Meta Join 2024 in September as coming this 12 months. Now it is right here, as an experimental launch for Quest 3 and Quest 3S. Meaning any developer can experiment with it, and even distribute APKs utilizing it on platforms like SideQuest, however they can not but embrace it in Meta Horizon Retailer apps. Meta has taken this strategy for brand spanking new APIs a number of instances previously, and sometimes makes the characteristic obtainable to be used in retailer apps inside just a few months at most.
Choose builders have had early entry to experiment with the potential for some time now, and Meta will host Niantic, Creature, and Decision to debate it at GDC subsequent week.
What Is Passthrough Digital camera Entry?
Whereas headsets like Quest 3 use cameras to allow you to see the actual world, till now solely the system software program received uncooked entry to those cameras. Third-party builders might use passthrough as a background, positive, however they did not really get entry to it. They as a substitute received higher-level knowledge derived by the system, comparable to hand and physique skeletal coordinates, a 3D mesh of your atmosphere with bounding packing containers for furnishings, and restricted object monitoring capabilities. That meant they could not run their very own laptop imaginative and prescient fashions, which severely restricted the augmentation capabilities of those headsets.
The exception was that on visionOS 2, Apple provides enterprise firms uncooked entry to Imaginative and prescient Professional’s passthrough cameras for private inner apps, however this requires a particular licence from Apple and is restricted to “in a enterprise setting solely”.
For the “Passthrough Digital camera API” to work, you because the consumer must grant the app permission to entry your headset cameras, simply as you’d the microphone. If granted, the app will get entry to the forward-facing shade cameras, together with metadata just like the lens intrinsics and headset pose, which it might probably leverage to run customized laptop imaginative and prescient fashions.
Examples of how apps might use this embrace scanning and monitoring QR codes, detecting a recreation board on a desk so as to add digital characters and objects to it, detecting bodily objects for enterprise information experiences, or integrating the visible AI performance of cloud-hosted giant language fashions (LLMs). Builders are solely restricted by which real-time laptop imaginative and prescient fashions can run on the XR2 Gen 2 chipset performantly, or which cloud-hosted picture fashions they’re keen to pay for.
Meta software program engineer Roberto Coviello’s QuestCameraKit samples.
The passthrough digicam stream is supplied to the app with as much as 1280×960 decision per digicam at 30FPS, with a acknowledged latency of 40-60 milliseconds. Meaning it is not appropriate for monitoring fast-paced objects, comparable to customized controllers, nor for discerning nice options like small textual content.
Technically, at a base degree, there is no such thing as a particular Meta Quest Digital camera Passthrough API, neither is it an extension to OpenXR. Builders do must request a Horizon OS particular Headset Cameras permission, however in any other case Quest’s passthrough digicam entry leverages Android’s current Camera2 API to additionally return the headset pose, obtained with OpenXR, and the Camera2 API is what builders of customized engines, or supply code for Unreal or Godot, use for it. This additionally means the identical code ought to work on Google’s upcoming Android XR platform, set to debut in Samsung’s standalone headset, with solely the permission request being totally different.
For Unity, builders can simply entry the cameras by Unity’s WebCamTexture API, which is how they already entry telephone, pill, and PC cameras and webcams within the engine. A limitation right here, nevertheless, is that Unity’s WebCamTexture API solely helps one digicam at a time, not each.
Walkthrough from Meta software program engineer Roberto Coviello.
builders can discover Quest passthrough digicam entry documentation right here: Unity / Native Android.
Meta has printed 5 official Unity samples on GitHub: CameraViewer, CameraToWorld, BrightnessEstimation, MultiObjectDectection, ShaderSample. Meta software program engineer Roberto Coviello has individually printed QuestCameraKit on GitHub, a set of 5 additional samples: Coloration Picker, Object Detection with Unity Sentis, QR Code Monitoring with ZXing, Frosted Glass Shader, and OpenAI imaginative and prescient mannequin.
