Researchers from Meta and ETH Zurich have created a revolutionary software program called TouchInsight, enabling users to transform any surface into a functional digital keyboard.
As we venture into virtual and augmented reality landscapes, the current experience is marred by clumsiness, with interactions unfolding at a glacial pace compared to those on conventional computers and mobile devices. Floating digital keyboards necessitate an unnatural palm position, forcing users to painstakingly tap individual keys while providing no tactile feedback or allowing for wrist relaxation. Quest headsets simplify pairing with a Bluetooth keyboard for fast and efficient typing, eliminating the need to lug around a bulky device that dwarfs the headset itself.
With the ability to transform any flat surface into a digital keyboard, headsets can provide partial haptic cues and alleviate wrist strain without requiring users to lug around a physical keyboard.
Currently, builders can construct surface-locked digital keyboards on Quest using advanced hand-monitoring technology that requires users to tap the floor to calibrate their position. Any slight mismatch between the digital floor peak and the actual floor peak ultimately leads to an unacceptably high error cost.
Meta has been actively working on improving language processing capabilities over the past year, with CTO Andrew Bosworth reportedly typing at a remarkable rate of almost 120 words per minute. The previous year’s results seemed to necessitate a series of fiducial markers on the table, effectively constituting a robust dynamic calibration system. Despite Meta’s reluctance to disclose the exact error rate of that system,
According to the TouchInsight paper, six researchers report that their novel approach yields an average typing speed exceeding 70 words per minute (WPM), with a median rate of 37 WPM. Typically, individuals sort around 40 words per minute (WPM) on a standard keyboard, while highly skilled typists reach speeds ranging from 70 to 120 WPM, depending on their proficiency level.
TouchInsight reportedly functions seamlessly with a standard Quest 3 headset, requiring no additional hardware or peripherals for optimal use. While Meta didn’t release the error rate for its previous year’s system, the study reveals that the average uncoded error rate for TouchInsight stands at a relatively low 2.9%.
“This paper presents a real-time pipeline capable of detecting contact entry from all ten fingers on any physical surface solely through self-contained hand monitoring.”
Our proprietary Technique, TouchInsight, employs a neural network that accurately predicts the timing, the instant the fingers make contact, and the precise contact location of a touch event. TouchInsight leverages a probabilistic framework, incorporating bivariate Gaussian distributions to quantify the uncertainties inherent in sensor readings. By incorporating contextual priors, the system accurately infers intended consumer inputs, effectively mitigating the impact of sensing inaccuracies.
TouchInsight enables intuitive interactions beyond mere keyboard input, allowing users to intuitively detect and respond to finger presses on various bodily surfaces. According to researchers, this technology could potentially enable a novel, immersive experience akin to a virtual game of Whac-A-Mole, where players use their fingers to physically squash tiny mole characters on a surface.
Researchers will demonstrate TouchInsight at the upcoming ACM Symposium on Consumer Interface Software Technology in Pittsburgh next week.
Will Meta ultimately merge TouchInsight with Horizon OS, and if not, what are the specific hurdles preventing this integration from occurring?
