A Quest 3 experiment: Is hand tracking finally ready for prime time?

Image: Holonautic

Hand tracking is considered the VR input of the future. But how close to the mainstream is it really? I tested it with someone who has no prior experience with VR or hand tracking.

Most experts agree that hand tracking will change the way we interact in and with VR and MR. More debatable is how well hand tracking works today, and how long it will take to catch up and replace controllers as the primary input — or if it ever will.

Some in the industry believe hand tracking will completely change the industry in just two years, while others are more skeptical. The debate has been further fueled by the introduction of Apple Vision Pro, which is known to rely only on eye and hand tracking.

Meanwhile, the buzz surrounding the device has died down since its launch in February. Apple’s first space computer won’t change the VR industry overnight, and some could argue that it has increased rather than decreased awareness of the benefits of the controllers, its precision controls and aperture.

For hand tracking to become the primary login method, it needs to be seamless and intuitive to use for most consumers. To put it to the test, I sought out a suitable test subject: my friend Esther (name changed for this article), who is in her early 70s and has no prior experience with VR, hand tracking, or video games.

The handprint is rated for its accessibility. If Esther could use the technology, then nothing should stand in the way of tracing the hand. Correct?

The first steps

I let Esther experiment with Meta Quest 3’s hand tracking for about an hour.

First, I ran through the Meta Quest hand tracking tutorial. Esther immediately understood the concept of touch input, which she was familiar with from mobile devices, but it didn’t always work the way she thought it would. For example, she had to select and tap several items several times before the system correctly interpreted the input. On the other hand, she was able to learn the gestures relatively quickly.

When I asked her what she thought of the user experience, Esther said:

“It was okay, I didn’t think it was very easy, I thought it could be easier, but once I saw how it worked a couple of times, it wasn’t hard to remember the simple gestures.”

A messy fast food stand

Next we tried Vacation Simulator. The VR game has had experimental support for hand tracking since 2020, which has become a regular feature since the last update.

Unlike the studio’s last project, Vacation Simulator was not designed from the ground up for hand tracking, which was evident in the experiment. Esther’s ten minutes at the beach food stand was quite exhausting and in the end she couldn’t even make the hot dog I asked her to. Most of the ingredients were distributed on the ground.

What Esther wanted to do with her hands and what play interpreted as an interaction were often two different things. She later described the experience as “disappointing”, but said she could imagine returning to this world and enjoying it more the second and third time around, as she was still a “complete novice” in the field. Looking at it, it felt like the hand tracking was more of a hindrance than a help to the enjoyment of the game.

The Midas Problem and other issues

Esther enjoyed the Cubism puzzle game the most, but again she had difficulties that had more to do with the hand-tracking technology than the game itself. After a few tries, she found it relatively easy to grasp the puzzle pieces with her fingers. It was more difficult to drop them in the right place, as the headset did not always recognize the movement and the pieces of the puzzle “stuck” to her fingers, as she said.

Hand tracing isn’t something that just works out of the box, it takes some learning. It’s not enough to just grab an object and let it go. You have to pick it up and let it go in a certain way. This is a general hurdle for hand tracking data that has yet to be overcome.

One problem is that computer vision algorithms do a poor job of recognizing subtle movements. Another is that the system may overinterpret hand movements as interactions when they are not. VR developer Antony Vitillo aptly described this phenomenon as the Midas touch problem (Greedy Midas wanted everything he touched to turn to gold. When even food and drink turned to gold, and he was in danger of dying from hunger, he asked the god Dionysus to get his wish back).

The experiment ended with the Hand Physics Lab, which brought a happy smile to Esther’s face. More than other apps, the hand-tracking experiment kit thrives on the technology’s shortcomings, which makes it easier to see them.

Handprinting: a long way from “good” to “perfect”

In the end, I realized that hand tracking is still an experimental technology. VR enthusiasts are aware of its shortcomings and know how to work around them. Beginners spending a lot of money on a device have a different requirement: it just has to work, especially if it’s going to be their primary method of VR and MR input.

First, hand tracking will only be mature when it feels like we’re interacting with our hands, rather than learning how to use them so that the system interprets them correctly. Second, it should work almost flawlessly. It’s not enough, for example, to be able to recognize a button press 90 percent of the time. That would be disappointing.

And for engineers, the task is getting harder, not easier: Going from 90 to 99.9 percent accurate pose and interaction recognition in the coming years is likely to be much harder than going from 0 to 90 percent.

With that in mind, I don’t see Meta selling Quest headsets without controllers anytime soon. And on the content side, there’s still a lot of work to do before hand tracking is considered mainstream. The vast majority of VR apps still require controllers, and for good reason.

Note: Links to online stores in articles may be so-called affiliate links. If you purchase through this link, MIXED receives a commission from the provider. The price does not change for you.