VR subversive moment: Throw away the controller and enter "senseless driving"

Unexpectedly, this spring VR manufacturers rolled up for "a pair of hands".

In the first half of 2022 alone, four VR giants have improved the gesture interaction technology of their own VR devices.

On April 20, Meta's VR headset Oculus Quest 2 gesture interaction version was iterated to 2.0, which solved the problem of hand positioning loss. On April 13, Pico, a VR brand owned by ByteDance, announced that its two enterprise-level VR headsets can enhance the gesture interaction experience through customized hand tracking accessories. VR manufacturer HTC also released a new wrist tracker in January this year to assist in predicting the position of the user's hand.

Comparison of Oculus Quest 2 Gesture Interaction Version 1.0 and Version 2.0 in Hand Fast Tracking

Whether it is bare-hand interaction technology or hand-tracking accessories, the gesture interaction technology for VR devices has been constantly iterating. At the same time, many VR companies are still trying to find new gesture interaction methods, such as VR gloves.

On May 6, Dutch VR glove manufacturer Manus announced that its new VR glove Quantum Metagloves with magnetic tracking technology has started pre-sale. On May 4, the domestic motion capture technology giant Noitom released the Hi5 2.0 VR interactive glove. As early as November 2021, Meta showed its own VR tactile glove model and shared its latest achievements in VR gesture interaction.

From left to right are the VR gloves of Manus, Noitom and Meta

Not only that, some potential VR players such as Apple, Google, and Microsoft have also been approved for a number of patents related to smart rings , trying to make smart rings a key input device for next-generation gesture interaction.

Gesture interaction technology is the most basic and natural interaction method, and it is also one of the mainstream interaction methods in VR devices. Generally speaking, gesture interaction refers to the technology of recognizing human gestures through mathematical algorithms , including multiple processes such as image acquisition, gesture tracking, gesture recognition and analysis.

Among them, hand tracking and gesture recognition are two key steps in gesture interaction. Hand tracking refers to accurately locating the position of the user's hands in the VR head-mounted display, and how to accurately recognize the movement of each finger and recognize various forms of the hand belongs to gesture recognition.

Gesture interaction technology also hides a huge market space. Markets and Markets, an international market analysis company, published a report stating that the market for "gesture recognition and touchless perception" will reach $34 billion in 2024.

What are the new ways of using the gesture interaction technology of VR devices? Why are manufacturers upgrading the gesture interaction technology of their own VR devices in the near future? What is the development of gesture interaction technology for VR devices?

After in-depth exchanges with Nuitom CTO Dai Ruoli, technical director of Dream Blossom (iQiyi Smart) Shi Ming, and HTC technical director, we tried to find the answer.

Compared with other mobile devices, gesture interaction technology for VR devices has progressed slowly . For example, after nearly five years of development from 3dof to 6dof, it is still in its infancy from handle interaction to bare hand interaction. With the continuous advancement of technology, bare-hand interaction has also begun to become popular on many consumer-grade VR headsets.

Meta released version 1.0 of gesture interaction in 2019, and introduced bare-hand interaction into the Oculus Quest VR headset, allowing users to directly see their hands in the virtual world and perform some simple interactive operations. After 2 years, Oculus Quest 2 has a gesture interaction version 2.0.

This upgrade mainly solves two problems. One is to solve the problem of difficult hand tracking when the user's hands overlap. In Quest 2's gesture interaction version 1.0, when the user's hands block or overlap each other, it is difficult for the system to recognize the user's gestures. The second is that when the user quickly waved his hands, the system could easily lose track of the hands. The occurrence of both of these situations may cause the hands to suddenly disappear from the virtual world.

Quest 2 gesture interaction version 1.0 and version 2.0 comparison

As for how to solve these two problems, Meta explained in its announcement that the new version improves the performance reliability of hand tracking based on "redesigned computer vision and machine learning methods" . However, since the version update, we have seen many users feedback that although the gesture interaction function has a better experience, the system also has obvious lag.

Regarding this phenomenon, Dai Ruoli explained that the processing and computing capabilities of the VR headset itself are relatively limited , while Meta acquires data based on computer vision and uses part of the computing resources to predict the position of the user's hand. The VR head-mounted display brings greater computing and processing pressure , which is also the main reason for the "system freeze" phenomenon.

HTC also launched a new wrist tracker VIVE Wrist Tracker in March this year, and upgraded its own hand tracking version at the end of last year. According to information on HTC's official website, when the user wears the wrist tracker on the wrist, even if the hands leave the camera's field of view for a period of time, the system can predict the movement of the hand through the high-frequency IMU data and kinematic model of the wrist tracker. motion track.

The relevant person in charge of HTC also shared with Zhishi that compared with the VIVE Focus 3 controller, VIVE Wrist Tracker removed buttons, tactile input and other components to reduce the size of the battery and make the controller smaller and lighter. According to reports, HTC's wrist tracker is 85% smaller and 50% lighter than the VIVE Focus 3 controller.

VIVE Wrist Tracker wrist tracker

We can see that Meta's bare-hand interaction solution is still based on computer vision, while HTC's bare-hand interaction solution needs to rely on peripherals such as wrist trackers. The former can help users quickly and effectively experience the fun of bare-hand interaction , while the latter improves the accuracy and reliability of gesture interaction to a certain extent through inertial sensors. Faced with different application scenarios, the bare-hand interaction solutions of VR equipment from different manufacturers have their own emphasis.

But will bare-hand interaction definitely bring people a better experience? Dai Ruoli said in an interview: "At a point in time when the technology has not yet reached maturity, embedding new human-computer interaction methods such as bare-handed interaction into the operating system will give people excessive expectations."

"In terms of success rate of operation and tactile feedback, the handle is far more fun and accurate than bare-hand interaction." He explained that in the field of human-computer interaction, 80% of the operation accuracy and 100% of the operation accuracy The interactive experience brought to users is completely different.

Tong Xin, chief researcher of the Network Graphics Group of Microsoft Asia Research Institute, confirmed this in an interview with the media in 2017. "When I hold the handle in my hand, the handle is not only an input device, but also an output device. It can give me an output feedback through vibration and force feedback. When I wave my hand in the air, I lose own output channels. "

Limited computing consumption, high learning costs, and low operational accuracy are the main reasons hindering the development of bare-hand interaction technology for VR devices. But in terms of interactive experience, bare-handed interaction will also provide VR content producers with more room to play . In many VR devices, bare-hand interaction is used as an auxiliary gesture interaction method.

In order to find a balance between the immersion brought by bare-hand interaction and the high accuracy provided by VR handles, VR manufacturers have come up with another way of gesture interaction-VR gloves.

At present, there are many forms of VR gloves, but the VR gloves that can be mass-produced and sold on a large scale are mainly divided into two types based on inertial sensors and bending sensors .

Different from the data processing method based on computer vision gesture interaction, VR gloves based on inertial sensors will collect a large amount of data on hand positioning and finger interaction through sensors. However, VR gloves based on bending sensors have low measurement accuracy, and due to the high requirements for fit, the comfort of wearable devices is also low.

We had the opportunity to experience the Hi5 2.0 VR interactive gloves first. A pair of Hi5 2.0 VR interactive gloves has a total of 12 inertial sensors , and each hand wears six wireless sensors, which are distributed on five fingers and the back of the hand. At the same time, the suit will also be equipped with a data transceiver , which can be plugged into a computer or VR headset to transmit data.

Hi5 2.0 VR Interactive Gloves

Turn on the sensor, install the sensor, fix the hand tracker, and select the calibration interface. The calibration method of Hi5 2.0 VR gloves is more complicated than we imagined.

During the experience, we found that the Hi5 2.0 VR interactive glove can provide a more accurate gesture recognition process. For example, the position of the hand and the change of the gesture can basically display the changes in real time. According to the official website, the device's attitude calculation frame rate has been increased from 180fps to 500fps, and the R&D personnel have also improved a certain degree of anti-magnetic interference through algorithms.

Noitom R&D personnel are calibrating Hi5 2.0 VR interactive gloves

In addition, researchers are also experimenting with different VR glove designs.

In November 2021, Meta announced the prototype of its 7-year-old VR tactile glove. This VR glove replaces the motor with a micro-air valve, and adjusts the inflatable instrument on the finger through a complex control system to form a realistic pressure. kind of touch.

Meta Touch Gloves

The Quantum Metagloods VR gloves of Dutch VR glove developer Manus are based on magnetic tracking technology.

Different from Noitom’s solution that uses algorithms to reduce the magnetic interference caused by the natural environment and ensure tracking stability, the gesture interaction solution through magnetic tracking requires the cooperation of two devices. One device is equipped with coils in the three directions of xyz and at the same time The VR glove will also contain a coil, and the two coils will generate magnetic force lines in a high-frequency vibration manner, and the two magnetic force lines will cut each other, thereby obtaining changes in the position of the hand .

Although VR gloves can provide more accurate gesture motion capture, the price and cost of VR gloves determine that they can only be used for a small number of geek players and commercial markets.

It is reported that the prototype cost of Meta gloves is about 5,000 US dollars (about 33,400 yuan), while the pre-sale price of Manus's Quantum Metagloods is 9,000 US dollars (about 60,000 yuan), and Noitom's Hi5 2.0 VR interactive gloves are priced at 9,800. Yuan Renminbi.

From left to right are the VR gloves of Manus, Noitom and Meta

Meta has described the project to develop VR haptic gloves as a "moonshot". "When we started developing VR haptic gloves, we hoped to create a mass-produced, affordable consumer device that allows people to experience the virtual world anywhere." Meta's project leader said: "But it's almost impossible."

During the investigation, we also found that there are many homemade VR gloves made by UP owners on station B. These prices range from 100 yuan to 350 yuan, and some VR gloves even have force feedback function. We have seen that these self-made VR gloves can be used in some simple VR games, and can make some simple gestures such as holding, pinching, and flipping, but the display delay is also obvious, and some ghost scenes often appear.

B station UP owners try to make homemade VR gloves

Faced with the huge price gap of VR gloves, an industry professional engaged in VR gloves analyzed that the cost of VR gloves for the B-end market is mainly in R&D investment, parts and human resource costs , and the price is determined according to market demand. This means that the price of making an ordinary VR glove may not be high, but making a pair of VR gloves that can be used in commercial scenarios and provide accurate precision requires a high cost.

Although VR gloves can take into account the immersion of VR equipment and the accuracy of gesture interaction, how to reduce the price of VR gloves will be a difficult problem that must be solved in the development of VR gloves.

To overcome problems from algorithms, engineering, and interaction design

Judging from the two methods of bare-hand interaction and VR glove interaction, the development of gesture interaction technology for VR devices is constantly being updated and iterated, and some of the difficulties faced before have gradually found solutions. But overall, there are still many difficulties in gesture interaction technology for VR devices.

What is preventing us from throwing away the VR controller and adopting new gesture interaction methods? Shi Ming, the research and development personnel of the domestic VR headset adventure series and the technical director of Dream Bloom (iQiyi Intelligence), gave three main points in an interview with Zhishi.

The first is the algorithm level. At present, the gesture interaction technology used for VR devices basically adopts deep learning technology, so the research and development process needs to rely on a large amount of business scene annotation data. However, the cost of acquiring 3D data based on gestures is high, and it is difficult to ensure data quality and consistency.

Keypoint recognition for gesture interaction

Secondly, gesture interaction technology is more complicated in terms of engineering optimization. Gesture interaction technology needs an excellent model as a premise, and needs to do small model reasoning and deduction. In the reasoning process, researchers must not only ensure the accuracy of the algorithm, but also ensure the real-time efficiency of the data.

The last is the design of interactive use. VR developers need to avoid visual blind spots when designing gesture interaction-related operations, and at the same time provide users with a better interactive experience.

Regarding the design of gesture interaction, Liu Xiaoyu, a former designer of Facebook Reality Labs, also mentioned in an interview that gesture interaction technology is prone to occlusion during use, and the user's learning cost needs to be considered when designing gesture interaction actions.

In addition, Tong Xin, chief researcher of the Network Graphics Group of Microsoft Asia Research Institute, also mentioned that for gesture recognition, it is difficult for machines to distinguish between input and non-input states . Unlike the key interaction method of the keyboard, it is difficult for the machine to distinguish whether you are shaking your hand, making a gesture of waving, or wanting to shake your hand to rest when you are tired of playing.

Generally speaking, the gesture interaction technology used for VR devices mainly faces two difficulties as a whole . On the one hand, which gesture interaction method can take into account immersion, accuracy and low cost, and can be used for all VR devices. On the other hand, how to create the most suitable gesture interaction function for this device based on the basic configuration of the VR device.

It is foreseeable that until the bare-hand interaction technology can achieve higher accuracy and the cost of VR gloves decreases, the handle will still be one of the mainstream solutions for gesture interaction of VR devices.

At present, the gesture interaction for VR devices is divided into three directions: the existing handle interaction method, the bare-hand interaction method that is gradually becoming the mainstream, and the interaction method of external wearable devices that is still in the minority stage. But it is worth noting that these three modes of interaction are not substitutes, but can coexist.

In the era of rapid development of mobile phones and computers, both typing and touch screen interaction modes have brought new interactive experiences to people. Gesture interaction will also bring new changes to VR device experience and VR content production.

The technical development of gesture interaction will always have endless directions, but the gesture interaction technology left behind by the times is after all a minority. How the gesture interaction technology used in VR devices will evolve, perhaps time will tell us the answer.