Facebook Wrist-Based Device: The Next Human-Computer Interaction

Facebook Wrist-Based device..

Mixed Reality (MR) and Augmented reality AR devices are the future of PCs and smartphones, and they will change the way we connect with people in the near or far future.

If you have ever tried any AR device (i.e., Microsoft Hololens, Magic Leap, Daqri, or even the handheld AR applications), The first thing you think “how to interact with the AR virtual objects”.

AR devices have introduced different methods for interaction (e.g., controller or hand gesture), which are “in my previous experience of using Microsoft Hololens on construction site” limited in function and doesn’t provide a seamless experience. You can check my paper on using Microsoft Hololens on construction site (Extended Reality (XR) Approach for Construction Quality Control), and on-site experiment video.

Nowadays, We have excellent XR devices with a high resolution, latency and image quality but with poor human-computer interaction (HCI). As I don’t believe Xbox controllers or motion tracker controllers are the best way of XR interaction. Hence, we desperately need a better way of interacting.

AR interaction and Facebook Vision

The Chief Scientist of Facebook Reality Labs (FRL) Michael Abrash has called AR interaction “one of the hardest and most interesting multi-disciplinary problems around”, which is true as we are dealing with an immersive display that literally shifts the way humans interact with computers. This is kind of similar to the 1960s when Doug Engelbart’s team invented the mouse and helped pave the way for the graphical user interfaces (GUIs) that dominate our world today. Generally speaking, The introduction of (GUI) revolutionised human-computer interaction we know today.

Nowadays, All types of XR interfaces will require a complete rethinking of how humans and computers interact, and this will transform our relationship with the digital world.

FRL Research are working on developing natural and intuitive way to interact with AR glasses (human-centric) with a mission of building an interface for AR that won’t force user to choose between interacting with devices and the world around.

“Imagine being able to teleport anywhere in the world to have shared experiences with the people who matter most in your life — no matter where they happen to be,” Andrew Bosworth, who leads FRL. “That’s the promise of AR glasses. It’s a fusion of the real world and the virtual world in a way that fundamentally enhances daily life for the better.”

Neural interfaces: Facebook wrist-based device

Facebook is building a series of research prototypes to understand wristband haptics. The first prototype called “Bellowband”  a soft and lightweight wristband, it has eight equally spaced pneumatic bellows that extend into the wrist for haptic feedback. The air pressure inside the bellows can be controlled to render pressure and vibration in complex patterns in space and time.

Bellowband is a pneumatic wristband that can deliver local pressure and vibration at eight positions around the wrist. The device is made from thermoplastic polyurethane (TPU), resulting in a lightweight wristband capable of rendering complex pressure and vibration to the user (source)

Another Facebook wrist-based device is Tasbi (Tactile and Squeeze Bracelet Interface) a multisensory haptic bracelet that incorporates two haptic modalities: vibration and squeeze. This interface is designed to use Electromyography (EMG) which is an electrodiagnostic medicine technique for evaluating and recording the electrical activity produced by skeletal muscles. EMG in Facebook wrist device uses sensors to translate electrical motor nerve signals that travel through the wrist to the hand into digital commands that you can use to control the functions of a device. These signals let user to communicate crisp one-bit commands to user’s device, a degree of control that’s highly personalisable and adaptable to many situations.

Tasbi enables a variety of interactions with the virtual world. Vibrotactile feedback may substitute for fingertip contact with virtual buttons or other types of user interfaces, while squeeze may convey the weight or stiffness associated with manipulating virtual objects. Squeeze and vibration can further add to telepresence and remote social interactions such as hand shaking or holding, and can also provide immersive feedback for gaming and training (source)

The signals through the wrist are so clear that EMG can understand finger motion of just a millimetre. That means input can be effortless. Ultimately, it may even be possible to sense just the “intention” to move a finger.

Initially, EMG will provide just one or two parts of control “click”, this is equivalent to tapping on a button. These are movement-based gestures like squeeze and release of the thumb and forefinger that are easy to execute, regardless of where you are or what you’re doing, (walking, talking, or sitting with your hands at your sides, in front of you, or in your pockets). You can use your fingers together for clicking Clicking without the need for a wake word, This would make it the first ubiquitous, ultra-low-friction interaction for AR.

But that’s just the beginning. EMG will ultimately progress to richer controls. In AR/VR, the user will be able to actually touch and move virtual UIs and objects, as you can see in this demo video.

Summary

Neural interfaces such as Facebook wrist-based device using EMG sensors on the wrist or  EEG on brain (e.g. neurallink’s  video of Monkey MindPong ) can enable user to interact with virtual objects or physical environment and take it to the next level. I can imagen how much applications we’ll have from this type interaction especially in the area of (Digital Twin) in the built environment. Since the device is also tracking neural signals, not the actual fingers, This technology can enable someone born without a hand to operate a virtually to activate reality – digital twin paradigm.

Finally, I would like to say “With today’s technology, there are tons of possibilities for innovation and discovery. Let’s join the revolution and change history!“.

References

• Explorations of Wrist Haptic Feedback for AR/VR Interactions with Tasbi
• E. Pezent et al., “Tasbi: Multisensory Squeeze and Vibrotactile Wrist Haptics for Augmented and Virtual Reality,” 2019 IEEE World Haptics Conference (WHC), Tokyo, Japan, 2019, pp. 1-6, doi: 10.1109/WHC.2019.8816098.
• Al-Adhami, M., Wu, S. and Ma, L., 2019, October. Extended Reality Approach for Construction Quality Control. International Council for Research and Innovation in Building and Construction (CIB).
• E. M. Young, A. H. Memar, P. Agarwal and N. Colonnese, “Bellowband: A Pneumatic Wristband for Delivering Local Pressure and Vibration,” 2019 IEEE World Haptics Conference (WHC), Tokyo, Japan, 2019, pp. 55-60, doi: 10.1109/WHC.2019.8816075.

What to do?

1. Subscribe to our Newsletter.
2. Follow our pages on social media Facebook, Twitter, Instagram and LinkedIn
3. Leave us a comment
4. For more information Contact Us.
5. Don’t forget to Check out our other articles.

THANK YOU & HAVE A WONDERFUL DAY!