A new technology developed by UW CSE and EE researchers could take the “touch” out of touchscreen and transform the way we interact with our mobile devices. FingerIO, which was developed in the Networks & Mobile Systems Lab led by CSE professor Shyam Gollakota, employs sonar to enable users to interact with their smartphones and smartwatches by gesturing or writing on any nearby surface.
From the UW news release:
“As mobile and wearable devices such as smartwatches grow smaller, it gets tougher for people to interact with screens the size of a matchbook.
“That could change with a new sonar technology developed by University of Washington computer scientists and electrical engineers that allows you to interact with mobile devices by writing or gesturing on any nearby surface—a tabletop, a sheet of paper, or even in mid-air….
“‘You can’t type very easily onto a smartwatch display, so we wanted to transform a desk or any area around a device into an input surface,’ said lead author Rajalakshmi Nandakumar, a UW doctoral student in Computer Science & Engineering. ‘I don’t need to instrument my fingers with any other sensors—I just use my finger to write something on a desk or any other surface and the device can track it with high resolution.'”
Vikram Iyer (left) and Rajalakshmi Nandakumar
FingerIO uses the device’s own speaker, emitting an inaudible sound wave that bounces off the user’s finger and is then recorded using the device’s microphone. That signal is used to calculate the finger’s location in space. The researchers employed a common technique in wireless communication, Orthogonal Frequency Division Multiplexing, to achieve sub-centimeter tracking accuracy.
Sonar has significant advantages over other approaches. For example, unlike camera-based finger tracking, it does not require a direct line of sight—which means it can work through fabric, such as a pocket or a sleeve. And compared to radar, sonar does not require as much computing power or custom hardware to work.
“‘Acoustic signals are great—because sound waves travel much slower than the radio waves used in radar, you don’t need as much processing bandwidth so everything is simpler,’ said Gollakota…’And from a cost perspective, almost every device has a speaker and microphones so you can achieve this without any special hardware.'”
The team, which also includes EE Ph.D. student Vikram Iyer and CSE affiliate faculty member Desney Tan (Principal Researcher at Microsoft Research), will present its research paper on FingerIO at the upcoming CHI 2016 conference in San Jose, California, where it has been recognized with an honorable mention. Nandakumar previously was awarded Best Presentation for FingerIO at the Microsoft Research Student Summit on Mobility, Systems, and Networking.
Read the full news release here, and visit the FingerIO web page here. Watch a video demonstration here, and view the Discovery Channel Canada segment here. Also check out coverage by The Oregonian, GeekWire, Gizmag, FastCompany and Motherboard.