Pointing with one’s finger is a natural and rapid way to denote an area or object of interest. It is routinely used in human-human interaction to increase both the speed and accuracy of communication, but it is rarely utilized in human-computer interactions. In this work, we show how the recent advent of wide-angle, rear-facing smartphone cameras, along with hardware-accelerated machine learning, opens the door to real-time, infrastructure-free, finger-pointing interactions on today’s mobile phones. We envision users raising their hands to point in front of their phones as a “wake gesture”. This can then be coupled with a voice command to trigger advanced functionality. For example, while composing an email, a user can point at a document on a table and say “attach”. Our interaction technique requires no navigation away from the current app and is both faster and more privacy-preserving than the current method of taking a photo.

Published at ACM Conference on Interactive Surfaces and Spaces (ISS ’23).

The ability to track a user’s arm pose could be valuable in a wide range of applications, including fitness, rehabilitation, augmented reality input, life logging, and context-aware assistants. Unfortunately, this capability is not readily available to consumers. Systems either require cameras, which carry privacy issues, or utilize multiple worn IMUs or markers. In this work, we describe how an off-the-shelf smartphone and smartwatch can work together to accurately estimate arm pose. Moving beyond prior work, we take advantage of more recent ultra-wideband (UWB) functionality on these devices to capture absolute distance between the two devices. This measurement is the perfect complement to inertial data, which is relative and suffers from drift. We quantify the performance of our software-only approach using off-the-shelf devices, showing it can estimate the wrist and elbow joints with a median positional error of 11.0 cm, without the user having to provide training data.

Published at UIST 2023.

Tracking body pose on-the-go could have powerful uses in fitness, mobile gaming, context-aware virtual assistants, and rehabilitation. However, users are unlikely to buy and wear special suits or sensor arrays to achieve this end. Instead, in this work, we explore the feasibility of estimating body pose using IMUs already in devices that many users own — namely smartphones, smartwatches, and earbuds. This approach has several challenges, including noisy data from low-cost commodity IMUs, and the fact that the number of instrumentation points on a user’s body is both sparse and in flux. Our pipeline receives whatever subset of IMU data is available, potentially from just a single device, and produces a best-guess pose.

Published at ACM CHI Conference on Human Factors in Computing Systems (CHI 2023)

Human activity recognition (HAR) has numerous applications, including real-time task assistance, automated exercise tracking, rehabilitation, and personal informatics. Over the years, researchers have explored numerous modalities to sense a user’s actions, and sound has proven to be a useful signal. Sounds resulting from physical activities, such as washing one’s hands or brushing one’s teeth, are often distinctive and enable accurate HAR. However, sampling audio at rates between 8 and 16 kHz carries a power consumption and compute cost. Moreover, these audio ranges capture human speech content and other sensitive information that users might not want to have recorded.

In a bid to protect user privacy, researchers have proposed to featurize recorded data. If done at the edge, featurization is considered privacy-sensitive, but it comes with considerable processing cost. Furthermore, an always-on acoustic activity recognition system increases the power burden; especially on resource-constrained devices such as smartwatches, where the battery cannot be made much larger. In response, we present SAMoSA - Sensing Activities with Motion and Subsampled Audio. Our approach first uses power- and compute-optimized IMUs sampled at 50 Hz to act as a trigger for detecting the start of activities of interest. IMUs are ubiquitous, are heavily engineered for efficiency, and numerous prior works have shown the effectiveness of IMU data to detect user activities. Once we detect the start of an activity using an IMU, we use a multimodal model that combines motion and audio data for classifying the activity. To further minimize the computation cost of processing audio data, we reduce the sampling rates to ≤ 1 kHz. Subsampling can be implemented directly and easily at the hardware level; i.e., instead of post-processing fast-sampled audio data (e.g., 44 kHz), the sensor is directly sampled at a reduced rate. This approach saves power needed to sample, move/store in memory, and featurize the data. Our approach was partially inspired by the always-on "Measure Sounds" (loudness) feature found on recent Apple Watch models. This software-based sensor relies on the microphone, but presumably senses at a very low rate so as to have minimal impact on battery life. At these lower rates, human speech is also unintelligible, thus offering a more privacy-sensitive approach.

We show in our evaluation that motion and sound signals are complementary and provide wide coverage of numerous activities of daily living. A similar approach (albeit without subsampled audio) is reportedly used in Apple Watch’s handwashing recognition implementation, but there is no official documentation of the approach and its implementation. SAMoSA provides a generic, open-source approach that extends to 25 additional activities. Overall, this paper makes the following contributions:

1. A practical activity recognition system that uses 50 Hz IMU along with audio sampled at ≤ 1 kHz, and yet still achieves performance comparable to models using more power-hungry and privacy-invasive 16 kHz audio data.
2. A comprehensive suite of experiments, quantifying the efficacy of our system across 8 audio sampling rates, 4 contexts, 60 environments, and 26 activity classes. We also present a power consumption and a speech intelligibility study for different audio sampling rates.
3. A new smartwatch sensor dataset with synchronized motion and sound data. We further showcase how our multimodal model resolves ambiguity among activities that are confused by a single modality alone.
4. Open-sourced data, processing pipeline, and trained models to facilitate replication and further exploration and deployment in the field.

Citation:

Vimal Mollyn, Karan Ahuja, Dhruv Verma, Chris Harrison, and Mayank Goel. 2022. SAMoSA: Sensing Activities with Motion and Subsampled Audio. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 3, Article 132 (September 2022), 19 pages. https://doi.org/10.1145/3550284

TOCHI '22.

This was work done with Dr. Jürgen Steimle and Adwait Sharma from Saarland University, during my research internship from May 2020 to June 2021

Microgestural interactions enable always-available input both while grasping everyday objects and with free-hands. To sense such gestures, minimal instrumentation of the hands is desirable. However, the choice of an effective but minimal IMU layout is challenging, due to the complex, multi-factorial space of finger gestures, grasps and objects. We propose SparseIMU, a rapid method for designing minimal inertial sensor-based layouts for effective gesture recognition. Furthermore, we contribute a Computational Tool to guide designers with optimal sensor placement. Our approach builds on an extensive Microgesture Dataset that we collected with a dense network of 17 inertial measurement units (IMUs). We performed a series of analyses, including an evaluation of the entire combinatorial space for Freehand and Grasping microgestures (393K layouts), and quantified the performance across different layout choices, revealing new gesture detection opportunities with IMUs. Finally, we demonstrate the versatility of our method with four scenarios.

Citation:

Adwait Sharma, Christina Salchow-Hömmen, Vimal Suresh Mollyn, Aditya Shekhar Nittala, Michael A. Hedderich, Marion Koelle, Thomas Seel, and Jürgen Steimle. 2022. SparseIMU: Computational Design of Sparse IMU Layouts for Sensing Fine-Grained Finger Microgestures. ACM Trans. Comput.-Hum. Interact. Just Accepted (October 2022). https://doi.org/10.1145/3569894

In September, I was presented with the opportunity to introduce myself to the HCI community at UIST. After filling out a form, submitting an abstract and a couple of emails, I'm super excited to present my Lighting Talk at UIST this year! Come say hi! :D

-- Update Nov 2021: Here's a link to the recorded talk! Shoot me an email if you share similar research interests!

In ACM Symposium on Eye Tracking Research and Applications (ETRA '20 Full Papers). Association for Computing Machinery, New York, NY, USA, Article 12, 1–9. DOI:https://doi.org/10.1145/3379155.3391324

This was work done with Dr. Pradipta Biswas and Vinay Sharma from the Indian Institute of Science, Bangalore, during my research internship from June 2019 to  August 2019

People with severe speech and motor impairment (SSMI) lack the fine-grained motor skills required to perform daily activities - often involving picking and placing objects. Backed by recent work, we realized that gaze could be used as an effective input for people with SSMI. This paper describes the design and development of an eye gaze controlled robotic arm that builds upon these ideas. We report two user studies on a pick and drop reachability task. Using the eye gaze controlled interface, participants could complete this pick and drop task at an average duration of less than 15 secs and reach a randomly designated target within 60 secs. It was extremely rewarding to see how the participants reacted to our system - they finally felt a sense of agency. Ultimately, this project led to a publication at ACM ETRA '20