HoloAssist: Multimodal Data Collection for Intelligent AI Agent

ABSTRACT

This work was spearheaded by Taein Kwon (PhD Candidate of Computer Vision Group at ETH Zurich).

This dataset comprises 166 hours of captured data and is designed to serve as a valuable resource for developing interactive AI assistants. It focuses on guiding humans in executing a range of tasks, such as using a coffee machine and assembling or disassembling furniture. The dataset captures scenarios where a task instructor, observing from an ego-centric perspective, intervenes and guides the activity in several ways:

1. Intervention in Case of Errors: The instructor steps in when the task performer makes mistakes.
2. Assistance During Hesitation: The instructor provides guidance when the performer shows uncertainty or hesitation.
3. Positive Reinforcement: When tasks are performed correctly, the instructor offers positive feedback to reinforce the correct actions.

Project Github

In this project, I successfully recruited over 30 participants and managed the data collection process. This entailed several key steps:

1. Participant Training: I guided the participants in using HoloLens 2, ensuring they were comfortable and proficient with the technology.
2. Observation and Assistance: During the project, my role involved closely monitoring the participants. I provided real-time corrections and guidance as they undertook new tasks. These tasks included operating various equipment such as printers, coffee machines, and Nintendo Switch, as well as assembling IKEA furniture.
3. Data Processing and Quality Assurance: After data collection, I focused on post-processing the data. A significant part of this process was conducting thorough quality checks to guarantee accurate hand pose estimation and effective gaze tracking.