Shared Input Multimodal Mobile Interfaces: Interaction Modality Effects on Menu Selection in Single-task and Dual-task Environments

Shared Input Multimodal Mobile Interfaces: Interaction Modality Effects on Menu Selection in Single-task and Dual-task Environments

Authors

Shengdong ZHAO,Duncan BRUMBY,Mark CHIGNELL,Dario SALVUCCI,and Sahil GOYAL

Paper

Abstract

Audio and visual modalities are two common output channels in the user interfaces embedded in today‚Äôs mobile devices. However, these user interfaces typically center on the visual modality as the primary output channel, with audio output serving a secondary role. This paper argues for an increased need for shared input multimodal user interfaces for mobile devices. A shared input multimodal interface can be operated independently using a specific output modality, leaving users to choose the preferred method of interaction in different scenarios. We evaluate the value of a shared input multimodal menu system in both a single-task desktop setting and in a dynamic dual-task setting, in which the user was required to interact with the shared input multimodal menu system while driving a simulated vehicle. Results indicate that users were faster at locating a target item in the menu when visual feedback was provided in the single-task desktop setting, but in the dual-task driving setting, visual output presented a significant source of visual distraction that interfered with driving performance. In contrast, auditory output mitigated some of the risk associated with menu selection while driving. A shared input multimodal interface allows users to take advantage of multiple feedback modalities properly, providing a better overall experience.

KEYWORDS: earPod, eyes-free, shared-input multimodal interfaces

Shen

Shen is an HCI professor at the National University of Singapore working on realizing his vision of HeadsUp Computing, a new Interaction paradigm that can transform the way we live and interact with computers. In his free time, Shen loves to read, run, spend time with family and friends, and explore nature.

Leave a Reply