Xiaoning Ma, Xin Yang, Shengdong Zhao, Chi-Wing Fu, Ziquan Lan, Yiming Pu

This paper proposes to put domestic robots as buddies on our contact lists, thereby extending the use of social media in interpersonal interaction further to human-robot interaction (HRI). In detail, we present a robot management system that employs complementary social media platforms for human to interact with the vacuuming robot Roomba, and a surveillance robot which is developed in this paper on top of an iRobot Create. The social media platforms adopted include short message services (SMS), instant messenger (MSN), online shared calendar (Google Calendar), and social networking site (Facebook). Hence, our system can provide a rich set of user-familiar, intuitive and highly-accessible interfaces, allowing users to flexibly choose their preferred tools in different situations. An in-lab experiment and a multi-day field study are also conducted to study the characteristics and strengths of each interface, and to investigate the users’ perception to the robots and behaviors in choosing the interfaces during the course of HRI.


Robots in My Contact List



Robots in my contact list paper


This paper proposes to put domestic robots as buddies on our contact lists, thereby extending the use of social media in interpersonal interaction further to human-robot interaction (HRI). In detail, we present a robot management system that employs complementary social media platforms for human to interact with the vacuuming robot Roomba, and a surveillance robot which is developed in this paper on top of an iRobot Create. The social media platforms adopted include short message services (SMS), instant messenger (MSN), online shared calendar (Google Calendar), and social networking site (Facebook). Hence, our system can provide a rich set of user-familiar, intuitive and highly-accessible interfaces, allowing users to flexibly choose their preferred tools in different situations. An in-lab experiment and a multi-day field study are also conducted to study the characteristics and strengths of each interface, and to investigate the users’ perception to the robots and behaviors in choosing the interfaces during the course of HRI.

Categories and Subject Descriptors

H.5.2. User Interfaces – [Interaction styles]; I.2.9 Robotics – [Commercial robots and applications]

General Terms

Human Factors, Experimentation.


Human-robot interaction, domestic robots, social media platforms, intuitive interaction.


Robots are starting to enter our homes, evidenced by the increasing proliferation of domestic helpers like the vacuuming robot Roomba (by iRobot Corp.), lawn mowing robot Robomower (by Friendly Robotics Ltd.), etc. In the future, homes are likely to be equipped with one or more robots to serve the need of users, especially those who may not stay at home all the time and thus have to rely on domestic robots to take care of the household and family. Therefore, it is crucial to provide a management system to enable people to efficiently, ubiquitously and intuitively interact with domestic robots, and hence bridge the gap between domestic robots and the general public.

To serve this purpose, the system between robots and users must provide intuitive interfaces for the users to learn and use since domestic robots target ordinary home users who often have limited computing knowledge. Moreover, it must be able to handle the varying contexts and scenarios of interaction in order to ubiquitously connect human with their robots. It will be desirable if the system could provide complementary HRI interfaces which fit in different interaction contexts, such as working in stationary office environment, standing on a bus, walking, etc.

To the best of our knowledge, we are unaware of any such systems that allow users to ubiquitously interact with multiple robots through a set of complementary and non-exclusive interfaces. Thus, in this paper, we present a highly-accessible and extensible robot management system which employed social media platforms to provide intuitive and easy-to-learn user interfaces. Specifically, four types of social media platforms are adopted in the system, including short text message services (SMS), instant messenger (i.e., MSN), shared online calendar (i.e., Google Calendar) and social networking sites (i.e., Facebook), to interact with domestic robots. Our two robots include a vacuuming robot Roomba, and a surveillance robot developed by us on top of an iRobot Create in the purpose of making our system more capable of doing household chores. The proposed approach, including the four social media platforms adopted, the picture of our robots and the user scenarios, is shown in Figure 1.

robots fig.1

Figure 1. Using social media to interact with domestic robots. 

We choose existing social media platforms to provide the user interfaces because social media platforms are highly popular with a large population of skilled users; therefore, reusing these platforms as the interaction media to domestic robots can minimize users’ efforts for learning. Besides, since different social media platforms are designed to serve different kinds of needs in different scenarios, supporting multiple complementary platforms as in our system can thus cater to user’s needs emerged from different scenarios, such as on the road or in the office.  The followings highlight the contributions of this paper:

First, this is the first paper we are aware of that harness complementary social media platforms to achieve better user experiences in HRI.

Second, we implemented a working system as described in this paper, deployed it into a multi-room apartment, and recruited users to try out the system in a real home environment for three days. To the best of our knowledge, we are unaware of any other work that attempted to deploy such a system into a real home to study its effect on HRI for a period of multiple days.


A growing number of researchers have begun to explore the field of domestic robots. While some researchers focused on the implementation and algorithmic aspects of domestic robots, such as [10, 17], others studied the application of domestic robots (a majority of them focused on the vacuuming robot, Roomba), e.g., on how design can influence HRI in home setting, e.g., [2, 4, 12]. Many researchers are also interested in designing novel interaction methods to enable natural and intuitive HRI, which includes the design of paper tag interfaces to facilitate implicit robot control [22], the use of tangible objects such as toys [8], accelerometer-based Wii-mote [7], laser pointers [9], sketching on a tablet computer [18], using gaze and blink (BlinkBot [15]), and using enhanced projector-camera (LuminAR [13]) to control robots. Moreover, researchers also worked on extending robots to other housework tasks beyond simple vacuum cleaning, such as [16] and [20].

So far, studies on social media platforms mostly concerned with human-to-human interaction [5] instead of human-robot interaction, except [3, 6, 11, 14], where intelligent virtual agents were used to communicate with humans via instant messengers [6], and to help humans plan their calendars [3]. In addition to virtual agents, an SMS interface has also been proposed to control home appliances [11]. Cellbots, an open source library available at http://www.cellbots.com/, also allows users to control different robots (iRobot, LEGO Mindstorm, etc.) using SMS. Moreover, Mavridis, et al. [14] proposed a social robot is used to wander in the lab, attempting to talk to people it encountered. This robot obtained people’s information via Facebook to enhance conversation and face recognition performance. In a separate effort, a Facebook-connected desktop pet robot called “Pingo” (by Arimaz Inc.) was brought to the market, which can read Facebook updates, news, sing songs, and give weather forecasts. While all these work leverages social media platforms, our work differs from them because our system involves autonomous robots instead of virtual agents [6] or stationary machines [11]. Being “robots” sets them apart from other types of electronic devices such as “desktop computers” or “home appliances.” More than these stationary devices, robots can share physical space with people and can take the initiative to display a variety of autonomy and intelligence over the information world as well as the physical world [21]. Moreover, unlike entertainment and social robots, domestic robots play a dual role of doing housework and act like human companions or even family members. These distinguish our work from [14] and “Pingo,” which employed Facebook only for socializing or entertainment.


We designed the following scenarios to illustrate how our approach can employ complementary social media platforms to facilitate HRI. All the tasks and interfaces in these scenarios have been developed in our system and were used to conduct the lab experiment and multi-day field study.


Jason and Maggie are a professional working couple who works from 9am to 6pm on working days. Their son Mike is studying abroad. They have two domestic robots, Tiddy (vacuuming) and Spotty (surveillance), to take care of household chores which include preparation work for the upcoming Christmas Eve party.

Party Scheduling through Calendar

On Dec. 20, Jason uses the Google Calendar to schedule a Christmas Eve party starting at 6pm on Dec. 24th. The calendar shows that Tiddy has been scheduled to vacuum the living room during that time. Hence, Jason reschedules Tiddy’s cleaning task to another time slot via the calendar interface.  Due to the rescheduling, Tiddy sends an automatic SMS to Maggie (the owner of the previous cleaning task) to inform her about the change. The calendar interface is shown in Figure 2.

robots fig.2

Figure 2. Using Google Calendar to interact with the robots.

Progress Update through Facebook

Since Jason has confirmed the schedule of the Christmas Eve party, Spotty and Tiddy start to post the tasks they have done for the party on Facebook. On Dec.22, Spotty receives a message from one of Jason’s Facebook friends asking about the Christmas tree in the living room. Hence, Spotty moves to the living room, takes a picture using its wireless camera and shares the picture on its Facebook wall. This scenario is shown in Figure 3.

robots fig.3

Figure 3. Using Facebook to interact with domestic robots.

 Video Chatting through IM

Mike could not join the party since he is aboard, but he still hopes to take a look at the Christmas tree.  Hence, he starts an MSN video chat with Spotty, as shown in Figure 4:

Mike: Could you show me the Christmas tree in living room? Spotty: I am moving to living room … Spotty: I am in the living room now.

Spotty: I am looking at the Christmas tree now.

Mike: Can you turn left a bit?

Spotty: I am turning left.

Mike: Thanks Spotty. It‘s fantastic! 

robots fig.4

Figure 4. Using MSN to interact with domestic robots. 

Arranging a Urgent Task by SMS

Early in the morning of the Christmas Eve party, Jason is on a bus heading to work, and suddenly remembers of the bits of paper he left in the bedroom. Realizing that he may not have time to clean them up before the guests arrives, Jason immediately sends an SMS to Tiddy, as shown in Figure 5. Soon after that, Tiddy acknowledges Jason with an SMS; ten minutes later, Tiddy sends another SMS to inform him of the task completion.

robots fig.5

Figure 5. Using SMS interface to interact with the robots.


We envision a flexible and extensible domestic robot management system which could accommodate both single and multiple users, with each user free to choose the desirable client to interact with each robot at home. Based on the above vision, we designed and implemented a working system based on the client-server architecture. Figure 6 depicts both the hardware setup and the software components in the client and server side.

Client Side

The client side requires no development or maintenance efforts from users, the only step required is to install (if needed) the standard version of the social media platforms in their computer/tablet/smartphone, then add the robot’s account to their contact list (just like adding a friend). For example, adding a robot to the user’s MSN simply means installing the standard version of MSN and then adding the robot to one’s contact list.

While there are a variety of social media available, we choose the following four platforms (SMS, MSN, Google Calendar and Facebook) due to their popularity and complementary abilities to serve a range of users’ needs.

Short Message Service (SMS)

We choose to support SMS because it is arguably the most widely used data application in the world, with 4.16 billion active users by the end of 2010 [1]. SMS is commonly used in mobile scenarios as it takes relatively short setup time and can be used almost anywhere covered by mobile phone network, hence supporting SMS in our system helps increase the system ubiquity. However, most phone models support only short text-based messages in chunks without graphics and video feeds, which may limit the type of feedback that the robots might send to the users.

Instant Messenger (IM) 

Similar to SMS, Instant Messenger (IM) clients are also widely adopted. Some popular clients have over hundreds of millions of active users (i.e., Windows Live Messenger: 330 million active users by June 2009; Yahoo Messenger: 248 million active users by Jan 17, 2008). IM offers well-designed notification functionality so that it can easily get user’s attention while he is working with other computer applications. The video chat capability of IM also enables additional services to be used with domestic robots. On the other hand, video chatting in IM typically need fast internet connection, which may make it less ubiquitously available for HRI as compared to SMS. In our system, we currently support the Windows Live Messenger client (MSN) as it is one of the mostly commonly used IM clients. To interact with a robot using MSN, users only need to add the robot’s MSN account to their contact lists, and then communicate with the robot just like chatting with anyone else.

robots fig.6

Figure 6. Overview of system implementation: users, social media interfaces, software architecture, and home-robot system.

Shared Online Calendar

Shared calendars are designed for both individual and group to manage, plan, and overview schedules. Hence, interacting with robots via shared calendar not only allows users to manage robot tasks together with their own workflow, but also allows robots to check each family member’s schedule and suggest reschedule existing tasks if there is any overlap of activities. Our system adopted Google Calendar to interact with robots. Unlike our SMS and MSN clients, Google Calendar does not support real-time communication because excessively frequent data retrieval is prohibited by the Google server. According to our experience, the minimum time between successive accesses is around 40 seconds. Thus, Google Calendar functions more as a shared task-planning interface than a real-time communication interface in our system.

Social Networking Site

Facebook is included in our system as a representative social networking site due to a number of reasons. As of January 2011, it has more than 600 million active users. Besides, Facebook is a community-based website designed for interaction amongst a large group of people for social networking purposes. It has also been largely explored for many research purposes, such as [14, 19]. Facebook mixes robots’ activities with those of human’s, and the viral and snowballing effect of Facebook could possibly help promote robot adoption to more users.

In order to talk to robots in Facebook, users can just add the robots’ Facebook account as a friend, then talk to them by leaving messages on robots’ walls. Feedbacks from robots are sent back via posts on users’ wall. However, to prevent spamming, Facebook does not allow frequent data retrieval, which makes it unsuitable for performing real-time interaction with robots.

Server Side Design

The bulk of the implementation is done on the server side, as illustrated in Figure 6. In the following sections, we first briefly introduce the hardware setup, and then describe the software components in detail.

Hardware Setup

Main Server. There is a dedicated desktop computer (referred to as the main server in later sections) used to host the entire server side software components. The model number is Dell OptiPlex 780, which runs Windows 7 Professional. A smartphone, a wireless IP camera, and a vision-based tracking server (all described in later sections) communicate with the main server via Wi-Fi network, and the robots communicate with it via Bluetooth.

Smartphone. There is also an Nexus One smartphone with Android 2.3.3 which runs an in-house developed Java application that exchanges messages between the main server and the phone.

Vision-based Tracking System. Aside from the dedicated main server mentioned previously, there is also a dedicated visionbased tracking server which connects two Logitech QuickCam® Pro cameras that installed in the ceiling 2.5 meters above the floor and covered an area of 2m×4m. This server tracks the robots’ coordinates in real-time by using a vision tracking method [22] to recognize the markers on top of the robots, and send the coordinates to the main server via Wi-Fi network.

Robots. We built our robots according to the hardware design shown in Figure 6. Both the Roomba and Create are connected with Bluetooth-to-serial converters called RooTooth so that they can communicate with main server over Bluetooth connection. Figure 7 shows a photo of the two robots.

robots fig.7

Figure 7. Spotty the surveillance robot (left) and Tiddy the vacuuming robot (right).

The iRobot Create is augmented with additional hardware components to enable it to work as a surveillance robot, which include a BlueSMiRF Bluetooth module, a servo controller, a two degree of freedom (2-DOF) CrustCrawler S3 Pan/Tilt device, and an Axis 207MW wireless IP camera mounted on the Pan/Tilt device. The BlueSMiRF Bluetooth module enables the servo controller to receive commands from main server. The servo controller, upon receiving the commands, will control the Pan/Tilt device to make the corresponding movement, which will alter the viewing angle of the wireless camera. Images from the wireless camera are then sent to the main server via Wi-Fi.

Software Components

A simplified work flow of the system is described as follows (illustrated in the middle part of Figure 6). Once a message is sent from the client-side, it will be received by its corresponding receiver within the client connection component on the server side. The client connection component will then pass the message along with the information of the sender and client type to the message processor, which further analyzes and converts such input into executable tasks. After a message is processed, the message processor will send feedback to the user via the client connection component. Depending on the urgency of the task, it will either be buffered in the task queue or executed immediately by the task executor. The real-time location information of the robots and objects in the environment are supplied by the visionbased tracking component. To control the robot, the task execution center needs to communicate with the robot controller, which handles specific commands to each robot.

Client Connection 

This component serves as the bridge connecting all clients, hence it is the only component needed to be changed when new social media platforms are introduced into the system. This component allows the server to receive and extract the necessary information from different types of clients, and it is also responsible for sending server-generated messages back to the clients. Currently, this component includes four client connection modules to communicate with the four social media platforms we mentioned. All the messages come from different clients or users are converted into the unified format [user-client-message], and then passed to the message processor.

SMS connection module. Sending and receiving SMS from the server is done by an in-house-developed Java application on an Nexus One smartphone which is connected to standard mobile phone network and co-located with the main server via local wireless network, so that the application can exchange text messages effectively with the main server.

MSN connection module. An open source application MSNPSharp (MSNP18 Release: 3.1.2 Beta by Xih Solutions) is used to develop an MSN client program running on the main server to communicate with the user’s MSN. Currently, only the surveillance robot is equipped with a wireless camera, so the video conferencing capability is only enabled with this robot.

Google Calendar connection module. This module is implemented using the Google calendar data API 2.0. It runs on the main server to communicate with the Google calendar client website. Since the Google calendar website will not inform our server upon users’ update, data are pulled from the client website every 40 seconds.

Facebook connection module. Using the official Facebook Client Library (facebook-0.1.0), we built the Facebook connection module as a Facebook application running on our main server.

This module queries message updates on robots’ Facebook wall every 90 seconds, and responds to users’ requests by posting text and photos on users’ wall.

Message Processor

The message processor is responsible for translating the incoming messages from the client connection component into executable tasks. It analyzes the incoming messages using a simple natural language processing (NLP) method: Each input sentence is first broken into words, and matches against the keywords from the following three categories in descending priorities: tasks (e.g., “vacuum the bedroom now”), general contextual inquiry (e.g., what’s your schedule?), and socialization (e.g., hello). If a sentence contains keywords in more than one category, keywords of the highest-priority category are used.

Task sentences are identified by a few action keywords (e.g., vacuum). Once a sentence is identified as tasks, we will further look for other details of the tasks such as time (e.g., “now”, “5 pm”), location (e.g., “bedroom”), item (e.g., “trash can”), and convert the message into a task object. The currently supported tasks and their corresponding keywords are listed in Table 1.

robots tab.1

Task Executor Center 

The task executor center consists of several parts: a task queue, a task dispatcher, a task executor, and a navigator. Each task in the task queue will be assigned to a task executor by the task dispatcher in a first-in-first-out (FIFO) order.

The navigator is responsible for navigating the robots to some specific locations. Robot and object locations are updated in real-time by a vision-based tracking system (which will be introduced in more detail in subsection 4.2.3). Certain fixed locations are pre-stored in the system map. Based on the robots’ and locations’ coordinates, the navigator computes the routes, and directs the robot controller to move the robots to the required location.

Robot Controller

The robot controller is responsible for communicating with the robots through wireless connection. Since multiple household tasks can be received simultaneously, a queue is built for buffering the tasks. The robot controller retrieves each task from the queue, translates it (such as “take a photo of the window”) into a series of basic movement commands for each robot (such as “move forward”, “stop”, “turn right”), and sends the commands to the robots via Bluetooth connection. The robot controller currently supports iRobot Roomba and iRobot Create by using the roombacomm Java library provided by hackingroomba.com.

Vision-based Tracking System

As described in former subsections, we set up a vision-based tracking system to support robot navigation. The vision tracking component uses proprietary 2-D planar ID-markers as shown in Figure 6 (upper right), which were similar to those in earlier work such as CyberCode [16] and ARTag [4]. A marker consists of a 3×3 black-and-white matrix pattern within a black border surrounded by white margin. Each marker is about 5×5 cm2, in which we managed to recognize stably using two 960×720 resolution ceiling cameras (2.5m high) covering a 2m×4m region on the floor.


We conducted our user studies in order to seek answers for the following questions: (i) Will the users feel comfortable, natural and intuitive to “chat” with robots using the interfaces which are originally designed for interpersonal communication? (ii) What are the factors that affected users’ feeling and decisions in choosing different interfaces? (iii) Do these interfaces complement each other when interacting with domestic robots in varying contexts/scenarios? We first conducted usability experiments in our lab to seek answers for questions (i) and (ii). Then we conducted a multi-day field study, attempting to seek answers for questions (i), (ii) and (iii) in real setting.

Usability Experiment


Twelve participants (6 females and 6 males, aged 19 to 30; mean 24.4, median 24.5) are involved in this experiment. Among them, 9 are from the university and 3 are from the community (working professional). Each received ~10 US dollars for the experiment. Table 2 summarizes their prior experience with the four employed social media platforms.

robots tab.2


We decorated a 4m×2m space in our lab to turn it into a simulated living room and a simulated bedroom, as shown in Figure 8. In Figure 8, the two robots were decorated with colorful paper to make the participants feel familiar with the whole environment. The experimental setup is described in the “System Implementation” section.

robots fig.8

Figure 8. Experiment setup for the usability experiment in the lab.


Upon arrival, each participant was asked to first complete the pre-study questionnaires. During the experiment, each participant performed a task for each of the four interfaces in a random order without any prior training, see Table 3. For each task, the participant was given a 2-minute time limit. If he/she failed to complete the task within this limit, the experimenter will demonstrate the procedure to him/her and ask him/her to complete it again. Upon finishing the experiment, the experimenter will do a post-study questionnaire and interview on each participant. The entire study, including questionnaires and interviews, is performed at one sitting within about 30 minutes.

robots tab.3


In summary, all participants completed the assigned tasks using the specified interface within 2 minutes without help from the experimenter, except one who failed in a Facebook task.

We recorded the learning time and response time taken by each of the twelve participants in each of the four tasks as listed in Table 3. The learning time is defined as the time taken from when the participant was given the task until he/she started typing on the interface; the response time is defined as the time taken from when the participant was given the task until he/she received the first response from the robot. The average learning time and response time for each interface are shown in Figure 9.

Both our observation during the experiments and the data presented in Figure 9 showed that Google Calendar and Facebook took longer learning time than SMS and MSN, which is mainly because SMS and MSN are one-to-one conversation interfaces while the Google Calendar and Facebook interfaces provide a variety of functionalities; hence the participant needed more time to figure out how to start the interaction. As for response time, SMS takes the longest time because it is slower and more troublesome to type on a phone than a computer. The Google Calendar and Facebook also take about one minute to respond as they are not designed for instant communication, so the users need to manually refresh the page to see the feedback. MSN is obviously the most instant and responsive interface among the four since it is a specialized tool for instant communication.

robots fig.9

Figure 9. Task preparation time and completion time taken on the four interfaces in usability experiment.

In the post-study questionnaire, we asked the participants to rank their perception towards the robots across the four interfaces, from a Likert scale of 1 (machine-like) to 7 (lifelike). Results are summarized in Figure 10.

robots fig.10

Figure 10. Average scale of lifelikeness for each interface. 

Figure 10 shows that among the four social media platform interfaces, MSN (5.77) makes the participants feel that the robots appear to be the most lifelike. Facebook (5.08) also made the robots appear to be quite lifelike compared with the Google Calendar interface (3.46) and SMS interface (4.38), whereas the Google Calendar interface made the robots appear the least lifelike. Based on the data shown in Figures 9 and 10, and the comments obtained from the post-study interviews, we identified the following factors that contribute to such variation in users’ perception towards the robots.

Interface Design. Participants commented that both MSN and Facebook made them feel robots to be more lifelike because they contain more “human” elements, such as icons and images representing people on their contact lists with profile pages. They also found both interfaces richer and more entertaining.

Prior Experiences. For most participants, interacting with robots using the MSN and SMS interfaces feels more sociable because they usually use these two interfaces to interact with other people.

Responsive-ness. The feedback speed is also a key factor that influences the participants’ perception to robots. Most users rank MSN higher than Facebook because they feel that robots in MSN respond much faster than in Facebook.

Field Study


Two participants were recruited. Their background information is listed below. Each participant spent 3 days for this study and received an amount of ~64 US dollars.

robots tab.4

Environment and Apparatus 

We rent a multi-room apartment for this field study and deployed the entire system in one of the bedrooms. The bedroom is about 3 meters × 5 meters, while the available space for robots to roam is only about 1.5 meters × 3 meters; therefore two ceiling cameras are enough to cover the entire space for robot navigation.

To hide the supporting equipment from participants’ normal lives, we installed the vision tracking server and main server in an empty wardrobe. The only equipments exposed to the participants are the two robots and their charging docks, and two ceiling cameras, as shown in Figure 11. Please refer to our supplementary video for a visual description of the settings.


The 3-day field study consists of two sessions. The first session was conducted in the first day of the 3 days. During the first day, participants needed to remotely carry out the tasks in Table 3 while working in the office. This session served as a tutorial of the four HRI interfaces. In the evening of the first day, the experimenter interviewed the participants for feedback.

The second session was conducted in the next two days, where the participants were free to use the robots as they like. In addition, the experimenter also sent some requests to the participants through SMS to trigger certain interactions. We carefully picked the time to send these messages so that we can cover more diverse set of scenarios that the participants may encounter (such as on the way to work, walking, sitting next to a computer, having meal with others, talking, etc.). The participants were so busying working that they ignored some of our notifications, therefore the numbers of tasks both participants actually completed are not equal.

After the second session, the experimenter interviewed the participant again to collect their overall feedback.

Results and Discussion

In summary, both participants enjoyed using existing social media platforms to interact with domestic robots. Although the robots’ capability is limited, both participants are convinced about the potential of domestic robot systems.

robots fig.11

Figure 11. System setup in the experimental apartment.

During the second session, Participant 1 chose to use SMS for 50% of the tasks (4 out of 8 tasks) and Facebook for the other 50%, whereas Participant 2 chose to use MSN for 100% of the tasks (7 out of 7). This is coherent with their prior experiences summarized in Table 4, which indicated that Participant 1 uses Facebook much more often than participant 2 (at least once in a week vs. once over a month), while she uses MSN much less often than participant 2 (at least once in a week vs. everyday). This demonstrated that prior experience in using the social media for interpersonal communication have strong influence on the participants’ choice to interact with the robots.

It is also obvious that both participants were trying to keep using the same social media platforms for various types of tasks, no matter whether they were using a phone or a computer, working in the office or walking down the street. Both of them never used Google Calendar at all throughout the second session even when they received request to schedule a repetitive routine task. In the post-study interview, both participants expressed that they do not have the habit of using Google Calendar and it makes them feel that the robots are less interactive.

The above observations seem to be contrary to our design purpose that different interfaces would be used in different contexts/scenarios. However, we argue that this is because the prior experience has greater effect on interface selection than the complementary ability of different platforms, and the Google Calendar interface could be more useful for overviewing scheduled tasks, in particular when the same robot is being deployed to more than one family members (which is not covered in our field study as it is hard to find a whole family to try out the system). More specifically, the users would only switch between those interfaces that they prefer to use. This is supported by Participant 1’s behavior in the second session because she chose Facebook whenever there is available Internet connection and SMS whenever there is no Internet. Participant 2 also explained in the interview that she would probably switch from MSN to SMS if she is driving or when the Internet is not available.

When asked about suggestions, both participants suggested to include more popular clients such as Skype, Google Talk, etc. in our system. More interestingly, participants hope to see more human-like features attached to robots by the interfaces. For instance, Participant 2 said she expects to see the notification

“Tiddy is typing…” in MSN chat window while talking with the robot, although she is aware that the robots are wirelessly communicating with the MSN client rather than physically tapping on keyboard. These suggestions made us believe that using social media platforms to interact with robots is a promising approach to bridge the gap between robots and ordinary users.


This paper explores the application of multiple popular social media platforms to support interaction between human and domestic robots. A working system integrating four complementary social media platforms (SMS, MSN, Google Calendar and Facebook) and two domestic robots (a vacuuming robot and a surveillance robot) was developed to extend our interpersonal communications further to domestic robots. We have conducted lab experiments and multi-day field studies which showed that the approach can contribute to delivering a more user-familiar, flexible, and intuitive interface for common users to interact with robots.

Our approach of leveraging complementary social media platforms for HRI could open up new prospective research directions. Researchers are encouraged to study the longer term effects, e.g., the security and privacy issues, of using the proposed (and other forms of) social media platforms when interacting with robots. With advancement in robot technologies, we envision the potentials of our approach as a practical and natural interaction style with robots, more easily to be adopted by the public.


This research is supported by National University of Singapore Academic Research Fund R-252-000-375-133 and by the Singapore National Research Foundation under its International Research Centre @ Singapore Funding Initiative and administered by the IDM Programme Office.


[1]                 T. T. Ahonen, Time to Confirm some Mobile User Numbers: SMS, MMS, Mobile Internet, M-News. 2011.

[2]                 C. Breazeal. Affective Interaction between Humans and Robots. Advances in Artificial Life, Lect. Notes in Comp. Sci. 2001; 2159: 582-591.

[3]                 A. Faulring and B. A. Myers. Enabling Rich Human-Agent Interaction for a Calendar Scheduling Agent. ACM CHI EA 2005, ACM (2005), 1367-1370.

[4]                 J. Forlizzi. How Robotic Products Become Social Products: An Ethnographic Study of Cleaning in the Home. ACM/IEEE HRI 2007, ACM (2007), 129-136.

[5]                 E. Gilbert and K. Karahalios. Predicting Tie Strength With Social Media. ACM CHI 2009, ACM (2009), 211-220.

[6]                 O. S. Goh, C. C. Fung, A. Depickere, and K. W. Wong. An Analysis of Man-Machine Interaction in Instant Messenger. Advances in Comm. Sys. and Electrical Engr. 2008. 197-210.

[7]                 C. Guo and E. Sharlin. Exploring the Use of Tangible User Interfaces for Human-Robot Interaction: a Comparative Study. ACM CHI 2008, ACM (2008), 121-130.

[8]                 C. Guo, J. E. Young, and E. Sharlin. Touch and Toys: New Techniques for Interaction with a Remote Group of Robots. ACM CHI 2009, ACM (2009), 491-500.

[9]                 K. Ishii, S. Zhao, M. Inami, T. Igarashi, and M. Imai. Designing Laser Gesture Interface for Robot Control. INTERACTION, Lect. Notes in Comp. Sci. 2009; 5727: 479-492.

[10]              K. Kawamura, R. T. Packa, M. Bishaya, and M. Iskarous. Design Philosophy for Service Robots. Rob. and Aut. Sys. 1996; 18(1-2): 109-116.

[11]              M. S. H. Khiyal, A. Khan, and E. Shehzadi. SMS Based Wireless Home Appliance Control System (HACS) for Automating Appliances and Security. J. Issues in Informing Sci. & Info. Tech. 2009; 6: 887-894.

[12]              H. Kim, H. Lee, S. Chung, and C. Kim. User-centered Approach to Path Planning of Cleaning Robots: Analyzing User’s Cleaning Behavior. ACM/IEEE HRI 2007, ACM (2007), 373-380. [13] N. Linder and P. Maes. LuminAR: Portable Robotic Augmented Reality Interface Design and Prototype. ACM Symposium on User Interface Software and Technology 2010, ACM (2010), 395-396.

[14]              N. Mavridis, C. Datta, S. Emami, A. Tanoto, C. BenAbdelkader, and T. Rabie. FaceBots: Robots Utilizing and Publishing Social Information in Facebook. ACM/IEEE HRI 2009, ACM (2009), 273-274.

[15]              P. Mistry, K. Ishii, M. Inami, and T. Igarashi. BlinkBot – Look at, Blink and Move. ACM Symposium on User Interface Software and Technology 2010, ACM (2010), 397-398.

[16]              K. Okada, T. Ogura, A. Haneda, J. Fujimoto, F. Gravot, and M. Inaba. Humanoid Motion Generation System on HRP2-JSK for Daily Life Environment. IEEE Intl. Conf. on Mechatronics & Automation 2005, IEEE Press (2005), 1772-1777.

[17]              P. Roßler and U. D. Hanebeck. Telepresence Techniques for Exception Handling in Household Robots. IEEE Intl. Conf. on Systems, Man, and Cybernetics 2004, IEEE Press (2004), 53-58. [18] D. Sakamoto, K. Honda, M. Inami, and T. Igarashi. Sketch and Run: A Stroke-based Interface for Home Robots. ACM CHI 2009, ACM (2009), 197-200.

[19]              D. J. Sim, X. Ma, S Zhao, J. T. Khoo, S. L. Bay, Z. Jiang  Farmer’s Tale: A Facebook Game to Promote Volunteerism.. ACM CHI 2011, ACM (2011), 581-584.

[20]              Y. Sugiura, D. Sakamoto, A. Withana, M. Inami, and T. Igarashi. Cooking with Robots: Designing a Household System Working in Open Environments. ACM CHI 2010, ACM (2010), 2427-2430.

[21]              J.-Y. Sung, L. Guo, R. E. Grinter, and H. I. Christensen. “My Roomba is Rambo”: Intimate Home Appliances. UbiComp 2007, Springer-Verlag (2007), 145-162.

[22]              S. Zhao, K. Nakamura, K. Ishii, and T. Igarashi. Magic Cards: A Paper Tag Interface for Implicit Robot Control. ACM CHI 2009, ACM (2009), 173-182.

Written by Shengdong Zhao

Shen is an Associate Professor in the Computer Science Department, National University of Singapore (NUS). He is the founding director of the NUS-HCI Lab, specializing in research and innovation in the area of human computer interaction.