I attended TEI’10 conference in Boston and presented there our work and demo of SensorComix (Pdf) , a research project at NRC I did with my colleagues Vidya Setlur, Tim Sohn, and Hiroshi Horii.
In SensorComix, we leverage and build upon a previous project called Nokia Mashups where SMS conversations are displayed with rich graphics representative of who’s having the conversation, the place where the communicants are located, and what they’re talking about.
The idea in SensorComix is twofold: add another dimension of expressiveness to SMS conversations, and add playfulness.
I personally have tons of stories about SMS mishaps and miscommunication, and from what I’ve seen in our projects and user trials, I’m surely not the only one. For example, a girl sends an SMS to her boyfriend asking him about something, he replies ‘yes’. The girl sends quickly another message telling him how rude he is. The boyfriend sends back that he’s in class and has no time to elaborate. Another example with yet another couple communicating, the boyfriend asks his girlfriend if she went to an evening work event, then quickly in another SMS, asks her if he’s going to see her in the evening. He gets a reply of ‘No too tired’ and assumes it’s the answer to the second message.
SMS messages are a strange way of communicating. They’re short, and because the input methods are limited to small physical keyboards, or tiny keypads, or onscreen keyboards, people are limited in how much they can write, and how clearly they can convey all the nuances in their message. SMS are sent from mobile phones, and have become a preferred way people communicate whenever and wherever they are. But often people are busy, they’re going places, they’re in meetings or in class, with other people, and they may not have time to write clear and unambiguous messages. SMS are asynchronous but nonetheless there is some expectation that the other person is going to reply, sometimes quickly, especially when SMS are between people in a close relationship or when the persons are suppose to meet or plan something together.
In SensorComix, the user writes the SMS and can use one of four different gestures that add different graphical elements to convey an emotion associated to that gesture. For example, rubbing the phone will make little hearts pop up. Shaking the phone will make a fire and flames appear. The four gestures we’ve included are those:
- Rubbing, to convey affection
- Shaking, to convey impatience or anger
- Tapping, to poke
- Knocking, to get attention
With SensorComix, one can write the message, and quickly add an emotion to it with a simple gesture.
It’s best seen on the video:
Details on the demo implementation
We used Nokia tablet devices N810, a SHAKE SK6 sensor and a MAC laptop to create the prototype. The demo does not send SMS or MMS, but is hacked together to let the user takes an N810, see a comic with preloaded images (that one can change), write a message, enter gestures on the sensor which will display animated graphical elements on the comic, and send the message. The other N810 will receive the message and display the message with the comic, message and animated elements, and one can then reply to the message.
The messaging application showing the comic and the input text for the message is actually an HTML web page, which is served by the MAC laptop. But the actual comic and the animation that appear when entering a gesture are done with a Flash/Flex component embedded in the Web page. The laptop runs a custom Python HTTP server which does 3 things:
- Serve Web pages to the clients. There are 2 types of web pages, one for sending a message, one for receiving a message. The server also provides all the images displayed in the comic.
- Connect to the SK6 sensor via Bluetooth. The server reads the data from the sensor and performs the gesture recognition.
- Communicate via permanent sockets with the Flash/Flex component embedded in the Web pages. When the Web page for sending a message is displayed, the Flash/Flex component opens a socket with the server, registers as ‘sender’ and will receive the information to update the comic: the message being typed in the HTML input textarea, and which animated elements to display which should correspond to the gesture recognized by the server from the sensor data.
To run the demo, I create a ‘Network’ on the laptop. The server runs on port 80 on the laptop and I open the browser on the N810 to the URL http://[IP address of the laptop]. This configuration is somewhat stable but not highly robust. I could demo it without too much trouble for a couple of hours at TEI. During the development, there were few hiccups to circumvent with the Flash/Flex – Python socket communication. The advantage is of course that you don’t need the N810 to develop the code, you can just run the Flash/Flex in your laptop browser all the same.
For the gesture recognition, I wrote some custom simple code to identify the 4 gestures. Only 2 different sensors out of the several provided on the SK6 are used: the touch sensor (rubbing and tapping) and accelerometer (knocking and shaking). The recognition is robust enough that I don’t recalibrate the SK6 at all, but the knocking gesture was the one that was not recognized well. I should just rewrite the code. We did the project before this SHAKE Drivers Google project started and haven’t had a chance to look into it.
There is tons of work going on at Nokia Design on gestures. See Gesture Design and the video here.
