Sensor Comix
This is a follow-up project to the SMS Comics in which we use sensors to detect gestures and create an animatable SMS comic.
The prototype of Sensor Comics works on a Nokia N810 or N900 with a separate sensor box. The user interface shows a template of an SMS comic, and the user writes a text message in the bubble. The user can then use one of four gestures to personalize the text message further: a shaking gesture to convey impatience or anger, a tap with the phone to create a ‘poke’ message, a knock to get attention, and a rubbing gesture on the phone to convey affection.
In this project too, I developed all pieces except for the Flash component that renders and animate the comic itself. I developed the algorithm to recognize the four gestures in python. The sensor box is actually connected to a local server through bluetooth. That server handles the communication between the N810 devices (sender and receiver) and the sensor boxes. The client application is programmed with PyQt, and integrates the Flash component that communicates itself with the local server via sockets. The setup is a little tricky, but not atypical for a hardware-software research prototype – you just need to find a way to make the pieces work together.
Publication:
V. Setlur, A. Battestini, T. Sohn, H. Horii, “Using Gestures on Mobile Phones to Create SMS Comics”, International Conference on Tangible, Embedded and Embodied Interaction (TEI), 2010. (slides)
I blogged about the project here.
See the project in action in this video.
Two examples of Sensor Comix: in the first one, animated hearts popup after the user 'rubs' the phone to convey affection, and in the second one, a ball is throws to the receiver to get attention.
The user interface of Sensor Comix to write a text message on a N810.
The sensor box is attached to the phone but is actually connected to a local server via Bluetooth. The gesture recognition happens on that server and the gesture codes are sent back to the N810.
Leave a Reply