YouMove: (Or - the most fun you can have in pyjamas)This was a really cool project I did with Tovi, Justin and George at Autodesk Research. We built a giant interactive mirror that could be used to train people how to perform new movements. I got to use LEGO, servo motors, build a big fort out of bed sheets and had people dress up in pyjamas and learn ballet. It wasn't just fun, I'm really proud of the output. The recording software is flexible and simple, we made some real contributions to interacting with augmented reality mirrors, and the training software actually works. It used knowledge from coaches (ballet and yoga instructors) as well as from the motor learning literature to provide proper feedback.
Thermal printer photo boothWhat's more fun than a pixelated version of your face printed in terrible quality on a thermal receipt printer? Nothing. By throwing a webcam, button and thermal printer on a Raspberry Pi, I built a little photo booth that prints you your picture along with an ID so you can retrieve a higher-res/color version later. Hopefully I'll have time to write up a little how-to later, but the short version is: use 'streamer' to capture from the webcam and use info from this blog to poke at the thermal printer.
Artificial Human ProjectThe goal of this project was to fully simulate a human (as judged by a PIR motion sensor). To prevent lights from automatically turning off when Michelle didn't move for a few minutes in her office, I built a little duct-tape flag on an Arduino-driven servo motor that swung around every 15 minutes. While this wouldn't pass a turing test, it's a totally convincing person to the motion sensor.
Gesture LearningGestures have the potential to be an efficient and expressive means of communication. They're well suited to mobile phones and other touch-screens that lack a separate physical input space. Current gestures are pretty simple, and pretty limited. To reach their full potential, I think gestures are going to have to become more complex and require learning (both in learning the pairing, and the motor performance). The first study I ran on this topic showed how the type of guide used in learning can help or hinder learning, and suggested new methods for evaluating gesture learning.
Scanpath AnalysisAnalyzing eye movements is a pretty effective way to understand where people are focussing their attention. It's also useful for understanding how the brain works, and how we perceive and understand the world. However, it's pretty difficult to compare people's eye movements, or even the eye movements of the same person looking at the same scene. Working with people at the BAR lab at UBC, I implemented some methods to compare eye movements, recorded data using the Dikabilis system, and some other eye related things.
Tabletop Therapy EvaluationThis is a project done with Michelle to compare interactive surface rehabilitation tasks to traditional tasks. We suited up some participants in motion capture gear, and stuck some EMG electrodes on their dominant arm to record how they moved. We then had them use a few of the games Michelle had built for surface, as well as do some static 'touch the sticker' type tasks that are currently used in therapy. We found that the design of the activity was far more important than the use of technology when considering movement patterns. The work is published at CHI (paper, poster).
ShippyThis game had simple mechanics, but had some fairly complex (Python) coding behind it. It was built for Psychology researchers at UBC who wanted to study participant behaviour while interacting with video games. To be useful, it had to be very customizable and easy to use (adding lag to the controls, switching buttons, change direction of play, speed, difficulty, etc). Each parameter could be changed at specific points in time during gameplay as well, e.g., at one minute in, swap the left and right keys and slow it down. It also recorded synchronized video of the participant and the game screen.
Kinect Physics TableThis project seems a bit dated already (done for my signal processing class in Nov. 2010) because of how quickly development with the Kinect has taken off. The system had an FTIR table with a Kinect pointing at the table surface from above. Users could put arbitrary physical objects on the surface, and from the Kinect's depth map a contour was extracted. The contours of the objects were fed into a Box2D simulation, allowing virtual objects to physically interact with the real-world objects. It was kind of like a really coarse virtual wind tunnel.
QuickSort Outreach AppThis is a project I worked on with Michelle for the Canadian Information Processing Society. The idea was to create something that would help get kids interested in computing science. What we ended up with is a touch-screen application that runs on an all-in-one computer that is meant to be displayed at conferences/open houses etc. The application challenges users to compete with a computer to see if they can sort objects faster than a computer. The user can choose different speeds, data sets, and sorting algorithms, and each choice has some witty-computer-banter along with some interesting facts. I did a lot of work on the back-end of this, writing the sorting algorithms (had to write my own stack/threading to fit into the Flex Timer framework). It was a nice break from the everyday reading/MATLAB grind.
ARM TrainerBefore receiving their custom myo-electric (muscle-signal-controlled) prosthetic, amputees are encouraged to train their muscles in preparation. Current methods of pre-training are far from ideal, so I developed a system that aims improve this training process. More details can be found in the ICDVRAT publication (which won best student paper!).
Evaluation of Surgical SkillFor my Master's thesis project I developed a system to analyze and quantify the skill of a surgeon. Motion sensors on the surgeon and force/torque sensors on the instrument record the surgeon's movements and provide a way to compare experts and novices. After designing the system, running the experiments, and staring at graphs for months, we finally had something. A couple of new methods to evaluate skill based on the curvature of the trajectory and the total energy, as well as a novel system design are fully detailed in my thesis (there's also some pretty graphs).
Bilateral Control with Four PHaNTOM OmnisWorking with Minh-tu Pham, a visiting professor from France, I helped develop the software to make two pairs haptic devices mirror each other. It was using fairly simple PID control, with some low-pass filtering to remove oscillations. The purpose was to see if we could improve novice/expert surgical training by having novices directly feel the expert's hand movements, and vice-versa.
Virtual Equine Assisted TherapyThis was a very fun project that died too soon. The idea was to use a robotic platform and a bunch of fancy VR hardware to simulate virtual horseback riding for therapeutic purposes. The idea would be that the robotic environment would be much more controllable, allowing a wider range of patients to benefit from EAT and to better study the therapy. Due to time, equipment, and a handful of other factors, we couldn't move forward with this project, but I still think it's very cool.
Wiihabilitation Games and AssessmentThis is work done with Michelle on developing rehabilitation-specific games and applications using Nintendo Wii peripherals. Therapists are currently using the Nintendo Wii for physical/occupational therapy, but the commercial games are not well suited for this purpose. Our system has a number of games that have configurable difficulty, progress monitoring, feedback, and other things useful for therapy. More details on the system can be found here.
Olfactory Interface for Virtual RealityOur virtual worlds are not smelly enough. To correct that problem, a WISEST 2010 student developed a low cost olfactory interface to deliver scents to the user. Using PC fans controlled by a Phidget relay interface kit, a program can turn the device to turn various scents 'on' and 'off'. The student also developed a large virtual environment using Google sketchup, and Vizard, and also ran some experiments investigating the application of smell illusions in virtual environments. This project is the start of work to explore the limits of visual capture in virtual reality.
e-MonkeyThis was a WISEST 2010 project to construct an interactive toy that I helped develop/supervise with Michelle. A Build-a-bear monkey was unstuffed, and filled with sensors, lights, actuators, etc. The electrical widgets all run to a couple of Arduinos that control the lil guy. This project helped the WISEST student (and Michelle and I) learn about Arduino's, as well as HCI and a few electrical bits. The intended use of the Monk-e will be disclosed after publication!
Interactive FloorLow-resolution interactive floors are pretty common now, using fancy IR cameras and other somewhat-esoteric hardware. To increase the potential of these devices, we wanted a higher resolution, cheaper version. Working with a team of undergraduates in the HCI course, and later more closely with Timothy, we developed and deployed a a low-cost, high resolution system that can track foot movements in 3D. The system uses off the shelf wiimotes and infrared emitters to track foot position. Will add more info as it becomes disclosable!
Emotional SpeechFor a linguistics course I wrote some software that analyzed the emotional content in speech. Based on the pitch, volume and speech rate, the system could identify the arousal and valence levels in the speech with around 12% error. Not quite good enough for publication, but I'm workin' on it. I really liked this project, as it let me play in affective computing a bit, an area I'll always find very interesting.
Morphing Point-based ModelsInstead of using those pesky little polygons to represent 3D objects, we can use a cloud of points instead. For my graphics course project, I wrote a program that takes two 3D models represented as point clouds and morphs one into the other. Below is a bent object (left) morphing into the original (right), which would be useful in analyzing mechanical deformations and other fun things.