Student modifies Microsoft Kinect
to recognize game players’ moods © Aug. 10, 2012, Norwich University Office of Communications

Computer science student Lauren Wyatt displays the grid that helps her map a face and detect subtle movements called micro expressions.

photo by Jordan Silverman, staffComputer science student Lauren Wyatt displays the grid that helps her map a face and detect subtle movements called micro expressions.

It was mid afternoon in the student center of Norwich University’s Northfield, Vt., campus, and the light was not working in Lauren Wyatt’s favor.

Moving her laptop to a different couch in the second-floor lounge, she re-adjusted the Kinect device attached to the computer and sat back to wait for a new window to pop up on the screen.

The window opened to a live video image of herself staring back. Nothing happened for a few seconds, but then a grid of white lines appeared over her face.

“OK, there we go,” she said, adjusting the computer to find better contrast.

The lines and shapes formed by the grid moved with her, bending and shifting as she spoke or turned her head. Wyatt, a 20-year-old from Central Massachusetts about to enter her third year in the computer science program, was spending the summer modifying a popular hands-free videogame controller to pick up information about the user most human beings would fail to recognize. She thinks the Kinect can detect if a player is lying, excited or even bored.

“Obviously, it sees that I’m talking,” said Wyatt, who obtained a fellowship from Norwich to stay on campus for 10 weeks to conduct her research. “The goal is to recognize expressions.”

The Kinect is a motion-sensing input device that can track the movement of multiple players, allowing them to play videogames on computers and the Xbox platform. It utilizes multiple cameras, a depth sensor and microphones. Microsoft, which released the Kinect in November 2010, also released software development tools and encourages users to create their own applications for the system.

The key to unlocking the users’ emotions, she said, is to train the software to recognize micro expressions—subtle twitches and facial movements that indicate what is going on beneath the surface. For example, a very brief smile is a reliable indication a person is lying, similar to a rapid glance upward and to the left, as it suggests a person is drawing on the creative portion of the brain.

“You can’t control [micro expressions], unless you’re really trained to,” said Wyatt, adding that less subtle expressions, such as a broad smile, are much easier to fake.

The cameras on the Kinect are sensitive enough to recognize the most fleeting of facial movements, she said, and she’s only “scratched the surface,” of the information she believes a computer will be able to gather from facial movement. But that doesn’t mean it is easy to make a machine recognize and interpret movement.

“It’s a lot more complicated than I thought it would be,” she said.

Each of the areas of a face where emotion is expressed, such as the corners of the mouth, between the eyes, the eyebrows and the nose, must all be mapped. She has learned the C# computer language in order to write code for the software to recognize tiny differences in the 30 images the cameras on the Kinect capture each second.

The sociological implications of using computers to observe emotions are, “interesting and a little bit scary,” said Wyatt, who became interested in micro expressions after writing a paper about them for a psychology class. Her intentions are fairly benign, however. A fan of videogames, she believes expression recognition may one day add a new dimension to gaming.

“If someone’s playing a videogame and they look bored, they can make it more challenging,” she said.

Jeremy Hansen, a computer science professor who served as advisor on this project, said people have used cameras and other technologies to record expressions. The interesting thing Wyatt is attempting is to modify a commercial, off-the-shelf product to do something it was not designed for. In the beginning, the Kinect was sold for game playing alone, but “homebrew hackers,” quickly began to modify the device and explore its capabilities. Catching on to the trend, Microsoft released the software tool kit a year or so after the Kinect went on the market.

“They’re really encouraging it now,” he said.

Wyatt said some technical problems prevented her from getting as far as she would have liked, but the progress she made and skills she has learned have been gratifying. She will continue to work on her project when classes start up again in the fall.

Hansen added that a project of this depth really requires a 10-week summer research period. Otherwise, a student has so many other commitments that he or she just doesn’t have the time to follow their muse.

“It’s just not enough [time] to really immerse yourself in it,” said Hansen, who was advisor to four summer research undertakings in 2012. “That’s why these summer research projects are great!”