It might not be until something seems off that you realize you’re paying attention to the tiniest twitches and scrunches of another person’s face.
Humans are hardwired to catalogue and interpret minuscule clues about emotion this way. We find meaning in broken eye contact, a passing brow crinkle, the pause that goes on a beat too long.
But how good are people, generally, at discerning how others feel based on the faces they’re making?
In 1872, Charles Darwin laid out a case for the universality of key facial expressions in his book, The Expression of Emotions in Man and Animals. “Of all parts of the body,” Darwin wrote, “the face is most considered and regarded, as is natural from its being the chief seat of expression and the source of the voice... The face, therefore, will have been subjected during many generations to much closer and more earnest self-attention than any other part of the body.”
Scientists have found lots of evidence to back up the idea that some facial expressions are innate, but there are many layers of social and cultural complexities to consider. One 2012 study found that Westerners demonstrated distinct facial expressions in conjunction with six main emotional states (happiness, sadness, surprise, fear, disgust, and anger), whereas Easterners were more likely to use dynamic eye expressions.
Today, machine learning is giving scientists a new way to interpret the subtle movements of the face. Researchers at Carnegie Mellon, for instance, are using a multi-modal algorithm to analyze facial expression based on 68 separate points on the face, including the eyebrows, eye corners, mouth, and nose.
A new system called MultiSense can also track in real-time a person’s head position, the direction of that person’s gaze, and the person’s body orientation. This level of detail can be surprisingly revealing. Looking at what a person’s nose and eyebrows are doing can differentiate between a happy smile and an angry smile, for example, or a smile that’s triggered by a social situation rather than an actual emotion. “So a lot of time what we see [on a person’s face] is the social norm,” says Louis-Philippe Morency, an assistant professor in Carnegie Mellon’s School of Computer Science. “Someone is smiling to smile back, so the dynamics of that smile would be different because of different emotional states and social states.”
Morency and his colleagues are particularly interested in using machine learning to trace connections between facial expressions and emotional state among depressed people. And what they’ve found so far is unexpected. For one thing, depressed people and non-depressed people smile with the same frequency. But the kinds of smiling they did were different. So while depressed people smiled as often as non-depressed people, the depressed people’s smiles lasted for a shorter period of time. (In addition to tracking smile duration, the sensor platform tracks smile intensity on a 100-point scale.)
There was also a pronounced gender difference in facial expressions among depressed people. In one University of Southern California study, Morency and three other researchers found that depressed men frown more often then non-depressed men, but observed the opposite effect among women: Depressed women frowned less frequently than non-depressed women.
“The really interesting next part,” Morency says, “is to see how [these findings are] aligned with social norms.” For instance, many women have had the experience of being told to smile. “Is it related to culture? Is it local? National? International? Or is there even another factor—social, cultural, physiological—that we don’t know yet?”
The implications of this technology, from a health-care perspective, could mean machine learning will help human doctors track their patients’s well being over time—and do so using objective, quantifiable data. In the short term, Morency believes MultiSense’s abilities are on par with an expert clinician. (“I think expert clinicians do see these cues,” he said. “They may not even realize it.”)
There are other implications for this kind of technology. It’s no surprise, for instance, that the U.S. military has funded much of the research into reading facial expressions. The Defense Department is interested in using facial-recognition platforms for treating people suffering from PTSD. It also has a longstanding goal of using such sensors as a way to understand and predict behaviors. Decades ago, the Department of Defense began amassing a huge database of facial expressions for this purpose.
Morency is excited by the surge in research, but he also urges caution, stressing that these tools are best suited to ongoing care for a patient with depression—and not as a diagnostic tool. Humans, it turns out, already pick up a number of these cues, but they do so in a subtle, less quantifiable way. And unlike machines, they balance their judgments against ethical considerations, about patient care, for instance.
“I personally would much prefer this technology as a decision-support tool, but not as a decision-making tool,” Morency said. “When we start talking about decision-making, this brings a large number of ethical issues that need to be addressed.”