May 22, 2013
As much time as we spend with our cell phones and laptops and tablets, it’s still pretty much a one-way relationship. We act, they respond. Sure, you can carry on a conversation with Siri on your iPhone, and while she is quick, it hardly qualifies as playful bantering. You ask questions, she gives answers.
But what if these devices could really read our emotions? What if they could interpret every little gesture, every facial cue so that they can gauge our feelings as well as–maybe better than–our best friends? And then they respond, not with information, but what might pass for empathy.
We’re not there yet, but we’re quickly moving in that direction, driven by a field of science known as affective computing. It’s built around software that can measure, interpret and react to human feelings. This might involve capturing your face on camera and then applying algorithms to every aspect of your expressions to try to make sense of each smirk and chin rub. Or it might involve reading your level of annoyance or pleasure by tracking how fast or with how much force you tap out a text or whether you use emoticons. And if you seem too agitated–or drunk–you could get a message suggesting that you might want to hold off pressing the send icon.
Seeing how difficult it is for us humans to make sense of other humans, this notion of programming machines to read our feelings is no small challenge. But it’s picking up speed, as scientists sharpen their focus on teaching devices emotional intelligence.
Every move you make
One of the better examples of how affective computing can work is the approach of a company called, appropriately, Affectiva. It records expressions and then, using proprietary algorithms, scrutinizes facial cues, tapping into a database of almost 300 million frames of elements of human faces. The software has been refined to the point where it can associate various combinations of those elements with different emotions.
When it was developed at M.I.T’s Media Lab by two scientists, Rosalind Picard and Rana el Kaliouby, the software, known as Affdex, was designed with the purpose of helping autistic children communicate better. But it clearly had loads of potential in the business world, and so M.I.T. spun the project off into a private company. It has since raised $21 million from investors.
So how is Affdex being used? Most often, it’s watching people watching commercials. it records people as they view ads on their computers–don’t worry, you need to opt in for this–and then, based on its database of facial cues, evaluates how the viewers feel about what they’ve seen. And the software doesn’t provide just an overall positive or negative verdict; it breaks down the viewers’ reactions second by second, which enables advertisers to identify, with more precision than ever before, what works in a commercial and what doesn’t.
It also is able to see that while people say one thing, their faces can say another. During an interview with the Huffington Post, el Kaliouby gave the example of the response to an ad for body lotion that aired in India. During the commercial, a husband playfully touches his wife’s exposed stomach. Afterwards, a number of women who had watched it said they found that scene offensive. But, according to el Kaliouby, the videos of the viewers showed that ever one of the women responded to the scene with what she called an “enjoyment smile.”
She sees opportunities beyond the world of advertising. Smart TVs could be that much smarter about what kind of programs we like if they’re able to develop a memory bank of our facial expressions. And politicians would be able to get real-time reactions to each line they utter during a debate and be able to adapt their messages on the fly. Plus, says el Kaliouby, there could be health applications. She says it’s possible to read a person’s heart rate with a webcam by analyzing the blood flow in his or her face.
“Imagine having a camera on all the time monitoring your heart rate,” she told the Huffington Post, “so that it can tell you if something’s wrong, if you need to get more fit, or if you’re furrowing your brow all the time and need to relax.”
So what do you think, creepy or cool?
Here are five other ways machines are reacting to human emotions:
- And how was my day?: Researchers at the University of Cambridge have developed an Android mobile app that monitors a person’s behavior throughout the day, using incoming calls and texts, plus social media posts to track their mood. The app, called “Emotion Sense,” is designed to create a “journey of discovery,” allowing users to have a digital record of the peaks and valleys of their daily lives. The data can be stored and used for therapy sessions.
- And this is me after the third cup of coffee: Then there’s Xpression, another mood-tracking app created by a British company called EI Technologies. Instead of relying on people in therapy to keep diaries of their mood shifts, the app listens for changes in a person’s voice to determine if they are in one of five emotional states: calm, happy, sad, angry or anxious/frightened. It then keeps a list of a person’s moods and when they change. And, if the person desires, this record can automatically be sent to a therapist at the end of every day.
- What if you just hate typing on a phone? : Scientists at Samsung are working on software that will gauge your frame of mind by how you type out your tweets on your smartphone. By analyzing how fast you type, how much the phone shakes, how often you backspace mistakes, and how many emoticons you use, the phone should be able to determine if you’re angry, surprised, happy, sad, fearful, or disgusted. And based on what conclusion it draws, it could include with your tweet the appropriate emoticon to tip off your followers to your state of mind.
- Just don’t invite your friends over to watch: Using a sensor worn on the wrist and a smartphone camera worn around the neck, researchers at M.I.T. have created a “lifelogging” system that collects images and data designed to show a person which events represented their emotional highs and lows. The system, called Inside-Out, includes a bio-sensor in a wristband that tracks heightened emotions through electrical charges in the skin while the smartphone tracks the person’s location and takes several photos a minute. Then, at the end of the day, the user can view their experiences, along with all the sensor data.
- Your brow says you have issues: This probably was inevitable. Researchers at the University of Southern California have created a robotic therapist that not only is programmed to encourage patients with well-timed “Uh-huhs,” but also is expert, using motion sensors and voice analysis, at interpreting a patient’s every gesture and voice inflection during a therapy session.
Video bonus: Want to see how bizarre this trend of devices reading human emotions can get? Check out this promotion of Tailly, a mechanical tail that picks up your level of excitement by tracking your heart rate and then wags appropriately.
More from Smithsonian.com
May 17, 2013
When art meets neuroscience, strange things happen.
Consider the Museum of Scientifically Accurate Fabric Brain Art in Oregon which features rugs and knitting based on a brain scan motif. Or the neuroscientist at the University of Nevada-Reno who scanned the brain of a portrait artist while he drew a picture of a face.
And then there’s the ongoing war of words between scientists who think it’s possible to use analysis of brain activity to define beauty–or even art–and their critics who argue that it’s absurd to try to make sense of something so interpretive and contextual by tying it to biology and the behavior of neurons.
Beauty and the brain
On one side you have the likes of Semir Zeki, who heads a research center called the Institute of Neuroesthetics at London’s University College. A few years ago he started studying what happens in a person’s brain when they look at a painting or listen to a piece of music they find beautiful. He looked at the flip side, too–what goes on in there when something strikes us as ugly.
What he found is that when his study’s subjects experienced a piece of art or music they described as beautiful, their medial orbito-frontal cortex–the part of the brain just behind the eyes–”lit up” in brain scans. Art they found ugly stimulated their motor cortex instead. Zeki also discovered that whether the beauty came through their ears, in music, or their eyes, in art, the brain’s response was the same–it had increased blood flow to what’s known as its pleasure center. Beauty gave the brains a dopamine reward.
Zeki doesn’t go so far as to suggest that the essence of art can be captured in a brain scan. He insists his research really isn’t about explaining what art is, but rather what our neurons’ response to it can tell us about how brains work. But if, in the process, we learn about common characteristics in things our brains find beautiful, his thinking goes, what harm is there in that?
Beware of brain rules?
Plenty, potentially, responds the critics’ chorus. Writing recently in the journal Nature, Philip Ball makes the point that this line of research ultimately could lead to rule-making about beauty, to “creating criteria of right or wrong, either in the art itself or in individual reactions to it.” It conceivably could devolve to “scientific” formulas for beauty, guidelines for what, in music or art or literature, gets the dopamine flowing.
Although it is worth knowing that musical ‘chills’ are neurologically akin to the responses invoked by sex or drugs, an approach that cannot distinguish Bach from barbiturates is surely limited.
Others, such as University of California philosophy professor Alva Noe, suggest that to this point at least, brain science is too limiting in what it can reveal, that it focuses more on beauty as shaped by people’s preferences, as opposed to addressing the big questions, such as “Why does art move us?” and “Why does art matter?”
And he wonders if a science built around analyzing events in an individual’s brain can ever answer them. As he wrote in the New York Times:
…there can be nothing like a settled, once-and-for-all account of what art is, just as there can be no all-purpose account of what happens when people communicate or when they laugh together. Art, even for those who make it and love it, is always a question, a problem for itself. What is art? The question must arise, but it allows no definitive answer.
Fad or fortune?
So what of neuroaesthetics? Is it just another part of the “neuro” wave, where brain scans are being billed as neurological Rosetta Stones that proponents claim can explain or even predict behavior–from who’s likely to commit crimes to why people make financial decisions to who’s going to gain weight in the next six months.
More jaded souls have suggested that neuroaesthetics and its bulky cousin, neurohumanities, are attempts to capture enough scientific sheen to attract research money back to liberal arts. Alissa Quart, writing in The Nation earlier this month, cut to the chase:
Neurohumanities offers a way to tap the popular enthusiasm for science and, in part, gin up more funding for humanities. It may also be a bid to give more authority to disciplines that are more qualitative and thus are construed, in today’s scientized and digitalized world, as less desirable or powerful.
Samir Zeki, of course, believes this is about much more than research grants. He really isn’t sure where neuroaesthetics will lead, but he’s convinced that only by “understanding the neural laws,” as he puts it, can we begin to make sense of morality, religion and yes, art.
Here’s some of the latest news about brain scans:
- I see your pain: A study published last month in the New England Journal of Medicine reported that scientists were not only able to “see” pain on brain scans, but also could measure its intensity and tell if a drug was helping to ease it.
- Don’t blame me, it’s my brain that hates calculus: A research team at Stanford University School of Medicine concluded that the size and connectivity of a child’s hippocampus, a brain area that is important for memory, is the key factor in how quickly he or she can learn math.
- There lies madness Researchers at Cambridge University in the U.K. say they will scan the brains of 300 teenagers and track how their brains evolve as they age. One thing the scientists want to see is how the brain’s wiring changes as teenagers become less impulsive.
- Trouble brewing: Brain scans may even be able to help detect if a recovering alcoholic is about to fall off the wagon. A study published in the journal JAMA Psychiatry contends that alcoholics with abnormal activity in areas of the brain that control emotions and desires are eight times more likely to relapse and start drink heavily.
- Robots are people, too: And finally, German researchers say that based on their analysis of brain scans of subjects in a study, people reacted just as strongly to scenes of robots being treated kindly or being abused as they did to humans getting the same treatments.
Video bonus: Samir Zeki explains, in this TED talk, why he’s sure beauty is in the brain of the beholder.
Video bonus bonus: Brain scans can be funny, in a bizarre Japanese humor kind of way. And no, I have no idea why the men in this video are all dressed as female nurses.
More from Smithsonian.com
May 10, 2013
To be honest, I’ve never associated motherhood with science. I assume this has everything to do with the fact that I’m one of eight kids, and while I’m sure we were a study in chaos theory, my mother didn’t have much time to nail the concept and work it into bedtime stories.
That said, moms remain a subject of scientific inquiry because, no matter how constant they may seem to us, they’re always changing to keep up with the times.
Here then are 10 recent studies or surveys that give a bit more insight into the institution of 21st century moms.
1) Have I got a story for you: According to a study published recently in the journal Sex Roles, moms are better than dads at telling stories and reminiscing with their kids, and that helps children develop their emotional skills. The researchers observed that moms tended to include more emotional terms in their stories and were more likely to then explain them to their children.
2) But how many of the answers were “Because I said so”: A survey of 1,000 moms in the United Kingdom found that the typical mother answers up to 300 questions a day from their kids. Four-year-old girls are the most inquisitive, averaging a fresh question about every two minutes. The most questions are asked during meals–an average of 11–followed by shopping trips–10 questions–and bedtime–nine questions.
3) That magic touch: The skin-to-skin touch of a mother can make a big difference in helping preemies or other at-risk babies deal with the pain and stress of injections. Researchers determined that the touch of a father or an unrelated women can also help lower the stress of an at-risk baby, but neither had quite the soothing effect of physical contact with the child’s mother.
4) Even mom spit is special: A recent article in the journal Pediatrics recommended that mothers clean off their child’s pacifier by putting it in their own mouths. That’s right. What the researchers found is that infants whose mothers sucked on their pacifiers to clean them developed fewer allergies than children whose mothers rinsed or boiled the pacifiers. The children of moms who gave pacifiers a mouth rinse also had lower rates of eczema, fewer signs of asthma and smaller amounts of a type of white blood cell that rises in response to allergies and other disorders. The findings are in line with the growing evidence that some exposure to germs at a young age can be good for kids.
5) Heigh-ho, heigh-ho, it’s off to work I go: About 40 percent of working mothers in the U.S. now say the ideal situation for them would be to work full time. That’s according to the latest research on the matter from the Pew Research Center. It’s almost twice as many who felt that way in 2007, when 21 percent of the women surveyed said that would be their preference. The researchers speculated that this is probably a reflection of tough economic times. But working part time is still the top choice among working women, although the percentage of women who said that would be the best situation for them dropped from 60 percent in 2007 to 50 percent in the most recent survey.
6) Don’t do what I do: Just as moms generally can do more good for their kids than dads, they also apparently can do more harm. A 34-year study by the British think tank Demos found that the alcohol drinking habits of mothers can have the greatest impact on how their children consume alcohol. While at age 16, a child’s drinking behavior was greatly influenced by peers, the researchers found that that changed as children reached maturity. Then, the scientists more often discovered clear connections between alcohol consumption–particularly binge drinking–and childhood memories of how their mothers would drink.
7) Crouching tiger, failing children: So much for the power of Tiger Moms, the stereotypical demanding Asian mother depicted in the much-debated Battle Hymn of the Tiger Mother in 2011. A University of Texas professor named Su Yeong Kim, who had been following more than 300 Asian-American families for a decade, recently published her findings. What she observed didn’t quite match the stereotype. Children of parents whom Kim classified as “tiger” had lower academic achievement–and more psychological problems–than the kids of parents characterized as “supportive” or “easygoing.”
8) Even in utero we know to take a vowel: According to a joint study of newborns in Washington State and in Stockholm, babies start learning language from their moms even before they leave the womb. The scientists said their research showed that the infants began locking on to the vowel sounds of their mothers before they were born. How did they know that? They studied 40 infants, all about 30 hours old, and they found that the babies–who were played vowel sounds in foreign languages and the language of their mothers–consistently sucked longer on pacifiers when they heard sounds different from the ones they had heard in utero.
9) Sure, but you’d know nothing about Legos without us: Judging by a bit of research done in Finland, boys, at least in times past, could take almost nine months off a mother’s life, compared to girls. The Finnish scientists analyzed the post-childbirth survival rates of 11,166 mothers and 6,360 fathers in pre-industrial Finland, between the 17th and 20th centuries. And they found that a mother who bore six sons would live on average another 32.4 years after the youngest son’s birth, while a mother who gave birth to girls would live approximately 33.1 years after her youngest daughter came along. The shorter life expectancy was the same regardless of the mom’s social or financial status. The researchers surmised that not only was bearing boys more physically demanding for the mothers, but also that daughters were more likely to prolong their mothers’ lives by helping with household responsibilities.
10) Putting it in words: And finally…this probably shouldn’t come as a big surprise, but a study just published in the journal Proceedings of the National Academy of Sciences suggests that caveman didn’t just grunt, but actually had a decent little vocabulary that included the equivalent of words for ‘thou’, ‘you’, ‘we,’ ‘bark,’ ‘fire,’ ‘spit’ and yes, ‘mother.’
More from Smithsonian.com
May 8, 2013
Cell phones are so many things now–computer, map, clock, calculator, camera, shopping device, concierge, and occasionally, a phone. But more than anything, that little device that never leaves your person is one amazingly prolific data engine.
Which is why last October, Verizon Wireless, the largest U.S, carrier with almost 100 million customers, launched a new division called Precision Market Insights. And why, at about the same time, Madrid-based Telefonica, one of the world’s largest mobile network providers, opened its own new business unit, Telefonica Dynamic Insights.
The point of these ventures is to mine, reconstitute and sell the enormous amount of data that phone companies gather about our behavior. Every time we make a mobile call or send a text message–which pings a cell tower–that info is recorded. So, with enough computer power, a company can draw pretty accurate conclusions about how and when people move through a city or a region. Or they can tell where people have come from to attend an event. As part of a recent case study, for example, Verizon was able to say that people with Baltimore area codes outnumbered those with San Francisco area codes by three to one inside the New Orleans Superdome for the Super Bowl in February.
In a world enamored of geolocation, this is digital gold. It’s one thing to know the demographic blend of a community, but to be able to find out how many people pass by a business and where they’re coming from, that adds a whole nother level of precision to target marketing.
Follow the crowd
But this data have value beyond companies zeroing in on potential customers. It’s being used for social science, even medical research. Recently IBM crunched numbers from 5 million phone users in the Ivory Coast in Africa and, by tracking movements of people through which cell towers they connected to, it was able to recommend 65 improvements to bus service in the city of Abidjan.
And computer scientists at the University of Birmingham in England have used cell phone data to fine tune analysis of how epidemics spread. Again, it’s about analyzing how people move around. Heretofore, much of what scientists knew about the spread of contagious diseases was based largely on guesswork. But now, thanks to so many pings from so many phones, there’s no need to guess.
It’s important to point out that no actual identities are connected to cell phone data. It all gets anonymized, meaning there shouldn’t be a way to track the data back to real people.
There shouldn’t be.
Leaving a trail
But a study published in Scientific Reports in March found that even anonymized data may not be so anonymous after all. A team of researchers from Louvain University in Belgium, Harvard and M.I.T. found that by using data from 15 months of phone use by 1.5 million people, together with a similar dataset from Foursquare, they could identify about 95 percent of the cell phones users with just four data points and 50 percent of them with just two data points. A data point is an individual’s approximate whereabouts at the approximate time they’re using their cell phone.
The reason that only four locations were necessary to identify most people is that we tend to move in consistent patterns. Just as everyone has unique fingerprints, everyone has unique daily travels. While someone wouldn’t necessarily be able to match the path of a mobile phone–known as a mobility trace–to a specific person, we make it much easier through geolocated tweets or location “check-ins,” such as when we use Foursquare.
“In the 1930s, it was shown that you need 12 points to uniquely identify and characterize a fingerprint,” the study’s lead author, Yves-Alexandre de Montijoye, told the BBC in a recent interview. “What we did here is the exact same thing, but with mobility traces. The way we move and the behavior is so unique that four points are enough to identify 95 percent of the people.”
“We think this data is more available than people think. When you share information, you look around and you feel like there are lots of people around–in a shopping center or a tourist place–so you feel this isn’t sensitive information.”
In other words, you feel anonymous. But are you really? De Montijoye said the point of his team’s research wasn’t to conjure up visions of Big Brother. He thinks there’s much good that can come from mining cell phone data, for businesses, for city planners, for scientists, for doctors. But he thinks it’s important to recognize that today’s technology makes true privacy very hard to keep.
The title of the study? “Unique in the Crowd.”
Here are other recent developments related to mobile phones and their data:
- Every picture tells your story: Scientists at Carnegie Mellon University’s Human Computer Interaction Center say their research of 100 smartphone apps found that about half of them raised privacy concerns. For instance, a photo-sharing app like Instagram provided information that allowed them to easily discover the location of the person who took the photo.
- Cabbies with cameras: In the Mexican city of Tuxtla Gutiérrez, taxi drivers have been provided with GPS-enabled cell phones and encouraged to send messages and photographs about accidents or potholes or broken streetlights.
- Follow that cell: Congress has started looking into the matter of how police use cell phone data to track down suspects. The key issue is whether they should be required to get a warrant first.
- Follow that cell II: Police in Italy have started using a data analysis tool called LogAnalysis that makes it especially easy to visualize the relationships among conspiring suspects based on their phone calls. In one particular case involving a series of robberies, the tool showed a flurry of phone activity among the suspects before and after the heists, but dead silence when the crimes were being committed.
Video bonus: If you’re at all paranoid about how much data can be gleaned from how you use your mobile phone, you may not want to watch this TED talk by Malte Spitz.
May 3, 2013
It may have been the word retrieval adventure I had the other night when I couldn’t remember the name of thinly sliced cured ham. (I nailed the “p,” but didn’t come close to conjuring up “prosciutto.”) Or it could have been the annoying pain I feel in a knuckle on my right hand these days. Probably both.
All I know is that when I read about a recent study in which scientists were able to slow down the aging process in mice, I was more than a little intrigued.
According to the researchers at the Albert Einstein College of Medicine in New York, the key to stalling the harsh march of aging is likely deep inside your brain, specifically the almond-size section called the hypothalamus.
It has long been associated with our sense of hunger and thirst, our body temperature and feelings of fatigue. But the scientists, in the study published in the journal Nature on Wednesday, say they found that by deactivating a molecule found in the hypothalamus called NF-kB, they were able to get mice to live 20 percent longer, and also show fewer physical signs of aging.
More specifically, when they blocked the substance from the hypothalamus, the animals lived up to 1,100 days, about 100 days longer than the normal limit for mice. But when they gave other mice more NF-kB, they all died within 900 days. The mice without NF-kB also had more muscle and bone, healthier skin and were better at learning.
During the study, the researchers also determined that NF-kB lowered levels of a hormone called GnRH. And when they gave the mice a daily treatment of that hormone, it too helped to extend the animals’ lives and even caused new neurons to develop in their brains.
This is where I need to raise the caveat about research with mice, namely that what works with them often doesn’t carry over to humans. Or as io9 noted, “comparing the aging processes of mice to humans is a precarious proposition at best.”
That said, the lead scientist for the study, Dongsheng Cai, says he’s excited by what the research suggests. “It supports the idea that aging is more than a passive deterioriation of different tissues,” he told The Guardian in an interview. “It is under control and can be manipulated.”
Thanks for my memory
Then there is Theodore Berger. He’s a neuroscientist at the University of Southern California in Los Angeles and he believes that one day in the not too distant future, it may be possible to use electrical implants in the brain to help people retrieve long-term memories.
So far, Berger and his research team have been able to show how a silicon chip externally connected to rat and monkey brains by electrodes can process information as actual neurons do. And last fall, the researchers demonstrated that they could help monkeys bring back long-term memories.
They focused on the prefrontal cortex, the part of the brain that retrieves the memories created by the hippocampus. The scientists placed electrodes in the monkeys’ brains to capture the neuron code formed in the prefrontal cortex that, the researchers believed, allowed the animals to remember an image they had been shown earlier. Then they drugged the monkeys with cocaine, which impaired activity in that part of their brains. Next they used the implanted electrodes to send electrical pulses carrying the captured code to the monkeys’ prefrontal cortex, and that, according to Berger, significantly improved the animals’ performance on a memory test.
Of course, the more you study the brain, the more complex it gets. And it’s quite possible that Berger hadn’t captured a code for how all memories are stored, but rather a code related only to the specific task of recalling an image. He says that within the next two years, he and his colleagues plan to implant a memory chip in animals, one that should, once and for all, determine if they have indeed cracked the code of creating long-term memories of many different situations and behaviors.
As he told M.I.T.’s Technology Review, ““I never thought I’d see this go into humans, and now our discussions are about when and how. I never thought I’d live to see the day, but now I think I will.”
The ticking clock
Here’s other recent research on aging and memory:
- Be still, my heart: After tracking more than 5,000 men for 40 years, Danish scientists concluded that those with high resting heart rates–above 80 beats per minute–were considerably more likely to die at a younger age, even if they were considered healthy.
- Not to mention it was a lot safer than actually having them drive: According to a study at the University of Iowa, elderly people who played a video game called “Road Tour” for as little as 10 hours, were able to measurably sharpen their cognitive skills.
- And throw in a side of olive oil: More kudos for the Mediterranean diet. A study published in the journal Neurology earlier this week found that people who followed the diet, built around eating fish, olive oil and vegetables and very little meat, were 19 percent less likely to suffer memory problems or cognitive decay.
- Although now they only dream in pink: And then there’s this report from German scientists: By having people listen to “pink noise” sounds that matched their brain wave oscillations as they slept, researchers were able to help them remember things they had learned the previous day.
- Dead and famous: Research by Australian scientists based on obituaries published in the New York Times over a two-year period found that people who were famous were more likely to die younger, particularly performers and athletes. The study also determined that performers were at a particularly high greatest risk of dying of lung cancer.
- We’re gonna need more fists: And finally, scientists at Montclair State University in New Jersey say their research shows that by clenching your right fist before memorizing something, and then your left when you want to remember it, you have a better chance of your memory coming through for you.
Video bonus: Here’s a short tutorial on why we age, told through the magic of whiteboard and markers:
Video bonus bonus: And a little visual proof that no one ages quite like a rock star.
More from Smithsonian.com