Smithsonian.com

Facial analysis at work. Image courtesy of Affectiva

As much time as we spend with our cell phones and laptops and tablets, it’s still pretty much a one-way relationship. We act, they respond. Sure, you can carry on a conversation with Siri on your iPhone, and while she is quick, it hardly qualifies as playful bantering. You ask questions, she gives answers.

But what if these devices could really read our emotions? What if they could interpret every little gesture, every facial cue so that they can gauge our feelings as well as–maybe better than–our best friends? And then they respond, not with information, but what might pass for empathy.

We’re not there yet, but we’re quickly moving in that direction, driven by a field of science known as affective computing. It’s built around software that can measure, interpret and react to human feelings. This might involve capturing your face on camera and then applying algorithms to every aspect of your expressions to try to make sense of each smirk and chin rub. Or it might involve reading your level of annoyance or pleasure by tracking how fast or with how much force you tap out a text or whether you use emoticons. And if you seem too agitated–or drunk–you could get a message suggesting that you might want to hold off pressing the send icon.

Seeing how difficult it is for us humans to make sense of other humans, this notion of programming machines to read our feelings is no small challenge. But it’s picking up speed, as scientists sharpen their focus on teaching devices emotional intelligence.

Every move you make

One of the better examples of how affective computing can work is the approach of a company called, appropriately, Affectiva. It records expressions and then, using proprietary algorithms, scrutinizes facial cues, tapping into a database of almost 300 million frames of elements of human faces. The software has been refined to the point where it can associate various combinations of those elements with different emotions.

When it was developed at M.I.T’s Media Lab by two scientists, Rosalind Picard and Rana el Kaliouby, the software, known as Affdex, was designed with the purpose of helping autistic children communicate better. But it clearly had loads of potential in the business world, and so M.I.T. spun the project off into a private company. It has since raised $21 million from investors.

So how is Affdex being used? Most often, it’s watching people watching commercials. it records people as they view ads on their computers–don’t worry, you need to opt in for this–and then, based on its database of facial cues, evaluates how the viewers feel about what they’ve seen. And the software doesn’t provide just an overall positive or negative verdict; it breaks down the viewers’ reactions second by second, which enables advertisers to identify, with more precision than ever before, what works in a commercial and what doesn’t.

It also is able to see that while people say one thing, their faces can say another. During an interview with the Huffington Post, el Kaliouby gave the example of the response to an ad for body lotion that aired in India. During the commercial, a husband playfully touches his wife’s exposed stomach. Afterwards, a number of women who had watched it said they found that scene offensive. But, according to el Kaliouby, the videos of the viewers showed that every one of the women responded to the scene with what she called an “enjoyment smile.”

She sees opportunities beyond the world of advertising. Smart TVs could be that much smarter about what kind of programs we like if they’re able to develop a memory bank of our facial expressions. And politicians would be able to get real-time reactions to each line they utter during a debate and be able to adapt their messages on the fly. Plus, says el Kaliouby, there could be health applications. She says it’s possible to read a person’s heart rate with a webcam by analyzing the blood flow in his or her face.

“Imagine having a camera on all the time monitoring your heart rate,” she told the Huffington Post, “so that it can tell you if something’s wrong, if you need to get more fit, or if you’re furrowing your brow all the time and need to relax.”

So what do you think, creepy or cool?

Tracking devices

Here are five other ways machines are reacting to human emotions:

• And how was my day?: Researchers at the University of Cambridge have developed an Android mobile app that monitors a person’s behavior throughout the day, using incoming calls and texts, plus social media posts to track their mood. The app, called “Emotion Sense,” is designed to create a “journey of discovery,” allowing users to have a digital record of the peaks and valleys of their daily lives. The data can be stored and used for therapy sessions.
• And this is me after the third cup of coffee: Then there’s Xpression, another mood-tracking app created by a British company called EI Technologies. Instead of relying on people in therapy to keep diaries of their mood shifts, the app listens for changes in a person’s voice to determine if they are in one of five emotional states: calm, happy, sad, angry or anxious/frightened. It then keeps a list of a person’s moods and when they change. And, if the person desires, this record can automatically be sent to a therapist at the end of every day.
• What if you just hate typing on a phone? : Scientists at Samsung are working on software that will gauge your frame of mind by how you type out your tweets on your smartphone. By analyzing how fast you type, how much the phone shakes, how often you backspace mistakes, and how many emoticons you use, the phone should be able to determine if you’re angry, surprised, happy, sad, fearful, or disgusted. And based on what conclusion it draws, it could include with your tweet the appropriate emoticon to tip off your followers to your state of mind.
• Just don’t invite your friends over to watch: Using a sensor worn on the wrist and a smartphone camera worn around the neck, researchers at M.I.T. have created a “lifelogging” system that collects images and data designed to show a person which events represented their emotional highs and lows. The system, called Inside-Out, includes a bio-sensor in a wristband that tracks heightened emotions through electrical charges in the skin while the smartphone tracks the person’s location and takes several photos a minute. Then, at the end of the day, the user can view their experiences, along with all the sensor data.
• Your brow says you have issues: This probably was inevitable. Researchers at the University of Southern California have created a robotic therapist that not only is programmed to encourage patients with well-timed “Uh-huhs,” but also is expert, using motion sensors and voice analysis, at interpreting a patient’s every gesture and voice inflection during a therapy session.

Video bonus: Want to see how bizarre this trend of devices reading human emotions can get? Check out this promotion of Tailly, a mechanical tail that picks up your level of excitement by tracking your heart rate and then wags appropriately.

More from Smithsonian.com

This New Robot Has a Sense of Touch

Cooking With Robots

Samsung TVs get smarter. Photo courtesy of Samsung

Since the beginning of mankind, we’ve wanted our kids to get smarter. Since the beginning of the 21st century, we’ve wanted our phones to get smarter.

So when are we going start wanting our TVs to get smarter? Or will we always be content with them being dumb, as long as they’re big and dumb? Okay, maybe not dumb, but most of us don’t yet feel a compelling need to have our TVs think like computers, as long as the picture looks pretty up there on the wall.

Which always makes things interesting at the Great Gadgetpalooza also known as the Consumer Electronics Show (CES). For the past several years, the big electronics companies that focus on hardware, such as Samsung and Panasonic, and the big tech companies that focus on software, such as Google, have been rolling out nifty products at the annual Las Vegas spectacle with the promise that this is the year that Smart TV goes mainstream.

Boob tube no more

And so it’s been at this year’s version of CES, which ends today. Samsung has done its part to convince us that the time has come for us to love TVs for their brains by unveiling what it calls its S-Recommendation engine.

It’s software that, as Samsung puts it, not only understands what you like, but recommends things it thinks you’ll like. (Sure, Amazon’s been doing this for years, but this is your big, dumb TV we’re talking about.) And it doesn’t just suggest TV shows, but could throw in streaming programs options from the Web, or even video you’ve shot on your smartphone.

The goal ultimately is to get you to do all those things you’re now doing on your smartphone or your tablet–say, watch Hulu or Skype with a family member or check out your Facebook page–on your TV instead. To encourage that behavior, Samsung has revamped its Smart Hub so you can flip through all of your entertainment options in five different index screens–one that tells you what’s on regular old TV now or soon, another that lists movies and on-demand TV, a third that pulls in photos or music or video stored on any other devices around the house, a fourth where you can Skype or pull up Facebook and a fifth that provides access to any apps you’ve downloaded.

And neither of the above requires pushing a lot of buttons on a remote. The S-Recommendation engine responds to voice commands and the Smart Hub is designed to be controlled with hand gestures.

For its part, Panasonic has rolled out a feature it calls My Home Screen, which allows each member of your family to create his or her own homepage on the TV, where easy access is provided to their favorite digital content, streaming video and apps. Some of the company’s Viera models actually come with their own cameras that tell the TV who turned it on. And as a smart TV should, it dutifully brings up that person’s home screen.

Plus, Panasonic unveiled “Swipe and Share 2.0″, which lets users move photos from a tablet or phone to a big TV screen, where they can then be edited with a touch pen.

But can you love a TV?

So that seals it, right? This must be the year when TVs take back center stage, especially now that they’re finally learning to care about our needs, right?

Maybe not. We’ve built some pretty strong personal connections to our cell phones and tablets. And a lot of people think it’s going to take a while for us to develop that kind of bond with a TV, no matter how smart it is.

As Greg Stuart, CEO of the Mobile Marketing Association told Ad Age earlier this week: “”People don’t have that kind of interactive relationship with their TV. The TV on the wall is a family device. It’s a multi-user device. If I want to share something, its going to be with a personal device, and that’s going to be my tablet or my mobile.”

TV or Not TV?

Here are other recent TV innovations:

• Robert, 6th Earl of Grantham, meet Tony Soprano: One day, thanks to Samsung, two people will be able to watch full-screen versions of Downton Abbey and Sopranos reruns at the same time. By adapting 3D technology, the company has created a TV that can display a different and full resolution image to each viewer depending on whether they’re sitting to the left or the right of the screen. Of course, both people would have to wear special glasses that come with headphones so you can hear only the sound for your show, but is that such a big price to pay for domestic peace?
• Read my lips. No more Gangham style: LG, the other South Korean TV giant, has upgraded its “Magic Remote” so that it now responds to natural language. You say the name of a show or even something like “videos with Gangham-style dancing,” and your choice pops up on the screen.
• I got my MoVo workin’: Also at CES, the Chinese TV manufacturer TCL showed off an HD TV called MoVo that uses facial recognition software to identify who’s watching and then make programming suggestions customized for that person.
• Okay, who blinked?: Meanwhile, Haier, another Chinese company, has developed a technology it calls Eye Control TV where, yes, you can change channels by moving your eyes.
• Ah, to be 65 and only see ads for meds: It was only a matter of time. A company called Gracenote will soon begin trials on a technology that, based on your viewing habits and personal data, will personalize the TV ads you see. Isn’t that special?

Video bonus: You didn’t make it to the big electronics show this year? Not to worry. Here’s the Samsung demo of its S-Recommendation engine. Remember, people tend to gush a lot at CES.

More from Smithsonian.com

Next Up? The Smart Watch

Getting Smart About Traffic

Pattern recognition of a butterfly wing. Image courtesy of Li Li

Here in Washington we have heard of this thing you call “advance planning,” but we are not yet ready to embrace it. A bit too futuristic.

Still, we can’t help but admire from afar those who attempt to predict what could happen more than a month from now. So I was impressed a few weeks ago when the big thinkers at IBM imagined the world five years hence and identified what they believe will be five areas of innovation that will have the greatest impact on our daily lives.

They’ve been doing this for a few years now, but this time the wonky whizzes followed a theme--the five human senses. Not that they’re saying that by 2018, we’ll all be able to see, hear and smell better, but rather that machines will–that by using quickly-evolving sensory and cognitive technologies, computers will accelerate their transformation from data retrieval and processing engines to thinking tools.

See a pattern?

Today, let’s deal with vision. It’a logical leap to assume that IBM might be referring to Google’s Project Glass. No question that it has redefined the role of glasses, from geeky accessory that helps us see better to combo smartphone/data dive device we’ll someday wear on our faces.

But that’s not what the IBMers are talking about. They’re focused on machine vision, specifically pattern recognition, whereby, through repeated exposure to images, computers are able to identify things.

As it turns out, Google happened to be involved in one of last year’s more notable pattern recognition experiments, a project in which a network of 1,000 computers using 16,000 processors was, after examining 10 million images from YouTube videos, able to teach itself what a cat looked like.

What made this particularly impressive is that the computers were able to do so without any human guidance about what to look for. All the learning was done through the machines working together to decide which features of cats merited their attention and which patterns mattered.

And that’s the model for how machines will learn vision. Here’s how John Smith, a senior manager in IBM’s Intelligent Information Management, explains it:

“Let’s say we wanted to teach a computer what a beach looks like. We would start by showing the computer many examples of beach scenes. The computer would turn those pictures into distinct features, such as color distributions, texture patterns, edge information, or motion information in the case of video. Then, the computer would begin to learn how to discriminate beach scenes from other scenes based on these different features. For instance, it would learn that for a beach scene, certain color distributions are typically found, compared to a downtown cityscape.”

How smart is smart?

Good for them. But face it, identifying a beach is pretty basic stuff for most of us humans. Could we be getting carried away about how much thinking machines will be able to do for us?

Gary Marcus, a psychology professor at New York University, thinks so. Writing recently on The New Yorker’s website, he concludes that while much progress has been made in what’s become known as “deep learning,” machines still have a long way to go before they should be considered truly intelligent.

“Realistically, deep learning is only part of the larger challenge of building intelligent machines. Such techniques lack ways of representing causal relationships (such as between diseases and their symptoms), and are likely to face challenges in acquiring abstract ideas like “sibling” or “identical to.” They have no obvious ways of performing logical inferences, and they are also still a long way from integrating abstract knowledge, such as information about what objects are, what they are for, and how they are typically used.”

The folks at IBM would no doubt acknowledge as much. Machine learning comes in steps, not leaps.

But they believe that within five years, deep learning will have taken enough forward steps that computers will, for instance, start playing a much bigger role in medical diagnosis, that they could actually become better than doctors when it comes to spotting tumors, blood clots or diseased tissue in MRIs, X-rays or CT scans.

And that could make a big difference in our lives.

Seeing is believing

Here are more ways machine vision is having an impact on our lives:

• Putting your best arm forward: Technology developed at the University of Pittsburgh uses pattern recognition to enable paraplegics to control a robotic arm with their brains.
• Your mouth says yes, but your brain says no: Researchers at Stanford found that using pattern recognition algorithms on MRI scans of brains could help them determine if someone actually had lower back pain or if they were faking it.
• When your moles are ready for their close ups: Last year a Romanian startup named SkinVision launched an iPhone app that allows people to take a picture of moles on their skin and then have SkinVision’s recognition software identify any irregularities and point out the risk level–without offering an actual diagnosis. Next step is to make it possible for people to send images of their skin directly to their dermatologist.
• Have I got a deal for you: Now under development is a marketing technology called Facedeals. It works like this: Once a camera at a store entrance recognizes you, you’re sent customized in-store deals on your smart phone. And yes, you’d have to opt in first.
• I’d know that seal anywhere: A computerized photo-ID system that uses pattern recognition is helping British scientists track gray seals, which have unique markings on their coats.

Video bonus: While we’re on the subject of artificial intelligence, here’s a robot swarm playing Beethoven, compliments of scientists at Georgia Tech. Bet you didn’t expect to see that today.

More from Smithsonian.com

A More Human Artificial Brain

How Technology Fights Terrorism

A mannequin that gathers intelligence about customers. Photo courtesy of Almax.

Black Thriday is over. So is Small Business Saturday and Cyber Monday. Today, in case you didn’t know, is either Green Tuesday or Giving Tuesday, depending on whether you feel like eco-shopping or giving to charity.

Not sure what tomorrow may bring (How about Weird Relative Gift Wednesday?), but I suppose shopping does feel less chaotic if someone’s organizing it into theme days, although that doesn’t always stop it from devolving into a contact sport.

Can you imagine American shoppers embracing something like iButterfly, a mobile app popular in Asia where customers earn coupons by tracking down virtual butterflies with their smartphones? Me neither.

In the U.S., it’s about cutting to the chase and here the chase is after the sweetest deals, pure and simple, without having to bother with running after faux flying insects. And retailers have ratcheted up the competition, using the latest tracking technology to closely monitor their competitors’ pricing decisions and undercut them, in close to real-time, on their own websites. When Best Buy, for instance, published advertising saying it would be selling a $1,500 Nikon camera for $1,000, Amazon responded on Thanksgiving morning by cutting its price for the same camera to $997.

To know you is to lure you

No question that the big hook remains big bargains. But a lot of companies are also getting much more aggressive about mining data to tap into the power of personalization. The more they know about you and your tastes and habits and what you say on Facebook, the more they can press your buy buttons–but in a way that feels like they’re doing it all for you.

Now grocery stores like Safeway and Kroger have even started to customize prices in offers to loyalty cardholders. As Stephanie Clifford noted in the New York Times:

“Hoping to improve razor-thin profit margins, they are creating specific offers and prices, based on shoppers’ behaviors, that could encourage them to spend more: a bigger box of Tide and bologna if the retailer’s data suggests a shopper has a large family, for example (and expensive bologna if the data indicates the shopper is not greatly price-conscious).”

And RetailMeNot, the most popular coupon site in the U.S., has just launched an app that steers you to coupons you’re more likely to use based on your Likes and other personal info gleaned from Facebook.

Data creep

But when does solicitousness turn creepy? Is it when you receive a pitch in your email for an outfit you pinned on Pinterest? Or when you start getting offered bargains from stores you happen to pass on the way to work every day?

If you believe a recent survey by Accenture Interactive, a clear majority–61 percent–of online shoppers in the U.S. and the U.K. are willing to give up some privacy if it means they can receive personalized offers from retailers.

And more than 50 percent of those surveyed in the U.S. said they’re comfortable with the idea of their favorite retailers tracking their personal data in order to fine tune recommendations for future purchases.

But only so comfortable. Almost 90 percent of the respondents said that’s entirely dependent on whether retailers offer them choices on how their personal info can be used.

As Kurt Kendall, a retail consultant, put it in a recent interview with Cox Newspapers: “People do not want to feel like they’re being stalked.”

I’ve got my fake eye on you

How about being watched? The obsession with gathering intelligence about customer behavior has reached the point where an Italian company is selling mannequins equipped with cameras to watch shoppers. This model, called the EyeSee, is being sold by Milan-based Almax for more than $5,000.

That’s a lot of money for a pretend person. But this one has a camera embedded in one eye that feeds data into facial-recognition software which logs the age, gender and race of passers-by. It’s all about collecting data–no video is actually stored.

Almax won’t reveal which of its clients have purchased EyeSee mannequins, but it has said that one added a children’s line of clothing when the camera observed that kids made up more than half its mid-afternoon traffic. Another, according to Almax, discovered that a third of its visitors using one of its doors after 4 p.m. were Asian, prompting it to place Chinese-speaking staff by that entrance.

But wait, there’s more. Almax is developing a model that will recognize words well enough that stores will be able to find out what customers are saying about the mannequin’s outfit–again without recording a thing.

The shipping news

Here are more examples of how companies are using technology to build relationships with customers.

• Or simply “Clothes That Don’t Make Me Look Fat”: For those who know what they like in fashion, Shop It to Me has just launched a site called Shop It to Me Threads that allows you to create a customized page that’s updated daily with the latest news and deals on your favorite fashion trends, designers, types of items, or combination of elements, such as “Michael Kors Bags and Shoes under $250″ or “Pencil skirts under $100.”
• Pickie picky: E-commerce start-up Pickie has come out with an iPad app that builds a personalized shopping catalog for you, based on your preferences expressed on Facebook, along with suggestions from your friends. And you’re able to order items directly from your customized Pickie site.
• Do it for the children: To counter the trend called “showrooming,” where people check out products in a store and then go home and buy it from another company online, Target is encouraging shoppers to go online while they’re in its stores. During the holidays, the retailer is featuring 20 hot toys at the front of its stores next to signs with QR codes. Shoppers with smart phones can scan the codes, buy a toy and have it shipped free.
• What about Pop Tarts and headphones?: Amazon, through its subsidiary Quidsi, is sharpening its aim at moms who shop online. Last month it launched another narrowly targeted site called AfterSchool.com. It lists more than 70,000 of the sort of things kids need after school, from ballet shoes and shin guards to basketballs and jewelry kits.
• And if you’re really loyal, a greeter washes your car: Earlier this month Walmart, through its Silicon Valley operation @WalmartLabs, rolled out Goodies, a food subscription service. For $7 a month, people who sign up will receive a box of gourmet snacks, such as Dang Toasted Coconut Chips and a Nutella & Go snack pack. And if they’re active on the Goodies site by rating products and writing reviews, they can earn enough loyalty points to start getting their monthly goodies for free.

Video bonus: Based on this video from Comiket, the huge comic book convention held in Tokyo, the Japanese and Americans have very different styles when it comes to the surging crowd thing.

More from Smithsonian.com

How Maker Culture Is Reshaping Retail Design

Are Your Eyes Also a Window to Your Brain

The cat face created by Google’s computer brain. Image courtesy of Google.

A few months ago Google shared with us another challenge it had taken on. It wasn’t as fanciful as a driverless car or as geekily sexy as augmented reality glasses, but in the end, it could be bigger than both. In fact, it likely will make both of them even more dynamic.

What Google did was create a synthetic brain, or at least the part of it that processes visual information. Technically, it built a mechanical version of a neural network, a small army of 16,000 computer processors that, by working together, was actually able to learn.

At the time, most of the attention focused on what all those machines learned, which mainly was how to identify cats on YouTube. That prompted a lot of yucks and cracks about whether the computers wondered why so many of the cats were flushing toilets.

But Google was going down a path that scientists have been exploring for many years, the idea of using computers to mimick the connections and interactions of human brain cells to the point where the machines actually start learning. The difference is that the search behemoth was able to marshal resources and computing power that few companies can.

The face is familiar

For 10 days, non-stop, 1,000 computers–using those 16,000 processors–examined random thumbnail images taken from 10 million different YouTube videos. And because the neural network was so big–it had more than a billion connections–it was able to learn to identify features on its own, without any real human guidance. Through the massive amount of information it absorbed, the network, by recognizing the relationships between data, basically taught itself the concept of a cat.

Impressive. But in the realm of knowledge, is this cause for great jubilation? Well, yes. Because eventually all the machines working together were able to decide which features of cats merited their attention and which patterns mattered, rather than being told by humans which particular shapes to look for. And from the knowledge gained through much repetition, the neural network was able to create its own digital image of a cat’s face.

That’s a big leap forward for artificial intelligence. It’s also likely to have nice payoffs for Google. One of its researchers who worked on the project, an engineer named Jeff Dean, recently told MIT’s Technology Review that now his group is testing computer models that understand images and text together.

“You give it ‘porpoise” and it gives you pictures of porpoises,” Dean explained. “If you give it a picture of a porpoise, it gives you ‘porpoise’ as a word.”

So Google’s image search could become far less dependent on accompanying text to identify what’s in a photo. And it’s likely to apply the same approach to refining speech recognition by being able to gather extra clues from video.

No question that the ability to use algorithms to absorb and weave together many streams of data, even different types of data, such as sound and images, will help make Google’s driverless car that much more autonomous. Same with Google glasses.

But now a slice of perspective. For all its progress, Google still has a long way to go to measure up to the real thing. Its massive neural network, the one with a billion connections, is, in terms of neurons and synapses, still a million times smaller than the human brain’s visual cortex.

A matter of intelligence

Here are more recent developments in artificial intelligence:

• A bee, or not a bee: A team of British scientists are attempting to create an accurate model of a honeybee’s brain. By reproducing the key systems that make up a bee’s perception, such as vision and scent, the researchers hope to eventually be able to install the artificial bee brain in a small flying robot.
• But does it take the cover into account?: New software called Booksai is using artificial intelligence to give you book recommendations based on the style, tone, mood and genre of things you already know you like to read.
• Do I always look this good?: Scientists at Yale have programmed a robot that can recognize itself in the mirror. In theory, that should make the robot, named Nico, better able to interact with its environment and humans.
• Lost in space no more: Astronomers in Germany have developed an artificial intelligence algorithm to help them chart and explain the structure and dynamics of the universe with amazing accuracy.
• Walk this way: Scientists at MIT have created a wearable intelligent device that creates a real-time map of where you’ve just walked. It’s designed as a tool to help first responders coordinate disaster search and rescue.

Video bonus: In France–where else?–an inventor has created a robot that not only prunes grape vines, but also has the intelligence to memorize the specific needs of each plant. And now it’s learning to pick grapes.

More from Smithsonian.com

Building a Human Brain

How Brains Make Money