Facial analysis at work. Image courtesy of Affectiva

As much time as we spend with our cell phones and laptops and tablets, it’s still pretty much a one-way relationship. We act, they respond. Sure, you can carry on a conversation with Siri on your iPhone, and while she is quick, it hardly qualifies as playful bantering. You ask questions, she gives answers.

But what if these devices could really read our emotions? What if they could interpret every little gesture, every facial cue so that they can gauge our feelings as well as–maybe better than–our best friends? And then they respond, not with information, but what might pass for empathy.

We’re not there yet, but we’re quickly moving in that direction, driven by a field of science known as affective computing. It’s built around software that can measure, interpret and react to human feelings. This might involve capturing your face on camera and then applying algorithms to every aspect of your expressions to try to make sense of each smirk and chin rub. Or it might involve reading your level of annoyance or pleasure by tracking how fast or with how much force you tap out a text or whether you use emoticons. And if you seem too agitated–or drunk–you could get a message suggesting that you might want to hold off pressing the send icon.

Seeing how difficult it is for us humans to make sense of other humans, this notion of programming machines to read our feelings is no small challenge. But it’s picking up speed, as scientists sharpen their focus on teaching devices emotional intelligence.

Every move you make

One of the better examples of how affective computing can work is the approach of a company called, appropriately, Affectiva. It records expressions and then, using proprietary algorithms, scrutinizes facial cues, tapping into a database of almost 300 million frames of elements of human faces. The software has been refined to the point where it can associate various combinations of those elements with different emotions.

When it was developed at M.I.T’s Media Lab by two scientists, Rosalind Picard and Rana el Kaliouby, the software, known as Affdex, was designed with the purpose of helping autistic children communicate better. But it clearly had loads of potential in the business world, and so M.I.T. spun the project off into a private company. It has since raised $21 million from investors.

So how is Affdex being used? Most often, it’s watching people watching commercials. it records people as they view ads on their computers–don’t worry, you need to opt in for this–and then, based on its database of facial cues, evaluates how the viewers feel about what they’ve seen. And the software doesn’t provide just an overall positive or negative verdict; it breaks down the viewers’ reactions second by second, which enables advertisers to identify, with more precision than ever before, what works in a commercial and what doesn’t.

It also is able to see that while people say one thing, their faces can say another. During an interview with the Huffington Post, el Kaliouby gave the example of the response to an ad for body lotion that aired in India. During the commercial, a husband playfully touches his wife’s exposed stomach. Afterwards, a number of women who had watched it said they found that scene offensive. But, according to el Kaliouby, the videos of the viewers showed that ever one of the women responded to the scene with what she called an “enjoyment smile.”

She sees opportunities beyond the world of advertising. Smart TVs could be that much smarter about what kind of programs we like if they’re able to develop a memory bank of our facial expressions. And politicians would be able to get real-time reactions to each line they utter during a debate and be able to adapt their messages on the fly. Plus, says el Kaliouby, there could be health applications. She says it’s possible to read a person’s heart rate with a webcam by analyzing the blood flow in his or her face.

“Imagine having a camera on all the time monitoring your heart rate,” she told the Huffington Post, “so that it can tell you if something’s wrong, if you need to get more fit, or if you’re furrowing your brow all the time and need to relax.”

So what do you think, creepy or cool?

Tracking devices

Here are five other ways machines are reacting to human emotions:

• And how was my day?: Researchers at the University of Cambridge have developed an Android mobile app that monitors a person’s behavior throughout the day, using incoming calls and texts, plus social media posts to track their mood. The app, called “Emotion Sense,” is designed to create a “journey of discovery,” allowing users to have a digital record of the peaks and valleys of their daily lives. The data can be stored and used for therapy sessions.
• And this is me after the third cup of coffee: Then there’s Xpression, another mood-tracking app created by a British company called EI Technologies. Instead of relying on people in therapy to keep diaries of their mood shifts, the app listens for changes in a person’s voice to determine if they are in one of five emotional states: calm, happy, sad, angry or anxious/frightened. It then keeps a list of a person’s moods and when they change. And, if the person desires, this record can automatically be sent to a therapist at the end of every day.
• What if you just hate typing on a phone? : Scientists at Samsung are working on software that will gauge your frame of mind by how you type out your tweets on your smartphone. By analyzing how fast you type, how much the phone shakes, how often you backspace mistakes, and how many emoticons you use, the phone should be able to determine if you’re angry, surprised, happy, sad, fearful, or disgusted. And based on what conclusion it draws, it could include with your tweet the appropriate emoticon to tip off your followers to your state of mind.
• Just don’t invite your friends over to watch: Using a sensor worn on the wrist and a smartphone camera worn around the neck, researchers at M.I.T. have created a “lifelogging” system that collects images and data designed to show a person which events represented their emotional highs and lows. The system, called Inside-Out, includes a bio-sensor in a wristband that tracks heightened emotions through electrical charges in the skin while the smartphone tracks the person’s location and takes several photos a minute. Then, at the end of the day, the user can view their experiences, along with all the sensor data.
• Your brow says you have issues: This probably was inevitable. Researchers at the University of Southern California have created a robotic therapist that not only is programmed to encourage patients with well-timed “Uh-huhs,” but also is expert, using motion sensors and voice analysis, at interpreting a patient’s every gesture and voice inflection during a therapy session.

Video bonus: Want to see how bizarre this trend of devices reading human emotions can get? Check out this promotion of Tailly, a mechanical tail that picks up your level of excitement by tracking your heart rate and then wags appropriately.

More from Smithsonian.com

This New Robot Has a Sense of Touch

Cooking With Robots

Cell phones have become prolific data engines. Photo courtesy of Flickr user Ed Yourdon

Cell phones are so many things now–computer, map, clock, calculator, camera, shopping device, concierge, and occasionally, a phone. But more than anything, that little device that never leaves your person is one amazingly prolific data engine.

Which is why last October, Verizon Wireless, the largest U.S, carrier with almost 100 million customers, launched a new division called Precision Market Insights. And why, at about the same time, Madrid-based Telefonica, one of the world’s largest mobile network providers, opened its own new business unit, Telefonica Dynamic Insights.

The point of these ventures is to mine, reconstitute and sell the enormous amount of data that phone companies gather about our behavior. Every time we make a mobile call or send a text message–which pings a cell tower–that info is recorded. So, with enough computer power, a company can draw pretty accurate conclusions about how and when people move through a city or a region. Or they can tell where people have come from to attend an event. As part of a recent case study, for example, Verizon was able to say that people with Baltimore area codes outnumbered those with San Francisco area codes by three to one inside the New Orleans Superdome for the Super Bowl in February.

In a world enamored of geolocation, this is digital gold. It’s one thing to know the demographic blend of a community, but to be able to find out how many people pass by a business and where they’re coming from, that adds a whole nother level of precision to target marketing.

Follow the crowd

But this data have value beyond companies zeroing in on potential customers. It’s being used for social science, even medical research. Recently IBM crunched numbers from 5 million phone users in the Ivory Coast in Africa and, by tracking movements of people through which cell towers they connected to, it was able to recommend 65 improvements to bus service in the city of Abidjan.

And computer scientists at the University of Birmingham in England have used cell phone data to fine tune analysis of how epidemics spread. Again, it’s about analyzing how people move around. Heretofore, much of what scientists knew about the spread of contagious diseases was based largely on guesswork. But now, thanks to so many pings from so many phones, there’s no need to guess.

It’s important to point out that no actual identities are connected to cell phone data. It all gets anonymized, meaning there shouldn’t be a way to track the data back to real people.

There shouldn’t be.

Leaving a trail

But a study published in Scientific Reports in March found that even anonymized data may not be so anonymous after all. A team of researchers from Louvain University in Belgium, Harvard and M.I.T. found that by using data from 15 months of phone use by 1.5 million people, together with a similar dataset from Foursquare, they could identify about 95 percent of the cell phones users with just four data points and 50 percent of them with just two data points. A data point is an individual’s approximate whereabouts at the approximate time they’re using their cell phone.

The reason that only four locations were necessary to identify most people is that we tend to move in consistent patterns. Just as everyone has unique fingerprints, everyone has unique daily travels. While someone wouldn’t necessarily be able to match the path of a mobile phone–known as a mobility trace–to a specific person, we make it much easier through geolocated tweets or location “check-ins,” such as when we use Foursquare.

“In the 1930s, it was shown that you need 12 points to uniquely identify and characterize a fingerprint,” the study’s lead author, Yves-Alexandre de Montijoye, told the BBC in a recent interview. “What we did here is the exact same thing, but with mobility traces. The way we move and the behavior is so unique that four points are enough to identify 95 percent of the people.”

“We think this data is more available than people think. When you share information, you look around and you feel like there are lots of people around–in a shopping center or a tourist place–so you feel this isn’t sensitive information.”

In other words, you feel anonymous. But are you really? De Montijoye said the point of his team’s research wasn’t to conjure up visions of Big Brother. He thinks there’s much good that can come from mining cell phone data, for businesses, for city planners, for scientists, for doctors. But he thinks it’s important to recognize that today’s technology makes true privacy very hard to keep.

The title of the study? “Unique in the Crowd.”

Private lives

Here are other recent developments related to mobile phones and their data:

• Every picture tells your story: Scientists at Carnegie Mellon University’s Human Computer Interaction Center say their research of 100 smartphone apps found that about half of them raised privacy concerns. For instance, a photo-sharing app like Instagram provided information that allowed them to easily discover the location of the person who took the photo.
• Cabbies with cameras: In the Mexican city of Tuxtla Gutiérrez, taxi drivers have been provided with GPS-enabled cell phones and encouraged to send messages and photographs about accidents or potholes or broken streetlights.
• Follow that cell: Congress has started looking into the matter of how police use cell phone data to track down suspects. The key issue is whether they should be required to get a warrant first.
• Follow that cell II: Police in Italy have started using a data analysis tool called LogAnalysis that makes it especially easy to visualize the relationships among conspiring suspects based on their phone calls. In one particular case involving a series of robberies, the tool showed a flurry of phone activity among the suspects before and after the heists, but dead silence when the crimes were being committed.

Video bonus: If you’re at all paranoid about how much data can be gleaned from how you use your mobile phone, you may not want to watch this TED talk by Malte Spitz.

More from Smithsonian.com

Future Parents Will Always, Always Know Where Their Kids Are

A Modern Sherlock Holmes and the Technology of Deduction

Smartphones are changing our notion of acceptable behavior. Photo courtesy of Flickr user Jennifer Conley

I committed my first texting heresy a few years ago when my son was away at college. I had asked him about a class he was taking and had needed three, maybe four sentences to express myself.

He responded with bemusement. Or maybe it was disgust. Who could tell?

But his message was clear: If I continued to be so lame as to send texts longer than two sentences–using complete words, no less–he would have little choice but to stop answering.

I was reminded of this less-than-tender father-son moment recently by a post by Nick Bilton for The New York Times’ Bits blog in which he railed against those who send “Thank you” emails, among other digital transgressions.

His contention is that such concise expressions of gratitude, while well-intended, end up being an imposition for recipients who have to open up an email to read a two-word message. Better to leave the sentiment unexpressed–although he does concede that it probably makes sense to indulge old folks, who are much more likely to appreciate the appreciation.

Behavior modification

Bilton’s larger point is that as technology changes how we communicate and gather information, we need to adapt what we consider proper etiquette. Why should we continue to leave voice mails, he argues, when a text is much more likely to be answered? And why, he asks, would anyone these days be so rude as to ask for directions?

Not that this is the first time that tech is forcing an etiquette rethink. Bilton harkens back to the early days of the telephone when people truly didn’t know what to say when they picked up a ringing phone. Alexander Graham Bell himself lobbied for “Ahoy,” while Thomas Edison pushed for “Hello.” Edison ruled, of course, although now that our phones tell who’s calling before we have to say a word, the typical greeting has devolved to “Hey” or the catatonically casual “‘S up.”

Sure, some of this is a generational thing–The Independent nailed that in a recent piece on how members of three generations of one family communicate–or not–with each other.

But it’s also about volume. Email never sleeps. For a lot of people, each day can bring a fire hose of digital messages. Imagine if you received 50 to 100 phone calls a day. You can bet you’d be telling people to stop calling.

If the purpose of etiquette is to be considerate of other people, Bilton would contend that that’s the whole idea behind cutting back on emails and voice mails. And he’d have a point.

Me, my phone and I

But then there’s the matter of device isolation. I’m sure you know it well by now–the person who starts texting away during a conversation, or a meal, or even a meeting, which is one of those things bosses tend not to like (not to mention that it probably also means the death of doodling.)

It’s hard to put a positive spin on this since it does send a pretty clear message: I’d rather focus my energy on connecting to someone through a device than in person. Maybe it’s just me, but that, I’d say, reeks of rude.

If anything, it’s going to get worse, especially with wearable tech about to go mainstream. Some think this is the year the smart watch could start to become the accessory of choice, which means people will be looking at their wrists a lot more in the future–not so much to check the time, which is rude enough, but more to see who’s sent them emails and texts.

And what about when Google Glass goes on the market later this year? They’re glasses that will enable you to check emails, go on the Web, watch videos, even take pictures, all while feigning eye contact with the people you’re with. And the Google Glass camera raises all kinds of issues. Will wearers have to make pre-date agreements not to take stealth photos, particularly any involving eating or drinking? Is anyone fair game in a Google Glass video?

But beyond questions of privacy and social boorishness, the impact of our obsession with digital devices, especially when it comes to the loss of personal connections, could go much deeper. In a piece in Sunday’s New York Times, Barbara Frederickson, a psychology professor at the University of North Carolina, cites research suggesting that if you don’t practice connecting face-to-face with others, you can start to lose your biological capacity to do so.

Writes Frederickson:

“When you share a smile or laugh with someone face to face, a discernible synchrony emerges between you, as your gestures and biochemistries, even your respective neural firings, come to mirror each other. It’s micro-moments like these, in which a wave of good feeling rolls through two brains and bodies at once, that build your capacity to empathize as well as to improve your health.”

Digital deviance

Here are other recent developments in how technology is affecting behavior:

• Yeah, but can I text while I meditate?: A course at the University of Washington is focusing on helping students improve their concentration skills by requiring them both to watch videos of themselves multitasking and to do meditation.
• And it really cuts down on shuffleboard injuries: A study at North Carolina State University found that seniors–people 63 years or older– who played video games had higher levels of well-being and “emotional functioning” and lower levels of depression than old folks who didn’t.
• Does loyalty go deeper than latte?: This May Starbucks will break new ground when it allows its loyalty cardholders to earn points by buying Starbucks products in grocery stores.

Video bonus: All kinds of embarrassing things can happen while you’re texting.

Video bonus bonus: More evidence of the obsession that is texting: Here’s a clip of a bride firing off one last message before she says her vows.

Sign up for our free email newsletter and receive the best stories from Smithsonian.com each week.

More from Smithsonian.com

Stop Texting While You’re Walking

How Smart Can a Watch Be?

It tells you what’s happening on your phone. And it tells time. Photo courtesy of Sony.

It’s amazing how putting a lower case “i” in front of the name of a gadget can make it righteous.

What that means, of course, is that Apple has deemed that particular piece of technology worthy of its attention. And with that comes both market credibility and geeky cool.

So when rumors started swirling a few weeks ago that Apple could unveil an “iWatch” later this year, tech writers around the Web were quick to ponder if 2013 will become “The Year of the Smartwatch.” Maybe. Maybe not. The iGod has not yet spoken on the subject. At least not officially.

The article that stirred the iWatch clamor was a recent piece by Nick Bilton in the New York Times’ Bits blog. It was high on speculation–Apple isn’t talking–and spiced with juicy questions: Will it come with Siri, the voice of the iPhone? What about Apple’s map software? Will an iWatch enable its wearers to track their steps taken? How about their heartbeats?

But the biggest tease was an allusion to glass. Specifically bendable glass. Imagine a watch face that could curve around your wrist. That sounds light, sleek and yes, geekily cool. That sounds so Apple.

The Wall Street Journal followed up, citing a source saying that Apple has been discussing the design of a smartwatch with its Chinese manufacturing partner. And then Bloomberg chimed in, reporting that Apple has a team of at least 100 people cranking away on a “wristwatch-like device.”

It also quoted Bruce Tognazzini, a tech consultant and former Apple employee: “The iWatch will fill a gaping hole in the Apple ecosystem.”

Keeping watch

So game over, right? Whenever Apple rolls out its device, it will define what a smartwatch should be, right?

Not so fast. Believe it or not, it’s already a crowded field, with more than half a dozen smartwatches out in the market. Maybe the best known, at least among gadget geeks, is the Pebble, which made a big splash a year ago, even before it existed. Its inventors made a pitch for investors on Kickstarter, hoping to drum up $100,000. Instead they raised $10 million, and a crowd-funding legend was born. The first Pebbles shipped earlier this year, to generally positive reviews.

Sony came out with its own model last year, sometimes to less than enthusiastic reviews. Others in the game include the MetaWatch Strata, the strangely-named I’m Watch,  the oddly-named Martian Passport, one called Buddy and another called Cookoo. Later this year, a model called The Pine is expected to hit the market.

But, aside from having names that you’d never imagined calling a wristwatch, what do all these products bring to modern life? Obviously, they tell time, but most also connect wirelessly to your smartphone so you can see who’s calling or texting or emailing or posting on your Facebook page without digging into your pocket for your phone. They can show you weather forecasts, sports scores or news headlines. Some have apps that let you control the music on your phone or track how far you’ve run or cycled.

And keep in mind, this is only the first wave. They probably can’t do enough yet to entice most people to shell out a few hundred bucks–they range from $130 for a Cookoo to more than $400 for an I’m Watch. But as more apps are added, they could be used to make mobile payments, navigate with GPS, take photos and shoot videos. A few already can  handle phone calls, albeit clunkily. So, the day is fast coming when you’ll be able to talk into your wristwatch without making people nervous.

Some say we’re on the cusp of a wearable tech boom, and that the smartphone, as something we need to actually carry around, will become passe. Others are more dubious, positing that the smartwatch is just another gadget phase we’re going through.

But there’s that bendable glass…

Fresh smart

It’s long been said that if you want to succeed, it helps to be smart. Now that applies to products, too.

• At last, a cure for expiration date anxiety: Researchers at the Eindhoven University of Technology in the Netherlands say they’ve developed packaging with sensors that will be able to tell if the food inside is still edible.
• When bottles share: A Florida entrepreneur thinks the time has come for medicine bottles to get smart. His idea is to put QR codes on bottles that once scanned, will play a video on your smartphone telling you all you really need to know about the meds inside.
• Let sleeping babies lie: And for anxious young parents who check every 30 seconds to see if their baby is still breathing, students at Brigham Young University are developing something they call the Owlet Baby Monitor. Using a built-in pulse oximeter, the wireless smart sock can track both a sleeping child’s heart and breathing rates.
• Say goodbye to the “You’ll just feel a little pinch” lie: Scientists at Purdue University have created bandages that could make the needle stick obsolete. Powered by a person’s body heat, the adhesive patches would be able to deliver medication without the need for a shot.
• Which is so much cooler than wearing a smart sock: In Japan, Fujitsu has unveiled its “Next Generation Cane.” Yep, it’s a smart cane and it can monitor a person’s vitals. It also comes with GPS so you can always know where Grandma’s taking a stroll.

Video bonus: Want the lowdown on how the Pebble smartwatch works? The Wall Street Journal’s Walt Mossberg lays it out a video review.

More from Smithsonian.com

Turning Your Hand Into a Remote Control

How Smart Should TVs Be?

Can drones be taught the rules of war? Photo courtesy of the Department of Defense

When John Brennan, President Obama’s choice to be the next head of the CIA, appeared before a Senate committee yesterday, one question supplanted all others at his confirmation hearing:

How are the decisions made to send killer drones after suspected terrorists?

The how and, for that matter, the why of ordering specific drone strikes remains largely a mystery, but at least one thing is clear–the decisions are being made by humans who, one would hope, wrestle with the thought of sending a deadly missile into an occupied building.

But what if humans weren’t involved? What if one day life-or-death decisions were left up to machines equipped with loads of data, but also a sense of right and wrong?

Moral quandary

That’s not so far fetched. It’s not going to happen any time soon, but there’s no question that as machines become more intelligent and more autonomous, a pivotal part of their transformation will be the ability to learn morality.

In fact, that may not be so far away. Gary Marcus, writing recently in The New Yorker, presented the scenario of one of Google’s driverless cars before forced to make a split-second decision: “Your car is speeding along a bridge at 50 miles per hour when errant school bus carrying 40 innocent children crosses its path. Should your car swerve, possibly risking the life of its owner (you), in order to save the children, or keep going, putting all 40 kids at risk? If the decision must be made in milliseconds, the computer will have to make the call.”

And what about robotic weapons or soldiers? Would a drone be able to learn not to fire on a house if it knew innocent civilians were also inside? Could machines be taught to follow the international rules of war?

Ronald Arkin, a computer science professor and robotics expert at Georgia Tech, certainly thinks so. He’s been developing software, referred to as an “ethical governor,” which would make machines capable of deciding when it’s appropriate to fire and when it’s not.

Arkin acknowledges that this could still be decades away, but he believes that robots might one day be both physically and ethically superior to human soldiers, not vulnerable to the emotional trauma of combat or desires for revenge. He doesn’t envision an all-robot army, but one in which machines serve with humans, doing high-risk jobs full of stressful snap decisions, such as clearing buildings.

Beware of killer robots

But others feel it’s time to squash this type of thinking before it goes too far. Late last year, Human Rights Watch and Harvard Law School’s Human Rights Clinic issued a report, “Losing Humanity: The Case Against Killer Robots,” which, true to its title, called on governments to ban all autonomous weapons because they would “increase the risk of death or injury to civilians during armed conflict.”

At about the same a time, a group of Cambridge University professors announced plans to launch what they call the Center for the Study of Existential Risk. When it opens later this year, it will push for serious scientific research into what could happen if and when machines get smarter than us.

The danger, says Huw Price, one of the Center’s co-founders, is that one day we could be dealing with “machines that are not malicious, but machines whose interests don’t include us”.

The art of deception

Shades of Skynet, the rogue artificial intelligence system that spawned a cyborg Arnold Schwarzenegger in The Terminator movies. Maybe this will always be the stuff of science fiction.

But consider other research Ronald Arkin is now doing as part of projects funded by the Department of Defense. He and colleagues have been studying how animals deceive one another, with the goal of teaching robots the art of deception.

For instance, they’ve been working on programming robots so that they can, if necessary, feign strength as animals often do. And they’ve been looking at teaching machines to mimic the behavior of creatures like the eastern gray squirrel. Squirrels hide their nuts from other animals, and when other squirrels or predators appear, the gray squirrels will sometimes visit places where they used to hide nuts to throw their competitors off the track. Robots programmed to follow a similar strategy have been able to confuse and slow down competitors.

It’s all in the interest, says Arkin, of developing machines that won’t be a threat to humans, but rather an asset, particularly in the ugly chaos of war. The key is to start focusing now on setting guidelines for appropriate robot behavior.

“When you start opening that Pandora’s Box, what should be done with this new capability?,” he said in a recent interview. “I believe that there is a potential for non-combatant casualties to be lessened by these intelligent robots, but we do have to be very careful about how they’re used and not just release them into the battlefield without appropriate concern.”

To believe New Yorker writer Gary Marcus, ethically advanced machines offer great potential beyond the battlefield.

The thought that haunts me the most is that that human ethics themselves are only a work-in-progress. We still confront situations for which we don’t have well-developed codes (e.g., in the case of assisted suicide) and need not look far into the past to find cases where our own codes were dubious, or worse (e.g., laws that permitted slavery and segregation).

What we really want are machines that can go a step further, endowed not only with the soundest codes of ethics that our best contemporary philosophers can devise, but also with the possibility of machines making their own moral progress, bringing them past our own limited early-twenty-first century idea of morality.”

Machines march on

Here are more recent robot developments:

• Hmmmm, ethical and sneaky: Researchers in Australia have developed a robot that can sneak around by moving only when there’s enough background noise to cover up its sound.
• What’s that buzzing sound?: British soldiers in Afghanistan have started using surveillance drones that can fit in the palms of their hands. Called the Black Hornet Nano, the little robot is only four inches long, but has a spy camera and can fly for 30 minutes on a full charge.
• Scratching the surface: NASA is developing a robot called RASSOR that weighs only 100 pounds, but will be able to mine minerals on the moon and other planets. It can move around on rough terrain and even over bolders by propping itself up on its arms.
• Ah, lust: And here’s an early Valentine’s Day story. Scientists at the University of Tokyo used a male moth to drive a robot. Actually, they used its mating movements to direct the device toward an object scented with female moth pheromones.

Video bonus: So you’re just not sure you could operate a 13-foot tall robot? No problem. Here’s a nifty demo that shows you how easy it can be. A happy model even shows you how to operate the “Smile Shot” feature. You smile, it fires BBs. How hard is that?

More from Smithsonian.com

This Robot Is a Better Dad Than Your Dad

Robots Get the Human Touch

At football stadiums today, it’s all about the screens. Photo courtesy of Mitsubishi Electric Diamond Vision

It’s the time of year when the National Football League gets a little bit smaller.

Sure, the Super Bowl on Sunday is its championship game and more than 100 million people will be watching, but if the outcome isn’t decided in the last two minutes, more people on Monday will be talking about the funniest TV commercials or how Beyonce sang–or didn’t–at halftime or the post-game homage to the Baltimore Ravens’ Ray Lewis as he dances off into the sunset.

It’s been this way for a while now. As the spectacle of everything around it has become bigger, what actually happens on the field during the Super Bowl has gotten smaller. And that’s been okay with the league as long as it’s only happened once a year.

But now, with the rise of giant home video screens and the ability to see every scoring play of every game on the NFL’s RedZone network or watch games from different angles on a computer tablet, people running the league and its teams have realized that they need to pump up the stadium experience. What happens on the field, they fear, soon may no longer be enough to keep the customers satisfied.

Hitting the big, big screen

No question that the Dallas Cowboys ratcheted things up in 2009 when they opened, with much hoopla, the new Cowboys Stadium. Not only did it cost more than $1 billion, but hanging 90 feet above the field is an HDTV screen so large–it stretches from 20-yard-line to 20-yard line–that players who are quite massive in real life look like little Lego men moving around below.

Next fall, the Houston Texans will one-up the Cowboys when they unveil their own field-dwarfing video screen, almost 25 percent larger than the one in Dallas. And now even colleges are starting to join the monster screen club. The University of Nevada, Las Vegas, hardly a football powerhouse, just released plans for a new stadium that will include a video screen 100 yards long.

That’s right, it will be as long as the playing field.

Stand up and cheer

Okay, so we can expect the screens to get bigger and bigger. But some think the stadiums may actually get smaller, or at least there will be fewer seats. Instead, more attention will be paid to where people can stand and what they can do while they’re there.

Here’s how Eric Grubman, the NFL’s executive vice president of business operations, described a football stadium of the future in a recent interview with the Los Angeles Times:

“What if a new stadium we built wasn’t 70,000, but it was 40,000 seats with 20,000 standing room? But the standing room was in a bar-type environment with three sides of screens, and one side where you see the field. Completely connected. And in those three sides of screens, you not only got every piece of NFL content, including replays, RedZone and analysis, but you got every other piece of news and sports content that you would like to have if you were at home.

Now you have the game, the bar and social setting, and you have the content. What’s that ticket worth? What’s that environment feel like to a young person? Where do you want to be? Do you want to be in that seat, or do you want to be in that pavilion?”

Phoning it in

Other stadium innovations are heading in a different direction. Instead of having the game be only part of a multi-screen, sports bar party experience, they would entertain fans by allowing them to immerse themselves more deeply into the game itself. And they would do it all on smart phones and tablets.

Take the case of the New England Patriots. At the beginning of this past season, they became the first NFL team to deploy a free Wi-Fi network for streaming video in their home field, Gillette Stadium. Fans were able to use mobile apps to watch instant replays on their phones and get real time stats.

And next season, they’ll have more options, ones that take them into the games within the game. There will be apps that allow them to tune into cameras following star players around, apps that let them watch what goes on in their team’s locker room at halftime, apps that listen in on players wearing microphones and eavesdrop on conversations between the coaches and the quarterback (with a 15-second delay, of course).

And there will an app that, by the fourth quarter, could be the most valuable of all. It will tell them where to find the shortest bathroom lines.

Wearing protection

Here are other recent advances in football tech:

• A red zone you don’t want to enter: Reebok has developed something it calls a Head Impact Indicator. It’s a thin skullcap lined with sensors that can detect dangerous hits to the head. If a yellow or red light goes on, it’s time for a player to head to the sidelines.
• Now if they could only do something about helmet hair: Meanwhile, engineers at Purdue University say they’ve developed the model for a football helmet that disperses the energy of a smack to the head instead of just protecting a player’s skull. They report that tests with a polymer-lined Army helmet they designed showed it could reduce the G-force a player’s brain absorbed by as much as 50 percent.
• Like we need another reason to boo the refs: You know that imaginary yellow line you see on TV games to show where the first down marker is? After this season, the NFL is going to take a look at technology that would project a laser line across the field so people in the stadium could see what everyone at home has been seeing for years.
• Hardbodies the easy way: When they run out on the field Sunday, four San Francisco 49ers players, including both of the team’s quarterbacks, will be wearing a form of customized body armor under their uniforms. It’s called EvoShield and it’s a gel that hardens to fit a player’s body when exposed to air.

Video bonus: Okay, here’s a sneak peek of two Super Bowl ads already being declared winners, a spot about how getting the keys to the family Audi jacks up the testosterone of a boy headed to his high school prom, and a Volkswagen ad using a Minnesotan-turned-Rastafarian to celebrate the power of German engineering.

More from Smithsonian.com

How Did Avocados Become the Official Super Bowl Food?

The Super Bowl’s Love Affair With Jet Packs

The HapiFork wants to make you less piggish. Photo courtesy of HapiLabs.

Utensil history was made last week and I, for one, took pleasure in seeing that we had finally evolved beyond the spork or, as some of you may know it, the foon.

But sadly, the unveiling of the HapiFork at the Consumer Electronics Show (CES) was not universally greeted with great jubilation, but rather with a fair amount of ridicule.

Produced by a Hong Kong company called HapiLabs, the HapiFork is curious little thing. It looks like a fork and works like a fork, but it vibrates like a cellphone. And why it buzzes is the reason the media largely responded with one big group eyeroll.

See, the HapiFork is a fork with a simple and noble mission–to get you to stop eating like a pig. It buzzes to remind you to slow down.

It tracks not only the number of bites you’ve taken, but also how much time has passed between them and how long it takes you to finish the meal. The slower you eat, the fewer calories you consume. And because all the data can be stored on your smart phone, you can measure how less a chowhound you’ve become.

But some critics were not enamored of the concept, portraying the HapiFork as the essence of nanny technology, another “smart” gadget enforcer of data-driven moderation. How, the thinking goes, did it come to this, where forks are telling us to shut our pieholes?

The measure of a man

But maybe, given the obesity epidemic in the U.S. and Europe, it’s time to start listening to buzzing silverware. In fact, there are those who believe the current boom in mobile apps and devices that track our health and bad habits could play a big role in helping the U.S. get its outrageous health care costs under control.

A major health trend this year, according to a new report from PricewaterhouseCoopers, will be a shift by employers and insurance companies to encourage employees to be a lot more proactive when it comes to taking care of themselves. That’s in part due to incentives in the Affordable Care Act, but also because today’s technology–whether it’s sensors, WiFi or smart phones–has made it so much easier to track every move we make, every breath we take.

We’ll likely see more companies turn to employee wellness programs focusing on prevention and tapping into all that data that our smart phones and other health gadgets are able to gather about us. Already, start-ups such as the Boston-based Healthrageous are being hired by companies to work closely with their employees with chronic conditions, such as diabetes or hypertension or even sleep disorders. Healthrageous provides both a tracking device–say a blood glucose monitor for diabetics–and a customized plan to help employees reach their personal goals, which could be anything from fitting into pants you last wore 10 years ago to being able to play with your grandkids.

PUSH Wellness, in Chicago, also contracts out an employee wellness program, but with a different spin. It actually pays cash incentives to workers who meet goals that raise their “PUSH” score–a number based on a person’s Body Mass Index (BMI), blood pressure, cholesterol and fitness level. With PUSH, it’s not enough for an employee to exercise; they have to show real measurable results or there’s no pay out.

The big health insurance companies are getting in on the act, too. Last month, Aetna unveiled Passage, a fitness app it developed with Microsoft, that allows people to feel like they’re running or biking in some of the world’s great cities–Rome, New York, or Barcelona, for instance.

Also last month, Cigna announced that it has made available, for free, to the first 20,000 people who download them, four apps bundled together as the “Healthy Living App Pack. One is designed to track your workouts, another to get you to relax, another the help you sleep. The fourth, Fooducate, is a food nutrition app designed to make you health savvy when you’re food shopping.

When sensors speak

Here are five other health devices that made a splash at CES last week:

• Would your wrist lie to you?: Another health wristband is coming on the market soon. Called Fitbit Flex, it will be able to track your daily activity–steps taken, calories burned–and also how you’ve slept, plus wake you up with a little buzz in the morning. For motivation, a display of four LED lights shows how far along you are in meeting that day’s goal. And at $100, it will be less expensive than the competitors already out there, Nike Fuel and Jawbone’s Up.
• Keep running or we’ll play “Gangham Style:” Or you can let little earbuds do the monitoring work. Coming out this spring are iRiver On headphones equipped with PerformTek Precision Biometrics technology that measures a range of body metrics, including heart rate, distance traveled, steps taken, respiration rate, speed, metabolic rate, energy expenditure, calories burned and recovery time.
• It was so much easier when pills looked like the Flintstones: For those dealing with a daily dose of multiple meds, there’s the uBox. The little box reminds people when it’s time to take their pills with a combination of beeps, blinking lights and smart phone reminders. And if you’ve already taken your meds, the box remains locked until it’s time for another set–the better to keep forgetful seniors from double dosing. It even lets other family members know if grandpa’s missed a med.
• Giving new meaning to “Let me hear your body talk”: Then there’s Metria, a small patch a person wears on their chest that measures heartbeat, skin hydration, breathing, steps taken and sleep patterns. (It records the duration and quality of sleep based on how much you’ve tossed and turned.) Each patch gathers information for seven days and can send it to a phone or tablet anywhere in the world. Metria’s designed primarily for elderly people who live alone, but the U.S. Air Force reportedly may use it to monitor pilots.
• Will walk for prizes: And bringing us back full circle to obesity is the ibitz PowerKey, a pedometer for kids. It doesn’t just track their activity, but rewards them with games, apps, shows and prizes for staying on the move. And yes, parents can check in on their kids’ progress on their own smart phones.

Video bonus: See why Stephen Colbert thinks the HapiFork is “unAmerican.”

More from Smithsonian.com

Take Two Pills and Charge Me in the Morning

Me, My Data and I

Samsung TVs get smarter. Photo courtesy of Samsung

Since the beginning of mankind, we’ve wanted our kids to get smarter. Since the beginning of the 21st century, we’ve wanted our phones to get smarter.

So when are we going start wanting our TVs to get smarter? Or will we always be content with them being dumb, as long as they’re big and dumb? Okay, maybe not dumb, but most of us don’t yet feel a compelling need to have our TVs think like computers, as long as the picture looks pretty up there on the wall.

Which always makes things interesting at the Great Gadgetpalooza also known as the Consumer Electronics Show (CES). For the past several years, the big electronics companies that focus on hardware, such as Samsung and Panasonic, and the big tech companies that focus on software, such as Google, have been rolling out nifty products at the annual Las Vegas spectacle with the promise that this is the year that Smart TV goes mainstream.

Boob tube no more

And so it’s been at this year’s version of CES, which ends today. Samsung has done its part to convince us that the time has come for us to love TVs for their brains by unveiling what it calls its S-Recommendation engine.

It’s software that, as Samsung puts it, not only understands what you like, but recommends things it thinks you’ll like. (Sure, Amazon’s been doing this for years, but this is your big, dumb TV we’re talking about.) And it doesn’t just suggest TV shows, but could throw in streaming programs options from the Web, or even video you’ve shot on your smartphone.

The goal ultimately is to get you to do all those things you’re now doing on your smartphone or your tablet–say, watch Hulu or Skype with a family member or check out your Facebook page–on your TV instead. To encourage that behavior, Samsung has revamped its Smart Hub so you can flip through all of your entertainment options in five different index screens–one that tells you what’s on regular old TV now or soon, another that lists movies and on-demand TV, a third that pulls in photos or music or video stored on any other devices around the house, a fourth where you can Skype or pull up Facebook and a fifth that provides access to any apps you’ve downloaded.

And neither of the above requires pushing a lot of buttons on a remote. The S-Recommendation engine responds to voice commands and the Smart Hub is designed to be controlled with hand gestures.

For its part, Panasonic has rolled out a feature it calls My Home Screen, which allows each member of your family to create his or her own homepage on the TV, where easy access is provided to their favorite digital content, streaming video and apps. Some of the company’s Viera models actually come with their own cameras that tell the TV who turned it on. And as a smart TV should, it dutifully brings up that person’s home screen.

Plus, Panasonic unveiled “Swipe and Share 2.0″, which lets users move photos from a tablet or phone to a big TV screen, where they can then be edited with a touch pen.

But can you love a TV?

So that seals it, right? This must be the year when TVs take back center stage, especially now that they’re finally learning to care about our needs, right?

Maybe not. We’ve built some pretty strong personal connections to our cell phones and tablets. And a lot of people think it’s going to take a while for us to develop that kind of bond with a TV, no matter how smart it is.

As Greg Stuart, CEO of the Mobile Marketing Association told Ad Age earlier this week: “”People don’t have that kind of interactive relationship with their TV. The TV on the wall is a family device. It’s a multi-user device. If I want to share something, its going to be with a personal device, and that’s going to be my tablet or my mobile.”

TV or Not TV?

Here are other recent TV innovations:

• Robert, 6th Earl of Grantham, meet Tony Soprano: One day, thanks to Samsung, two people will be able to watch full-screen versions of Downton Abbey and Sopranos reruns at the same time. By adapting 3D technology, the company has created a TV that can display a different and full resolution image to each viewer depending on whether they’re sitting to the left or the right of the screen. Of course, both people would have to wear special glasses that come with headphones so you can hear only the sound for your show, but is that such a big price to pay for domestic peace?
• Read my lips. No more Gangham style: LG, the other South Korean TV giant, has upgraded its “Magic Remote” so that it now responds to natural language. You say the name of a show or even something like “videos with Gangham-style dancing,” and your choice pops up on the screen.
• I got my MoVo workin’: Also at CES, the Chinese TV manufacturer TCL showed off an HD TV called MoVo that uses facial recognition software to identify who’s watching and then make programming suggestions customized for that person.
• Okay, who blinked?: Meanwhile, Haier, another Chinese company, has developed a technology it calls Eye Control TV where, yes, you can change channels by moving your eyes.
• Ah, to be 65 and only see ads for meds: It was only a matter of time. A company called Gracenote will soon begin trials on a technology that, based on your viewing habits and personal data, will personalize the TV ads you see. Isn’t that special?

Video bonus: You didn’t make it to the big electronics show this year? Not to worry. Here’s the Samsung demo of its S-Recommendation engine. Remember, people tend to gush a lot at CES.

More from Smithsonian.com

Next Up? The Smart Watch

Getting Smart About Traffic

Nanshu Lu’s electronic tattoo that reads your vital signs. Photo courtesy of Nanshu Lu

In the spirit of the post-holiday season, allow me to present my final list of 2012: six innovators who are pushing technology in fresh directions, some to solve stubborn problems, others to make our lives a little fuller.

Watch for more from all of them in the new year.

1. Keep your hands off my robot: We’ve all seem videos of adorably cute robots,, but when you actually have to work with one, they apparently can be less than lovable. That’s where Leila Takayama comes in. She’s a social scientist with Willow Garage, a San Francisco area company that develops robots, and her job is to figure out how to get humans to connect with mechanical co-workers.

She’s seen cases where robots have gotten on people’s nerves so much that they park them in a closet. One of the keys, she’s found, is to make robots seem more fallible. Like having them shake their heads when they fail at something. Oddly enough, Takayama says, a reaction like that can make a robot “seem more competent.”

She’s worked on robots designed to help elderly people, recommending that the number of cameras on the robots’ heads be reduced because too many could make people uneasy. More recently, she’s been analyzing a robot called Project Texai, which is operated directly by humans, rather than running on its own. And she’s discovered some interesting things, such as how people who operate the robot don’t like it when other people stand too close to it or touch its buttons. “There comes a point for a lot of people when they feel as if the robot is their body.”

Another key question she’s wrestling with: Is it better to have a robot at eye level with a person when he or she is sitting or standing?

2. One day even lamp posts won’t be dumb: As Chris Harrison sees it, the world is full of surfaces, so why are we spending so much time touching little screens or tapping on cramped keyboards. Harrison, a researcher at Carnegie-Mellon University, has been a leader in finding ways to turn everyday objects–a couch, a doorknob, a glass of water–into interactive devices.

His approach is to use the natural conductivity of objects–or attach electrodes to those that aren’t–and connect them to a controller that responds to different types of signals. A couch, for instance, could be wired to turn on the TV if someone sits on it in a certain spot. Or you could turn off all the lights in your place by twisting the doorknob or tapping on a table. Almost anything with a surface could be connected to a computer and allow you to make things happen with simple gestures or touches.

3. Finally, a tatt for Grandma: There’s no questions that health tech is booming–although that’s not always a good thing considering that health apps don’t always live up to their hype. But Nanshu Lu, an engineering professor at the University of Texas, has created a product that could have a huge impact on how we monitor what’s going on inside our bodies.

She has refined what are known as “epidermal electronics,” but basically they’re electronic tattoos that can track your vital signs, including your temperature, heart beat and brain and muscle activity. Lu has managed to develop ultra-thin, water-soluble silicon patches that contain tiny sensors and can actually bond with skin. No adhesives necessary. They last through showers and exercise, never losing their ability to gather your most personal data. The hope is that one day her tattoos will be able to treat diseases.

4. In phones we trust: When you’re out on the road or on vacation in a new place, it can get frustrating to have to search for info on your smart phone. Really, if your phone is so smart, shouldn’t it be able to anticipate your needs and feed you info as you need it, based on where you are and what time of day it is?

That’s the premise behind the mobile apps software developed by Flybits, brainchild of Hossein Rahnama, director of the Digital Media Zone at Toronto’s Ryerson University. Flybits is already being used at several Canadian airports and Toronto’s transit system to coordinate with a traveler’s itinerary and provide information that’s both personalized and contextually relevant, such as directions to the car rental counters or the gate to your connecting flight after you get off a plane.

The company has also developed software it calls Flybits Lite, which lets you know friends and other contacts who are taking in the same concert or watching the same movie you are.

5. Do you really want to know how many times you’ve ordered donuts?: It would be easy to dismiss the Memoto Camera as the epitome of 21st century self-indulgence. It’s a postage-stamp sized wearable camera that documents your life by taking two photos every minute, or roughly 2,000 pictures a day.

For most of us that’s one big load of digital tedium. Martin Kallstrom, the man behind the concept and CEO of the Swedish startup Memoto, would acknowledge as much. But he also knows how many memorable moments are missed–”the day your daughter took her first step, or that night you laughed the night away with friends.”

Clearly, he’s not alone in believing that a “lifelogging” camera is an idea whose time has come. He and his partners had hoped to raise $75,000 on Kickstarter. By the time the fundraising campaign ended earlier this month, online backers had pledged more than $550,000.

6. And no, it won’t fetch you a beer: For several years now, Steve Castellotti has been all about brain-powered machines. But his latest innovation, Puzzlebox Orbit, is taking the concept to the public. It’s a little helicopter you control with your mind.

Given that this is not something we do every day, it comes enclosed in a protective sphere so the rotor blades don’t chop up the furniture. It also comes with a device called the Puzzlebox Pyramid, which serves as a combination base/remote control unit for the chopper. But since your mind is doing the controlling, the Pyramid’s role is to wirelessly transmit your brain activity from a headset you wear. It also lets you know how you’re doing–a circle of LED lights on the Pyramid’s face is designed to reflect your level of concentration or relaxation.

Thanks to a funding boost from Kickstarter, Castellotti and his chief engineer and partner Hao Zhang plan to start selling the Puzzlebox Orbit for about $90 next year. But Castellotti believes it won’t become just another pricey tool that ends up in the basement. He sees it as teaching tool that can be used in schools to introduce kids to neuroscience and also as a way for people for people to start to become familiar with the potential of biofeedback.

To spur that process, the company will make its source code and hardware schematics available and encourage developers to hack away. For example, says Castellotti, a “motivated experimenter” might hack the Puzzlebox system so his TV would automatically change channels when his concentration level stays too low for too long. Say so long to vegging out.

Video bonus: Take at look at Chris Harrison’s most recent project, called Skinput, It involves the use of an armband with bio-acoustic sensors that can turn a body into a touch screen.

Video bonus bonus: And here’s a Puzzlebox Orbit tutorial that was part of the Kickstarter pitch for its nifty brain-controlled toy.

More from Smithsonian.com

The Best Inventions of 2012 You Haven’t Heard of Yet

Innovators to Watch in 2012

Finally, a suitcase that follows you around. Photo courtesy of Hop!

Earlier this week I rolled out the first half of a list of a dozen of the more innovative ideas of 2012.

We’re not talking Google Glass or invisibility cloaks or other flashes from the future. No, these are less splashy things, yet, in their own ways, no less inspired and probably more likely to become a part of our daily lives. They’re the creations of people joined under a common maxim, namely, “There’s gotta be a better way.”

So, muffled drum roll, please…the Fresh Ideas of 2012, Part 2:

7) While you’re at it, can you pick up a paper and some gum: Yes, suitcases with wheels were a big breakthrough, but Madrid designer Rodrigo Garcia Gonzalez says why stop there? Why should we still have lug luggage?

So he has invented a new kind of suitcase he calls Hop!. What makes it so special is that it follows you around like the most loyal of pets.

Well, technically it follows your smart phone. The suitcase contains three receivers that communicate, via Bluetooth, with an app on your smart phone and, put simply, it follows that signal. The same controller also directs a dual caterpillar track-type system on the bottom of the suitcase to move it along. If the signal gets lost, the bag locks itself and vibrates its owner’s phone.

Of course, there are issues to resolve–think of the security challenges of an airport full of roaming luggage–but Gonzalez deserves props for giving us hope that we’ll one day break loose from our bags.

8) Anticipation was so overrated: It never really made much sense: In a world increasingly geared to instant gratification, we waited for ketchup. It took forever to come out of the bottle, but we seemed willing to live with that.

Not Kripa Varanasi and his team of MIT engineers. They’ve developed a substance called LiquiGlide, which, when coating the inside surface of bottles, helps ketchup and mustard slide right out. Now this may seem a trivial modern indulgence, but, as the LiquiGlide team estimates, roughly a million tons of food could avoid being tossed in the garbage if it wasn’t getting stuck in bottles. So it’s only right that we go with the flow.

9) Which gives new meaning to “All you can eat”: While we’re on the subject of food waste, let’s give it up for WikiCells. These are the edible membranes created by Harvard professor David Edwards and French designer Francois Azambourg that encase food and liquids. In other words, it’s packaging you can eat.

The membranes, meant to mimic the skin of a grape, are made of food particles, such as cheese or dried fruit, and are held together by calcium or magnesium ions. So far, the pair have conjured up a tomato membrane containing gazpacho, an orange one filled with orange juice, a chocolate version holding hot chocolate. They’ve even created a grape-flavored pouch filled with wine. The goal is to do away with plastic bottles and packaging. Let’s raise our membranes to that.

10) Talk to the glove: Four Ukrainian students have designed gloves that can communicate with a smart phone and, as a result, developed a way for people with speech and hearing disabilities to talk to people who don’t use or understand sign language.

Their invention, which they call Enable Talk, works like this: The gloves are lined with 15 flex sensors in their fingers that can recognize sign language and transmit the message to a smart phone where it’s converted to text. The phone then says the words that the gloves sent.

11) So now we can stay focused on not changing the oil: If you’re like me, you have no idea when you last checked your tire pressure. It’s a blind faith thing. As long as the tires keep rolling, no need to look for that little gauge you bought many tires ago.

Goodyear understands this so they’ve gone ahead and invented a tire that takes us out the equation. It does this by inflating itself. A regulator in the tire senses when the pressure drops below a pre-set point and opens to allow air flow into the pumping tube. As the tire rolls, deformation flattens the tube, pushing air into the tire cavity. And we’ll no longer have to worry about keeping our tires pumped up. Not that we ever did.

12) No longer will a charger come between you and your phone: A few years ago the 11-year-old daughter of Wake Forest University scientist David Carroll wondered aloud if a cell phone could be charged solely by human body heat. Good question, thought Carroll, and earlier this year he and his team came up with an answer.

They unveiled Power Felt, a fabric that uses nanotechnology to convert heat into electricity. It’s still in the early stages of development but initial results suggest they’re on to something big–an inexpensive material that could use the heat from your car’s engine to run its AC and radio and the sun to power your home’s appliances and yes, your own personal warmth to keep your cell phone alive. Thanks, Dad.

Video bonus: While we’re talking about nifty ideas whose time is about to come, it’s pretty certain that Samsung will come out with a bendable phone next year. As you can see in this video, it passes the hammer test.

More on Smithsonian.com

10 Bright Ideas to Get You Through February

Are Machines Dumbing Us Down?