- A new computer program can identify sarcasm in online messages.
- The program is about as effective as a real person with bad social skills.
- It can help people avoid confusion in a medium where there are normally few social cues.
Sarcasm can be difficult to convey over electronic messages where social cues (like facial expressions) are few. Sarcasm is a useful way to blunt the impact of criticism by adding a twist of snide humor, as in this recent tweet: "Really love when the scope of a project I've been working on for a week changes in 10 min #sarcasm."
Without the help of a hash tag, though, people often miss the irony, especially when it's delivered online, where there are no contextual hints or social cues. To cut through the confusion, researchers have developed a computer program that can identify sarcasm in online communities with an accuracy rate of about 80 percent.
While there is still a long way to go before computers will be able to understand all the subtleties of humor, the new work might, among other practical applications, help companies sort through comments about their products to find out what customers really think.
Consider, for example, a website that allows users to post their opinions about the products it sells. One comment says, "The size of this camera is great. It fits right into my pocket." Another says, "The size of this camera is great. I need a porter to carry it."
"A typical summarization system will conclude that people are very, very happy with the size and weight" of the camera, said Oren Tsur, a computer scientist at The Hebrew University in Jerusalem. "Obviously, this is not the case."
Funny enough, Tsur's attempt to program a sarcasm detector started as a joke.
When he was a University freshman, he received an email that thanked him for his previous contributions to an annual conference and asked him to be that year's program chair. The email was clearly intended for someone else, but Tsur thought it was funny, so he responded with what he thought was an obviously sarcastic tone. His reply was taken seriously.
"They allowed me to postpone the deadline for submission and asked me what I was working on," Tsur said. "I wrote back that I was working on detecting irony in email. They didn't get that either."
Since then, Tsur's interest in sarcasm had turned serious. This week he plans to present a paper about his work at a meeting of the Association for the Advancement of Artificial Intelligence in Washington, D.C.
His program uses a strategy called "machine learning." To begin, he and colleagues fed the computer 80 sarcastic sentences and several hundred non-sarcastic sentences that they had plucked from Amazon user reviews. Sarcastic comments included: "Trees died for this book?" and, for a smart phone: "All the features you want -- too bad they don't work!"
The program analyzed the sentences and created hundreds of patterns that it used to evaluate a total of Amazon 66,000 reviews for 120 products sold on Amazon.com. Each review contained an average of 15 sentences. One of the patterns it figured out, for example, was that sentences that start with "I guess" and end with an ellipsis are often, though not always sarcastic.
To test how well the program was working, the researchers gave 200 of the same product reviews to three independent reviewers. Results showed about an 80 percent agreement between computer and humans. Given a few million tweets, the program performed with a similar level of accuracy.
The program's performance is still far from perfect, probably because sarcasm is such a complicated social construct, said Katherine Rankin, a neuropsychologist at the University of California, San Francisco.
"That's about as good as a person with bad social skills would do," she said. While such a program might be good enough to help rank reviews, she added, "if the purpose of having a computer program recognize sarcasm is to be like prosthetic for people with poor social skills, I'm not entirely sure those people are really going to benefit .
To truly get interpret a comment like "Oh, I LOVE working on Saturdays," Rankin said, people usually need to know something about the context of a situation and the person who's talking. Cues like eye rolling and a lilting tone of voice help. None of those are available in online communities.
"Our brains pick up complex social cues and process many subtle things," she said. "Computers are nowhere near getting there."
