Government - Written by on Wednesday, February 3, 2010 12:28 - 1 Comment

Naumi Haque
Analyzing the State of the Union: Speeches as data points

Last week President Obama addressed the nation in his second State of the Union. Analyzing these speeches has been an interest of mine for some time, but I’m struck by how much better the analytics tools have become. Even if you don’t care about the State of the Union, it’s interesting to see how words, texts, and public response have become data that is now easily accessible and measurable. Speeches are meant to move, inspire, and articulate a vision. To view them as simple data points may seem crude to some, but the latest informatics capabilities are actually used to record emotional response—how inspiring was Obama?

When I originally started looking State of the Union addresses, I simply found transcripts online and did a manual count of words in text documents. This was laborious, but provided some interesting findings (note sites like Speech Wars can now automate this process). Last January I highlighted Wordle and used tag clouds to create a visualization of State of the Union addresses from notable past Presidents. This year, I’ve been spending a fair bit of time researching sentiment analysis, so I was pleasantly surprised to see that vendor Crimson Hexagon and CNN had teamed up to analyze public sentiment towards the 2010 State of the Union in real-time. Check out the video after the jump.

 

The impact of the new technology was not lost on the news media. The Huffington Post picked up the story and reported that, “The moment that ends up being most pivotal in changing the way the media covers big, live events may well have happened on CNN, where John King used the ‘Magic Wall’ to analyze almost 150,000 Twitter responses to President Obama’s speech.” In the article, CNN’s Senior Vice President and Washington Bureau Chief, David Bohrman is quoted as saying, “Twitter is all noise, but to be able to harness it and group it and actually intelligently cluster it and derive moods and opinions from it is very interesting.”

Whatever you might think of Twitter (Jon Stewart used the Magic Wall as an opportunity to make fun of both CNN and Twitter), this is exactly the type of technology companies are starting to think about for managing their brands, conduct market research, and pre-emptively deal with customer issues. The next level of granularity that sentiment analysis vendors are starting to offer is the ability to go beyond positive and negative sentiment to look at why sentiment is the way it is. Why are people pro-Obama? What types of issues are most often related to “Obama is too liberal?” This type of analysis is available, and I’ve seen demos from some vendors that offer fairly sophisticated drill-downs. However, some people remain sceptical about the general accuracy of this capability, as well as the limitations of most systems to crunch this type of data in real-time. Maybe we’ll see this for next year’s State of the Union—I’m hoping so.



1 Comment

You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.

Kamal Latif
Feb 13, 2010 19:44

Fascinating article Naumi. Even I find my list of tweets is often nothing more than noise, but what is interesting is noting the general trend in the words used, and subjects discussed in all the tweets. When Google Buzz came out or the iPad was launched a diverse range of the people I follow on Twitter commented on both events and there were definite groups of responses, in terms of words and sentiments. Using this type of data/ word analysis to generate meaningful information on the feelings of people/ customers towards a certain event, be it a political speech or product launch will definitely become more vital for any person or organisation promoting or maintaining a brand. It is the next level on from what I have often worked on as a financial analyst mining through sales data in data warehouses to uncover trends in consumer spending and habits.

What is really interesting is linking the trends in word use with the personalities or digital/ online profiles of the people making the comments. I would be interested, for example, if pro-Democrat party Americans used similar words or displayed similar sentiments as each other on Twitter or blogs, and if that differed significantly from the language and sentiments voiced by pro-Republican party commentators.
The language of different groups and the way they project their opinions will be something politicians and businessmen would definitely want to understand and play on.
Great stuff.

Now available in paperback!
Don Tapscott and Anthony D. William's latest collaboration, Macrowikinomics: New Solutions for a Connected Planet. Learn more.

Business - Oct 5, 2010 12:00 - 0 Comments

DRM and us

More In Business


Entertainment - Aug 3, 2010 13:14 - 2 Comments

Want to see the future? Look to the games

More In Entertainment


Society - Aug 6, 2010 8:19 - 4 Comments

The Empire strikes a light

More In Society