A recent article by Nate Anderson in Ars Technica highlights professor Mayer-Schönberger book, Delete: The Virtue of Forgetting in the Digital Age. The message: “Technology has now made ‘remembering’ the default approach to information, and in doing so, threatens to make ‘forgetfulness’ obsolete.” This is not only a profound change from 20 years ago, it can also be detrimental to our ability to think and analyze information. The article goes on to say: “Selective forgetfulness is a boon to humanity; it keeps us from drowning in our own recorded data. It allows us to sift and sort, then to think at a higher level of abstraction instead of wallowing in detail.”

But, this may all soon change.  Perhaps, computers can learn to forget too.

Researchers led by doctoral candidate Roxana Geambasu, at the University of Washington in Seattle are working on project called Vanish. The idea is to encapsulate data such as e-mails, selected text in messages, or documents that are sent over the Internet. The system would create corresponding keys for decapsulation that are widely available online, but that would deteriorate over time so that the data in readable form would only be available for a certain period of time. The overview page of the Vanish project states, “We strongly believe that realizing Vanish’s vision would represent a significant step toward achieving privacy in today’s unforgetful age.” Mayer-Schönberger suggests a similar solution that uses metadata to tag data objects with expiration dates and cites the work of Lawrence Lessig who has proposed a broader approach to combine policy and software to force privacy compliance.

nGenera’s research project Leading in an Age of Unbounded Data is looking at new sources of data available to the enterprise and how these will lead to new insights, opportunities, and challenges, as well as change enterprise processes and decision-making. One of the assumptions we make is that data will continue to grow and companies, through analytics, will develop a type of ‘sixth sense’ or situational awareness about the organization thanks to information captured from across the business ecosystem. We have already found that the growth of personal information and digital identity data will lead to rich digital profiles containing social graph information. These rich profiles present opportunities to better engage with customers and employees, improve customization, and facilitate knowledge management by anticipating user needs and connecting them to relevant people and information.

Projects like Vanish force us to think about data, not as an asset with an indefinite lifespan, but rather as something that depreciates over time, just like physical assets do. This would effectively reduce the amount of data that we need to manage and improve signal-to-noise ratio as more important facts and information would be retained while less significant information would be deleted. By eliminating the perfect memory of computers, we might also feel less pressure to maintain digital facades and manicure our online profiles. Additionally, the idea of adding expiration dates and metadata to data could accelerate the shift in power away from marketer towards consumer as it would allow individuals to dictate what personal data is used, who has access, for how long, and for what purpose.

But, self-destructing data would also diminish the value of many of the ‘big data’ opportunities that we talk about such as using large data sets to infer the truth about various situations, and using sentiment analysis to mine online customer comments and status updates for market research and product insights. It would confound companies and marketers that store petabytes of information to generate longitudinal trends and rely on usage data to drive Web analytics and build reputation and ratings, as well as improve information management through technologies such as collaborative filtering (e.g. the technology used by Amazon to recommend books to you based on the activity of people with similar behaviors). By collectively deleting our less-than-favorable digital trails, would we also be doing a disservice to future generations of anthropologists that could benefit from a complete digital history and behavior map—both good, bad, and questionable actions—of their ancestors?

The idea that all data should live on forever is a relatively new concept that many people have already taken for granted. In general, I think enterprises, governments, and individuals would benefit from more discussion on the topic instead of seeing it as a foregone conclusion. The idea of having an information lifecycle for all data is a powerful one. Personally, I would welcome more initiatives such as those by the Vanish team and professor Mayer-Schönberger that broach the topic and reintroduce a little forgetfulness into our digital lives.

I’ve written a couple of posts about the GreenXchange and a related sustainability workshop, in the run-up to Davos. What I haven’t talked about is an interesting behind-the-scenes aspect of the GreenXchange technology platform.

The technology platform itself is currently a working beta version, proving the GreenXchange concept, developed by Salesforce.com (with help from a Force.com partner), 2Degrees, and nGenera. You can view the “public facing” part of the GreenXchange at http://greenxchange.force.com.

What you won’t see on the website is a sophisticated “listener” capability that the GreenXchange platform provides to its founding members, through nGenera’s collaboration server.

Listening isn’t a new concept. Among its simplest forms is a stored search that gets “pushed” to you via e-mail or other messaging. Think Google Alerts, as an example. But, for modern brand managers, communications professionals, and marketing executives, listeners have become increasingly important applications, because of the ability for reputation to be shaped by parties external to your company, from your customers to your competitors.

In fact, listeners have become part of what is recognized as a “missing layer” of social media and collaborative application infrastructure. This layer of collaborative infrastructure fills the gaps left uncovered between the traditional IT systems management products, from vendors like IBM and HP, and the point solutions vendors of collaboration platforms, as shown in the figure.

In the case of the GreenXchange launch at Davos, we employed listeners primarily for the purpose of an external monitoring activity, to track, rate, and analyze the sentiment of the articles, posts, and other communications that were being generated by the announcement.

Specifically, we wanted to analyze the spike in activity found on the web pre-Davos and post-Davos. So, a listener was created to listen for the combination of “GreenXchange” AND any of the following words or phrases: World Economic Forum, WEF, Davos, or Annual Meeting.

The listener was first run on 1/22/2010; however, we exclude results from the first run because it pulls results from very early time periods and typically finds a very large number of web clips. We ignore the first run and measure velocity on the second run forward, so that it will provide meaningful metrics.

The next screen shot demonstrates the distribution of keywords based on the listener criteria. It is showing that GreenXchange and Davos have the highest key word matches.

This figure also shows that we had a total of 900 unique web clips that had been found since we ran the listener on Jan 25^th that matched our criteria. It also shows that we had a total of 995 occurrences, meaning that there was some duplication of web clips across various channels.

The next screenshot, in the figure below, shows us the velocity between runs. What is interesting is that 34 unique web clips were found on or before Jan 26^th prior to the launch and over 935 new web clips were found since the launch on Jan 27^th, as of Feb 2nd.

This screenshot also shows the top scoring or most popular content (articles), including articles posted on websites like businessweek.com and eqentia.com

The next screenshot, in the figure below, shows an example of the web clips that were retrieved from various channels, such as Yahoo, GoogleNews, and Bing. The score indicates our confidence that this article matches the search criteria and that this content is relevant.

The sentiment shows our confidence that this web clip is either positive or negative in sentiment. All but one in the example, appear to be positive with varying degrees of confidence. This sentiment can be corrected, with the sentiment engine adapting and becoming more accurate over time.

The web channel, publish date, domain, and keyword match are also listed. And we assign a status so that you can easily filter on new content that was discovered on the most recent listener run.

The next figure simply demonstrates how convenient it is to click through the listeners results, directly to the source content. In this example, one of the articles that came up as popular content was from businessgreen.com entitled “Just share it – top brands usher in era of green co-opetition.”

The next figure further shows this click through feature, invoked by clicking on the magnifying glass to the right of the article link and view the content within the listener.

After previewing the web clips, you can decide to add them as content to your collaboration platform and request that others inside your organization, or who have been granted secure access to it from outside your organization, can comment or collaborate on the content. This action is taken by clicking the “Add” link next to the magnifying glass.

Lastly, the figure below shows an activity history. For the GreenXchange listener illustrated, the first run occurred on Jan 22^nd, with 141 web clips returned to build the baseline. But as mentioned earlier, we ignored that result set, for velocity tracking and trending going forward.

For the first several days after the GreenXchange launch, we found over 100 new, unique web clips during the daily runs, on average, with some tapering off in recent runs.

As I mentioned at the beginning of this post, social media listening is just one component of the “missing layer” of collaborative infrastructure. Other capabilities built into nGenera’s collaboration server include:

• Policy management – Use listening technology to detect non-compliant email (port 25) and web interaction (port 80), either on the Internet or internally, and set policy rules to allow, quarantine or block this traffic
• Social metadata aggregation – continuously build rich profiles from multiple sources (Active Directory, SharePoint, HRIS, social networks, esp. Facebook, user-provided), continuously track relationship strength between profiles, and continuously aggregate log-level activities from a variety of integrated services
• Federated search – Provide ranked search results across all elements of the collaborative environment (internal and external)
• Basic collaborative metrics – Aggregated statistics of tightly and loosely coupled services

These capabilities and others – like powerful analytics engines and more sophisticated reporting – are clearly meeting a demand that large enterprises are more aware than ever that they need. In addition to nGenera’s collaboration server, vendors like Radian6, Socialware, and a number of others are helping to build this sector of software that is bound to be very lively for some time.

But, when you consider the near-term cost and potential long-term damage to one’s brand – like what we’ve been seeing with the auto recall struggles of Toyota – it’s easy to understand why “learning to listen” in new ways using these tools has become an imperative for doing business.

When I originally started looking State of the Union addresses, I simply found transcripts online and did a manual count of words in text documents. This was laborious, but provided some interesting findings (note sites like Speech Wars can now automate this process). Last January I highlighted Wordle and used tag clouds to create a visualization of State of the Union addresses from notable past Presidents. This year, I’ve been spending a fair bit of time researching sentiment analysis, so I was pleasantly surprised to see that vendor Crimson Hexagon and CNN had teamed up to analyze public sentiment towards the 2010 State of the Union in real-time. Check out the video after the jump.

The impact of the new technology was not lost on the news media. The Huffington Post picked up the story and reported that, “The moment that ends up being most pivotal in changing the way the media covers big, live events may well have happened on CNN, where John King used the ‘Magic Wall’ to analyze almost 150,000 Twitter responses to President Obama’s speech.” In the article, CNN’s Senior Vice President and Washington Bureau Chief, David Bohrman is quoted as saying, “Twitter is all noise, but to be able to harness it and group it and actually intelligently cluster it and derive moods and opinions from it is very interesting.”

Whatever you might think of Twitter (Jon Stewart used the Magic Wall as an opportunity to make fun of both CNN and Twitter), this is exactly the type of technology companies are starting to think about for managing their brands, conduct market research, and pre-emptively deal with customer issues. The next level of granularity that sentiment analysis vendors are starting to offer is the ability to go beyond positive and negative sentiment to look at why sentiment is the way it is. Why are people pro-Obama? What types of issues are most often related to “Obama is too liberal?” This type of analysis is available, and I’ve seen demos from some vendors that offer fairly sophisticated drill-downs. However, some people remain sceptical about the general accuracy of this capability, as well as the limitations of most systems to crunch this type of data in real-time. Maybe we’ll see this for next year’s State of the Union—I’m hoping so.

Earlier this week I spoke at a conference hosted by the Royal Flemish Society of Engineers on the topic of complexity. The keynote speaker was Prof. Geoffrey West, former President of the Santa Fe Institute that pioneered the study of complexity science using a combination of economic theory and biology/physics (the founders were an economist and a physicist – both Nobel Laureates).  The end goal of complexity research is to develop new integrated conceptual frameworks for understanding the interdependence between various complex adaptive systems that define our world, including cities, financial systems, and the environment.

West’s research suggests we may be able to use the same rubric to study both cities, and forests, and maybe even economies. Complex adaptive systems share certain characteristics. Among other things, they: have many nodes, are interconnected, are adaptive and resilient, have many participants that create bottom-up disruptive change, result in emergent phenomena, and are often subject to unintended consequences. Sounds a lot like the type of emerging business ecosystems we talk about here on the Wikinomics blog. Collaboration between large groups of disperse and diverse individuals is extremely complex; when you add in financial systems, various incentives, supply chains, and a global information network, it becomes even more so.

West also talks about different types of networks—often layered on top of each other—as a characteristic of complex adaptive systems. The better we can understand networks and their interdependence, the better equipped we will be to understand complexity. He believes that underlying all complex systems are simple rules or patterns. For example, if you look at the metabolic rate, size, and lifespan of various organisms, you can determine that every biological organism grows in the same fundamental way. Here West asks some compelling questions: Are cities and companies just very large organisms satisfying the laws of biology? If so, why do all companies eventually die, while almost all cities survive? To this end, I think there’s probably great value in studying the evolution and “biology” of collaborative networks, informal networks within enterprises, business ecosystems, information flow and knowledge networks, and the multitude of other networks that collectively define Wikinomics-enabled business practices.

So, what are the best types of structures to deal with complexity? If we base our answer on how the Santa Fe Institute is structured, we find that the solution to complexity requires a multidisciplinary approach that involves participants that can bring different perspectives and diverse expertise. It also necessitates an open, distributed, and collaborative approach, a willingness to take risks, and a relatively small executive team that is able to meet face-to-face in order to build consensus and drive decision making at the highest level. This sounds remarkably similar to what we prescribe for next generation enterprises that want to thrive in today’s dynamic business ecosystems.

Another interesting thought at the conference came from Prof. Francis Heylighen who spoke of the Internet as a global brain that may act to combat complexity at a macro level by reinforcing strong signals between parties and building “synapses.” Tied to the he global brain theory is his theory of human stigmergy—a mechanism of spontaneous, indirect coordination between agents or actions (think of the way ants and other insects develop collective intelligence that enables coordinated and fairly complicated activities).

All of this is very much related to the research we’re conducting here at nGenera regarding what we call the highly-instrumented enterprise where actions are increasingly digitized, sensors and software track and analyze new sources of data, and create new understanding of complex systems and emergent phenomena. Some examples of these types of tools in a Wikinomics context might include reality mining tools that track the behaviours of individuals; automated sentiment analysis of text, voice, and even video; platforms that generate data and offer venues for consensus-building; and enterprise monitoring tools that map the informal networks and measure productivity within organizations. In fact, I’m sure there are many more connections to be made here, and look forward to thinking more about this one and hearing thoughts from our Wikinomics readers.

Collecting and analyzing customer stories – the impact you have on specific individual – is one of three approaches that I think makes sense when calculating overall ROI in a customer experience context. The other two are mission and reach. Mission is easy – most organizations have some sort of mission statement, or if they don’t, they usually at least have a CEO or a board with a well-articulated vision. Reach is also pretty straightforward: How many people to you touch? What reach does is calculate the influence of an organization or an individual. It acts as a proxy for several influence measures including: authority, frequency (how often you create the opportunity to influence consumers), independence (lack of bias in their opinion), charisma (in the case of an individual), and persuasiveness. At the OSC, the measure of reach includes both the number of people that came through the physical location and the over five million that engage with them online.

For me, impact is the most interesting measure. When assessing impact, Kevin uses the example of Canadian astronaut Chris Hadfield. Having someone like Hadfield say that going to the OSC as a child helped inspire him to be what he is today means the OSC had a strong impact on him. Most companies have at least a couple of these types of exemplary stories. But at a more pedestrian level, any organization can listen to and analyze the stories of people that come through the doors and that write online in blogs, forums, and social networks every day. There are now several companies such as Attensity, Scout Labs, Radian6, Visible Technologies, Crimson Hexagon (and many others), that have software to help analyze unstructured information like customer stories. These companies can identify basic metrics like the percentage of positive and negative sentiment, as well as provide deeper analytics about specific product and service features that lead to customers having positive and negative experiences. The end goal for those looking for quantifiable ROI numbers around customer experience is to convert all unstructured data to these types of “numeric” representations that are consistent, tracked over time, and can be charted in a dashboard. In short, the systematic gathering and analysis of anecdotes.

Gartner’s Hype Cycle for social media classifies Social CRM (i.e. the integration of social media into CRM systems) as a transformational technology that is two-to-five years away from mainstream adoption in customer service applications and five-to-ten years away from adoption in community marketing. While I agree that Social CRM will be transformational, I think the adoption will (and must) happen more quickly. Specifically, our research at nGenera has uncovered new data, new tools, new channels, and a new mindset that are accelerating the trend towards Social CRM.

Data: The data that is included in traditional CRM is limited to very basic identity and transactional information about customers. It does not typically include the type of rich digital profile information contained in places like Facebook and LinkedIn.  Customer feedback is collected through surveys, a method of data collection that is expensive, time-consuming, temporal, and often annoying for customers. But this is all changing. Customer data can be gathered from many sources, some old – such as the contact center – and some new. With respect to the contact center, the amount of unused customer data that is generated is astounding. One interviewee recently confided that his contact center writes the equivalent of a book every day – a book that nobody reads.  A first basic step is to generate organizational learning from contact centers. Once you’ve mastered this, you’re ready to move on to new sources of data. In this case, I’m thinking about reality mining, social networks, forums, blogs, and other digital venues where customers are engaging in behaviors that affect the company’s brand.

Tools: Listening platforms and sentiment analysis tools allow companies to capture customer preferences, complaints, feedback, and queries expressed online, while social network analysis can provide insight into the connections between individuals and identify key influencers. Companies can also track prosumer activity across branded communities and company-sponsored networks. When integrated with CRM databases, this information helps create accurate, up-to-date, and meaningful customer records. Although the CRM systems that currently offer applications to incorporate social media data only include data from a limited number of social networking sites – of which Twitter is the most common – this will likely change. Data will eventually be collected from all public online discussions as the concept of Social CRM becomes more accepted and companies develop strategies to deal with larger volumes of data. Once customer conversations have been successfully captured and incorporated into CRM databases, one can imagine a future where companies will be able to capture other forms of rich data, such as emotional data, photos, voice, and even video content (i.e. not just video metadata). According to a vendor I interviewed, companies can already correctly identify individuals online using available profile data with up to 90% accuracy.  This allows comapanies to find existing and potential customers online and gather new data about them. The contact center of the future will have a much richer digital picture of customers, allowing companies to personalizing product and service offerings, engage customers in meaningful conversations, and generate sophisticated trend data.

Channels: Many contact centers, such as those at Best Buy and Comcast now support social media channels and have dedicated teams devoted to responding to customers and prospects in public and branded digital venues. The question of whether or not to use social media as a listening platform or a contact center channel is major one for organizations as it affects the number of touchpoints that need to be managed and the complexity of customer support operations. However, as sentiment analysis tools get better, and integrate more readily with CRM, we expect this distinction to become less and less of a concern. In the future, the new sources of data (inputs) will be the same as the channels for customer interaction (output). As these channels mature, I fully expect the data and analytics to help “close the loop” with respect to customer engagement metrics – directly connecting social media investments with customer sales information. In this way, companies will be able to measure the value of customer intention and calculate the ROI of social media interactions.

Mindset: The notion of ‘relationship management’ brings with it a particular bias that data is controlled by the party that is doing the managing, rather than ownership of the data by the individual. So, in the case of CRM, it is assumed that the company is managing customer relationships by controlling the data about them and their interactions. New notions of relationship management seem to embrace the idea that ownership of both identity information and the customer-vendor relationship should reside with individuals, not companies.  Exchange of information should be based a two-way value proposition in which individuals selectively share aspects of their rich digital profiles, as well as their discretionary effort in exchange for useful and targeted messages, promotions, and reputation.  Paul Greenberg from ZDNet discusses this in more depth and notes, “Co-creation and mutually derived value, is at the core of Social CRM.” As an example, The Internet Identity Workshop and Project VRM (Vendor Relationship Management) at Harvard is exploring a highly customer-centric view of identity information where the customer controls their data and manages relationships with various vendors.

I’ve heard the argument that traditional CRM “is dead,” but this is far from the truth. In fact, as Brian notes, Social CRM does not replace transactional CRM systems, rather it augments them. What CRM is in desperate need of is new data sources and tools that help integrate and analyze this data. The future vision of CRM also requires that companies get involved in new channels and cede a certain amount of control to the customer – it’s less about management and more about engagement.

What’s equally impressive is some of the display technology being developed to display this type of data. All vendors have some form of executive dashboard, but these are highly utilitarian. From what I have seen, the bar for sentiment visualization is being set by other innovative thinkers. For example, ongoing projects like We Feel Fine from Jonathan Harris and Sep Kamvar as well as Bio Mapping from Christian Nold aim to visualize emotional data in new, interesting, and useful ways. We Feel Fine is more of an art project than a rigorous sentiment analysis tool, but it provides a useful example for how we might organize, display, and search for comments. Users on We Feel Fine can search by emotion (key word only), gender, age, location, weather, and date. It also connects emotions to associated images that are found within the document.

Bio Mapping (shown below) uses a lie detector connected to a GPS unit to measure location and physiological arousal at the same time. This is then plotted using Google Maps and other visualization software. Note, in this case the sentiment metric is intensity of emotion, not the specific emotion itself. The spikes shown on the map are locations of interest, but that is all we can determine from the data.

Although most of the complex visualization technology is still nascent you can imagine where this is type of analysis is going. Companies could segment customers based on emotional response, plot the spread of viral buzz, identify ideal test markets, and optimize local campaigns based on near-time feedback loops. Employees could gain access to a new lens on customer activity, behaviour, and satisfaction in a user-friendly display that makes analytics fun. Large retailers could use similar mapping technology powered by emotion data to optimize store layout and measure display/product appeal. The biggest challenge to wide adoption of these types of tools is the lack of valid emotional data in significant volumes. Currently, mining user comments online is the best available data source, but some early research suggests promising breakthroughs in the area of voice analysis and facial recognition as well – stay tuned.