Share this content

Jim Sterne: Social media measurement and "the accuracy problem"

29th Apr 2010
Share this content

Social media is so important that the results of your social media efforts should be continuously assessed. However, many organisations are struggling with measurement. But social media isn't hard to measure, says Jim Sterne - providing we're realistic about the results...

"It is a lot of hype. It is very exciting. Everyone says you have to do it or you’ll die a horrible death. And nobody knows what they’re doing!"
Welcome to the world of social media, as summarised by speaker, author, consultant and co-founder of the Web Analytics Association, Jim Sterne. The master of all things within the world of metrics and measurement, Sterne has been spending an increasing amount of his time pondering social media. Why? Because while we all acknowledge its significance, we don’t yet know how to measure the business results of our business investments in this new medium.
"Social media is so important that the results of your social media efforts should be continuously assessed, because frankly you’re going to be making investments in it, so you really do need to measure it!" says Sterne. But the reason he’s been devoting so much time to this area – and has authored a recently launched book, Social Media Metrics: How to Optimize Your Marketing Investment, on this very topic – is that this area is vexing a great many businesses.
Furthermore, a great many measurements being used to assess performance, are in fact fairly useless. "Measuring how many followers I have on Twitter is the same as measuring hits on my website. It is interesting only because it is a number, it is actually not useful," Sterne emphasises. "When you take it in combination with other factors, well, then you can work it up into some kind of performance indicator. But pure number of followers is actually meaningless."
At best then, some social media measurements can be misleading. At worst, however, they could lead to the termination of social media initiatives. The shift from ‘broadcast’ to ‘engagement’ demands that organisations hone their listening skills rather than their publishing skills. And explaining this to upper management in order to secure funding for the tools and labour to make sense of what is being said out in the social world is particularly difficult when there are fuzzy stats to back up the business case. Especially when management expects such efforts to be infinitely measurable.
"The internet is supposedly the most measurable approach, and so my expectation is that I can measure everything and it must be perfect measurement. And social media is online, therefore it too must be perfect measurement! So it is sad because those of us who do measure and do provide real insight and real business value to the organisation have to fight this accuracy problem. It is not accurate, but it is useful."
In fact, by using time honoured metrics for advertising, like reach, frequency and awareness, as well as more challenging metrics such as influence and sentiment, businesses can achieve social media measurability to a degree that should be more than satisfactory - certainly to a level that has historically been satisfactory for the likes of TV, print and radio.
Here, Sterne provides a whistle stop tour through the measurement process.
Awareness, uptake and repetition
"We start with awareness, which is related to reach and frequency – good, old-fashioned advertising metrics," says Sterne. "I put up an outdoor sign in a high street and there are x thousand people who walk by who have an opportunity to see my message. If I post something out on the blogosphere or I tweet something there is a certain opportunity to see by the number of RSS readers, by the number of followers, and if it gets picked up and retweeted then my opportunity to see and my reach grows. That is a standard classic metric. And the way I validate whether it is working, whether the reach and frequency are having any impact on awareness, is I have to go out and interview people to ask them if they are aware."
This of course is another time-honoured classic advertising metric involving random digit dialling, specifically asking the respondent questions such as whether they have seen the ad on television, whether they have visited your website, and did they know that you have a new product feature. All of this is standard stuff being applied to a new communication channel being used in a straightforward broadcast manner.
"What gets tricky, however, is when it comes to measuring the ‘virality’ of some of the messages. If I do what we used to lovingly call ‘publicity stunts’ but what we now call viral marketing, is it being retweeted, is it being picked up on the blogs? And so now we go to the classic public relations metrics of uptake and repetition – whether the comment was repeated in one way or another."
So at this point we have reach (opportunity to see); awareness (did they get the message at all); and a virality or value metric for the message itself (was it worth repeating, did people remark about it). And then we get to the nebulous area of influence and the murky realm of sentiment.
Influence and sentiment
In the event that you are fortunate enough to have your message retweeted, for instance, influence needs to be factored into the equation. Many of your followers may have only very small amounts of followers themselves – but some may have a huge following and be enormously influential figures. "If these 2,800 people retweet it, it doesn’t really mean much. But if this other 12 people repeat it, it is huge because they have a big following," Sterne explains. "I need to know who the influencers are - and not just by the number of followers.
"Therefore, I need to take a measure of the people who retweeted, looking at how many people are following them and how many retweeted them. Now we really understand that they influenced others behaviours. And you can follow that down a logical chain until it gets a bit murky and dissipates."
But the murkiest area of all is sentiment. Historically, of course, this was all about market research, going out into the world to ask people about what brands they buy, whether they’ve heard of your products, how they feel about them, whether they plan to buy them and whether they would recommend them to friends. Traditionally the responses were skewed as you only had data from people who were happy to respond to the research. The opinions of the vast majority of us, who would sprint at the very sight of a man with a clipboard, were unaccounted for. Meanwhile, the natural human inclination is to say what people want to hear, because they want to be friendly – rather than saying the product tastes like dog food.
With social media, however, we can now tune into all kinds of sources for opinions on our products and services – blogs, review sites, twitter, etc etc. Once again, however, it is important to keep our expectations in check, according to Sterne. "I can really get an idea for what people are feeling - except that it does not scale," he explains. "So if five people a day comment about me or my products or my company, I can read all of them in the morning over tea. But if it is 5,000 or 50,000 I now have to resort to tools."
Sterne suggests that while real-time visual analysis, speech recognition and textual analysis have come a long way, and content can now be successfully summarised, when it comes to concluding whether the overall tone is positive or negative, there is still a challenge. And it is one that we’re unlikely to address in the short-term.
"What we are able to do is come up with some data, some numbers, that are accurate within some range that give us the value about on equivalent of standard market research and going out and interviewing people. So it has value. Is it perfect? No. Is it accurate? No. Is it useful? Well, yes. Because if the machine says that comment about this new feature of your product are primarily positive, and suddenly there is a spike in negative comments, whether it is 100% accurate or not, that should trigger an alarm and it would be worthwhile taking a look. And now the humans can get involved. So people who say, we can tell what people are thinking… no, sorry. But you will get alerts that say people are talking about the fact that your automobile accelerates when customers don’t want it to. And you should probably take a look at that!"
Business outcomes
After awareness, uptake, repetition, influence and sentiment, the final metric at the end of the chain is business outcomes. So while the conversation may be taking place in the social media world, are they then actually visiting the website – and if they are, what are they doing when they’re there? What does their behaviour reveal about them? And finally, did they fulfil the business outcome – whether that be buy the product, download the whitepaper, sign up for the webinar, join the membership, etc.
"There is a landscape of what we can measure, and when somebody says social media is too difficult to measure, my response is ‘what is it you’re trying to find out?’" says Sterne. "It is like when people look at all the cool web analytics stuff and ask what they should be measuring. Well, that depends on what you’re trying to accomplish. What are your business goals?"
So, measurability isn’t such a challenge for social media after all, according to Sterne. We just need to be clear about the outcomes that we’re trying to achieve – and have realistic expectations about the insight that we can achieve.
"There is unreasonable expectation in measurement. Social media is very measurable. But people expect perfection and it is absurd," concludes Sterne.
"Measuring television advertising is not accurate. Measuring the value of any kind of promotional effort is not accurate. But it is useful. Therefore we’re dealing not with actual, but with probabilities, which are hugely valuable in every other form of measurement we have. Statistically speaking you should not smoke cigarettes. How many people die from cigarettes? Well, it is kind of fuzzy. But is it useful information? Oh yeah…"

Replies (0)

Please login or register to join the discussion.

There are currently no replies, be the first to post a reply.