#SMTLive Schedule

Where Social Media Measurement Falls Short

We are living in a world where statistics tell us everything. They explain our successes (and failings) and even what kind of a risk we are to mortgage lenders. Statistics determine if we get approved for credit cards, can rent a car, and they can even tell us which movie is going to be a success.

But “sanity is not statistical,” as our friend, Winston, remarked in George Orwell's book 1984.

What this means is that we cannot rely only on statistics to govern how we do things. We need to balance it with human intelligence (and spirit). Too often I hear horror stories of client requests to “measure everything!” and even more often nothing significant comes of it.

But if statistics can provide us with so much relevant information, where exactly does it fall short?

1. Show me the money.

In the book Super Crunchers, the author talks about a company that uses statistics to determine which movies will be huge hits - by only reading scripts. The accuracy is astounding and some movie studios use this method to determine not only which films to make but also to get suggestions on how to improve  to make the movie more profitable. Pretty cool, right?

But then I think about all the terribly unsuccessful movies that I love. I can't imagine if movies like 'Army of Darkness' or 'Fight Club' were never made. Just because a movie bombs in the box office doesn't mean that it's not good. On the flip side, just because a movie makes millions of dollars, it doesn't mean it is good (Twilight anyone?).

2. There is no social media 'To Do' list.

Unfortunately, measurement seems to be something that people like to have but then do nothing with. It's as though they think, "Social media measurement? Check. Okay, item number two...Pick up dry cleaning."

The purpose of measurement is to find out where we can improve, what we can do better and what is working. There is no point in measuring something if you aren't going to do anything about it.

3. It's creepy and unfair.

My husband recently received an e-coupon from AIR MILES for yogurt. Well, not just ANY yogurt. The exact brand of yogurt we buy on a weekly basis. They know this because when we buy it we swipe our AIR MILES card to get points.

Some of you may think that they are providing us with a customized experience. But I know that they did a very complicated algorithm to figure out the minimum discount they could give us based on our purchasing habits. That means that there is probably someone else out there who got a discount for the same yogurt but their discount was for 25% more (or less). How fair is that? If I went into a store and they charged different amounts for the same product, no one would stand for it.

It's also big brother-ish. I don't want AIR MILES (or anyone) to be that aware of my purchasing habits. It actually makes me not want to use my card when I purchase yogurt or to start buying a different brand. But they probably have enough data to predict that.

4. It encourages organizations to ignore the outliers.

I attended a Data Mining forum where a speaker was talking about how he developed a system for a bank. The system enabled tellers to immediately identify who was a high value customer (HVC) just by pulling up their profile. The purpose was to let tellers know instantly who they should be giving exceptional customer service.


It reminded me of when I was working as a dealer at the casino in Niagara Falls Ontario. We would see all sorts of customers, high value and low value. But, as dealers, we were trained to treat them all the same. Each one of them was to get exceptional customer service. While the casino had other ways of rewarding HVCs, increased customer service was not one of them. Every person that walked through our doors was to be treated like an HVC.

The bank used measurement to focus only on the very successful parts of their business, while ignoring what they should improve. Instead of using statistics to decide who should get the best customer service, perhaps they should be figuring out why a person isn't an HVC and how to convert them. I can be certain that providing exceptional customer service is just a start.

5. It's about people.

While it's obvious that companies need to start looking at how they do things based on statistics (not just intuition), it's easy to fall into a false sense of security. It's easy to rely on statistics to tell us the best way to do things or that things are just being done. But we need to remember that a good measurement strategy starts with good questions and questions come from intuition. Statistics can give us the answers we need, but we need people to act on those answers.

Join The Conversation

  • Aug 27 Posted 6 years ago ZoeSiskos

    Hi Michelle

    Sorry for the delayed reponse! I think with the new site switching over that my email notifications got all screwy.

    Anyway, thanks for sharing that post, I hadn't come across it. I agree, it's really about having good content (or products) that will make people buy/use/engage with your brand.

    thanks for the comment!



  • Jun 15 Posted 7 years ago MichelleC Nice post Zoe, and nice to find you via SocialMediaToday :)
    I wonder if you saw there was a video of Seth Godin recently posted talking about SCRM and the fact that marketers can't control peoples' behavior, no matter how badly they'd like to (http://bt.io/FQ9D). The stats can offer guidance but a company has to personalize its strategy.  Likewise, I think as consumers we have to be aware of the information that we choose to share, ie I am aware that every time I use a fidelity card I am getting discounts because I am giving up certain information.
    I definitely agree with your last point, though. I'm also interested in your findings and keeping in touch.
  • JoshCanHelp's picture
    Jun 5 Posted 7 years ago JoshCunningham Thanks for the link, Zoe... downloading and reading later.

    I would love to see what might come from a collaboration between language academics and social media data analysts. We might not get very far automating this analysis but we might learn some very interesting things about ourselves and how we express ourselves online.

  • Jun 4 Posted 7 years ago ZoeSiskos Thanks so much for the great feedback Josh! I'm looking forward to contributing more posts.

    As for the automated sentiment analysis - couldn't agree more. It's somewhat of a touch point where I work, in the Measurement Science department at Syncapse. We recently wrote a white paper on an experiment we conducted on the topic. If you are interested in reading it, here is the link - http://www.syncapse.com/media/syncapse_sentiment_analysis.pdf

    Our biggest finding was just as you said - even human sentiment scoring isn't that accurate or reliable. This is because there are many versions of the truth.





  • JoshCanHelp's picture
    Jun 4 Posted 7 years ago JoshCunningham Great post, Zoe, very well put (so good, in fact, we posted it on Facebook)! Everything you're saying is right on point, in my opinion. I'm a big number cruncher and I love data but, at a certain point, it starts to go the wrong direction and lead to poor decisions instead of informing the correct one.

    If I could add one more point... the one thing I would add to this list is "automated sentiment analysis." I've used several products that try to suss out what the general consensus is on a subject and they all fall short... very short. It starts to become creepy when it's correct and terribly misleading when it isn't. It's head hard enough to analyze what my finace meant over IM, how is an application have any chance?

    Thanks again!

Webinars On Demand

  • May 09, 2017
    With all of the technologies available to marketers today, have we lost that personal touch? Join VP of Content Marketing for ON24, Mark Bornste...
  • April 05, 2017
    In the ever-changing world of digital marketing, operational efficiency, quick turn-around times, testing and adapting to change are crucial to...