Lack of Processes (or the Wrong Processes) Biggest Problem in Social Media Reporting and ROI

Posted on March 26th 2012

Lack of Processes (or the Wrong Processes) Biggest Problem in Social Media Reporting and ROI

I was thinking about a recent post by a friend, two recent conversations with close friends of mine who work in social media agencies , plus a recent review of my book and the same thing keeps coming up – everyone seems to want to measure Social Media ROI, but for the most part, comes up short.

Why is that?

I have the answer, finally (had it all along, just didn’t realize it, or call it for what it was – it’s the Pink Elephant in the room – the thing no one wanted to really address – cause they simply don’t know how, or it would take too much work to define, and get the necessary consensus).

The answer, of course, is “Process” – there’s isn’t any – neither in measurement or analytics – everyone seems to be making it up as they go (and one person has a different definition of the same deliverable as everyone else – there is just about no consensus, on anything).  That’s essentially a condition that reflects an immature discipline – Social Media Analytics, Social Media ROI, long hours and the office with don’t seem to go anywhere – are more often than not, a reflection of, I believe, of wrong choices, wrong platforms, wrong processes (or none), wrong people and ultimately, disgruntled stakeholders, employees and clients, who never seem to get what they wanted (like a simple “audit”, or a simple “ROI” report on their social media efforts and spend).

But, if you think about it, why would you get a report or audit when there was no common definition of what those things are, how long they take, what’s in them, and what they could and should cost?  How does one know if the result, furthermore, are good or bad, useful or not?

I hate to say it – because bad processes are probably worse than no process, but the real problem with the SXSW panel I linked to above, along with the all nighters my friends are regularly pulling, I believe, is there is no common lexicon to describe what makes up success in Social Media, and a common definition of how it should be measured (and I don’t mean “loose definitions” or something defined that is, what we used to call,  “wishy washy“).

What I mean is the deliverable for almost anything in Social Media are “undefined”.  They are, in other words, what you make them to be, and a lot of people simply have no idea what their goals are, what success is, how to measure it – and a groping, asking for opinion after opinion, becoming increasingly dissatisfied with that they hear.  No wonder.

At a South by Southwest Interactive Festival session, Liz Strauss, Eric Swayne, Petri Darby, Matt Ridings and Ford Motor’s Craig Daitch pledged to lay bare this difficult topic. While I would have loved to get some hands-on examples of how to work with this often-dodged topic, I got no such answers. The name of the panel, “What’s So [Bleeping] Hard About Social ROI?,” was a little misleading — yet there was a ring of truth to it — because the session seemed to be a roundup of all of the reasons its so “bleeping” hard to find ROI and not a substantive session on how to actually grapple with this thorny issue. Lots of comments on Twitter agreed.

Some terrific points were made — I will most certainly share — but more questions were asked than answers given. This left me a little disturbed, given how much preparation some of the presenters had. I think it could have been more hands-on with slides showing concrete examples. I wanted to get my proverbial hands dirty with a few calculations.

I’m afraid the problem isn’t going to be solved anytime soon Lori, unless we’re willing, collectively (some standards body at least take a few stabs at it) define thing such as “What is Social Media Success” and what are it’s signs and measures (so everyone can refer to these “common definitions”).  Same thing with something as simple as an “audit” – what’s in it?

Any one agree?   How much work goes into it and what does it include?

Try asking 10 people, and while you will get some answers that are common, you’ll get something different, I suspect with each person you ask.  No one seems to what anything means in Social Media, just what they want at the moment – that’s why Social Media ROI is so hard.

It’s got, believe it or not, not much to do with the data, even the unstructured nature of the data, sure that makes it more work, and a scalability issue, but for all that – it’s solvable.

But what if … a loaf of bread (a common meaning) is different for everyone?

That’s why, I’ll go on record and say, 70% of the time pulling analytics at agencies is “wasted time”, it’s work analysts are doing over and over again, because everyone has their own idea of what the deliverable is going to look like, and every single client has a different idea, or no idea, too.  Try wrapping that up into 40 hours of work -  no one can deliver what they don’t understand the parts of – where the parts are undefined.

And I’m not saying other disciplines don’t have some of the same problems, but at least, in analytics, most of the other types of analytics have much more definition on deliverable, and the data is often structured (though it may still need to be cleaned, anyway) while Social Data, isn’t by nature structured.

So, you have a lack of structure and a lack of process – and then people to execute non – existent procedures they are making up as they go, with formulas and metrics they are also trying to guesstimate, and then the make up some ROI numbers to keep the boss happy.

Stopping to think about it some more, the lack of defined process makes it next to impossible to meaningfully compare one deliverable from another, except subjectively, which opens up the Analytics to be a “subjective exercise” or add on, and no the essential thing that everyone seems to want to have (and hype) now.

The last agency I worked at also tried to create some processes – doesn’t sound like they got too far, every couple of months someone else would change their mind on what they wanted.

Just imagine if Math or Physics was like that – try sending a manned space mission, or blasting off the Space Shuttle with that kind of approach, and see what kind of results you’d get.  Same thing for the value of currency – there’s a common definition set up by a series of trading exchanges, at any moment you have a locked down definition of what a dollar is worth vs. what a British Pound is worth – but I don’t find Social Media, for all it’s talk, has evolved any solid processes and definitions that would make the work we do both scalable and accurate – it’s all subjective, wishy washy, and the like.

So much for the loss explanation of Social Media ROI – if the able people in the panel at South By could not come up with much – it just shows you they were, for the most part, as clueless as everyone else has been – because they had no defined processes that could be defined and scaled, I suspect.

I suppose that would not be so terrible, had not social analytics been sold as a metered service, suggesting, in fact, that defined processes and standards existed, that really don’t.

webmetricsguru

Marshall Sponder

See Full Profile >

Comments

It would be maddening to get too specific with standards as platforms are continuously being introduced and reimagined, but a basic set of processes is definitely needed. Who would have the final, official say?

Good points!  I've been chatting more and more about analytics and my first response to analytics & ROI quesitons is that Social Media is a communications tool, like a cell phone or an email program.  How does one measure these assets?  To measure ROI it has to sit on the asset side of the balance sheet. Just as with the difficulty of measuring how much business you gained by having an email address is practically an effort in futility so is measuring some sort of return on the time spent in social media engagement.

But I don't think that that's 100% accurate either.  Afterall, there are tons of analytics available but there is not one that tracks from source to sales order over time.  Most companies aren't tracking, and never have, the 'first appearance' of a new "lead"  (whether it be via a paricular trade show, magazine buy, or google search) and Social Media CRM is not yet evolved to a point where it can do it well either.

Even if they where evolved, how would you track the influence of "brand identity" exposures via social media: How do you measure that one key phrase that caught your attention when you glanced at Twitter one day and forgot about, and then mentally noted that the company was in a magazine ad at a later time, or their name in a trade show registry before one synapse connected to another and said "I remember that tweet they did! Maybe THAT product is the solution I need!"  

For now, I think many people are chasing ghosts and you're right, you need to have a plan on what to track from the start. But there again, another problem crops up: If you've never gone fishing, you have to first learn how to fish to develop your fishing strategy. ..and to even further the complexity, the social media platforms themselves are in a constant state of change so what you once thought was something valid to measure is now irrelevant. It's like learnign how to play three dimensional chess in the middle of a hurricane on the deck of a ship. 

Hi Marshal: Timely and highly relevant post, thank you. I think you and I have had a couple of brief chats online about the implications of this, so now is as good as a time as any to blow them out a little ;)

I think you allude to a few critical challenges with respect to process.

For one, I think the lack of process has more to do with the fact that most companies - especially agencies - can't get a handle on Big Data, and in particular, unstructured data. Secondly, in any instance or use case, I think this ladders up to an economic and business problem that can't be identified because the agency or a corporation will look for solutions that are creative or communications or brand-based, as opposed to understanding the value of the impact social media or social communications can have on the business. Third, and probably the most confounding, I don't think any of these organizations are approaching each situation contextually -- defining the real business or economic challenge, and then organizing respective internal groups to convene strategically around a solution-set, and from different angles. Lastly, I don't think most orgs are operationally capable of doing what I just mentioned -- they are literally hamstrung in their ability to problem-solve beyond a tactical outlook because, operationally, they can't get the various internal groups to talk constructively around what a solution-set could even look like.

So what about "ROI"?

This is a much larger conversation, but I think it boils down to breaking the legacy that vendors and agencies have created for the industry, which is essentially really bad "best practices" through digital measurement ("sales funnel" or "impressions/exposures" type crap), or simply no practices at all. The search and display spaces are glaring examples of this -- just look at how privacy is grossly mishandled. To boot, ROI is thought of as some iteration of a direct sale, as opposed to the possibility that attribution could be direct, latent or both. Even worse, analytics folks have largely been taught to think in only terms of numbers, not text or conversations, and semantics or NLP (for example) is an elusive discipline few companies really understand how to do. In sum, measurement has largely been thought of as a quantitative exercise, with very rigid thresholds for analysis.

So, I suppose ROI processes can emerge out of re-education or a redefinition of business goals, and those would emerge out of orgs learning how to operate better -- to understand that how they operate directly serves the functions for business or sales impact. It pretty much starts there, because no software or social technology can do that for you.

To quote Jeff Jonas of IBM's Entity Analytics Group: "When the data finds the data, the insight finds the user."

Orgs have a lot of work to do in preparing the data piece in this equation.

Best,

Gunther