“How are we doing?” It's a natural question for an individual or an organization. In some cases the answer is easy because the objective criteria for success are simple, obvious, easy to collect and easy to analyze. For example, a bricklayer is measured on the number of bricks laid in a day and the quality of the workmanship.
Few organizations are simple, obvious, or easy to evaluate. This is especially so when the organization has multiple objectives â€” which may be in conflict. This is the life of service organizations. Service organizations need to be:
• Efficient in their use of reÂsources
• Effective in the eyes of the customer. (Just to complicate matters, many service organizations have multiple customer constituencies, presenting more opportunities for conflicting effectiveness objectives.)
Why are these in conflict? Typically, to be more effective means having a buffer of resources to deal with unexpected demand. To make a rational trade-off decision requires good measurement of both efficiency and effectiveness. Yet, efficiency metrics typically win out. Why? Because efficiency is easy to define and measure with the operational systems we have in place to manage the service process. For example,
• Efficiency: Calls per day per service agent. Piece of cake to measure
• Effectiveness: Quality of the service delivered. Hmmm…
The Balanced Scorecard concept drives home this need for internal measures of efficiency balanced with external measures of effectiveness to keep the service organization aligned with corporate goals. But how do we measure effectiveness? How do we measure how good we are in the eyes of the customer?
Two broad approaches exist, internal and external measurement techniques. We'll address each with their respective approaches.
“internal” measurement techniques, we mean that we don't involve the
customer directly in the measurement process. Rather, employees or
contractors put themselves in the shoes of the customer. Two internal
measurement techniques can be applied.
This technique is used in most call centers. Someone listens in on the service interactions and judges the quality of the service delivered along a pre-established score sheet. The monitor might do more than listen in; they may also be “shadow monitoring” by watching the agents computer screens remotely.
Advantages: The advantages of the technique are several-fold. First, it is unobtrusive. We don't burden the customer, and the service agents do not know when they are being monitored. Second, evaluation can be done in real time. Thus, identified process issues can be addressed and remedied quickly before more customers become dissatisfied. Third, every service agent is measured against established criteria.
technique also has shortcomings. First, the monitors are applying their
own qualitative assessments of the agents' words and deeds. Multiple
monitors are needed to ensure fairness. Second, the monitors must be
trained and calibrated to be sure they're all measuring the same way.
Most importantly, the criteria on the score sheet must reflect what is
truly of concern to the customer. Some score sheets have arbitrary
requirements that may not be important to customers, such as greeting
the caller by name at least three times. Also, the score sheets
typically derive an overall score by applying weights to the different
sections. How were these weights derived? Typically, they are a
managerial judgment, perhaps driven by short-term hot buttons, and not
st of us have heard of this technique since it's used heavily in consumer industries and search engine ads that barrage us to “get paid while you shop or eat.” However, the concept is applicable in most any service environment. (I conducted one of these studies for a technology vendor to see how their warranty processes performed in comparison to chief competitors. It was perhaps the most fun project I have ever done.) With mystery shopping, a contracted person pretends to be a customer, and tries to “exercise” the various paths possible in a service interaction. The “shopper” has a score sheet similar in concept to call monitoring score sheet.
Advantages: The primary advantage of this measurement approach lies in the ability to explicitly test service scenarios to see how the agents â€” and the system as a whole â€” perform. In other measurement techniques, you take what you get. Also, while a shopper does waste a service agent's time, no real customers are directly affected.
Shortcomings: The shortcomings start with the cost. Clearly, this process is labor-intensive. It would be prohibitively expensive to generate several “shopping” experiences with each service agent. Thus, this technique is not suitable for quality control monitoring. And just as with the previously mentioned call monitors, the shoppers need to be trained and calibrated, the score sheet needs rigorous development, and summaries must be legitimate.
contrast to the internal techniques where we simulate being a customer,
with external measurement processes we capture feedback directly from
the consumers of the service.
Our first external measurement technique is one that just appears. Complaints or compliments in which the customer took the initiative to tell us about his or her feelings. Few customers will take the time to send feedback (except people like me). So it behooves an organization to make it easy to deliver comments. Toll-free lines and survey forms off the company's Internet home page are a couple ways to promote this free research.
Advantages: The key advantage is cost. The book, A Complaint is a Gift, by Janelle Barlow, makes the argument that a complaint is free market research. It's not free if we create systems to encourage comments, but it's still very inexpensive for the value received. If someone took the time to complain, it probably has a germ of truth to it. They're telling us where our system failed or supported them. And some people will expound in great detail (like me).
shortcomings are high. Clearly, these comments represent extreme
positions. They are not typical and need to be interpreted as such.
Many companies post survey forms on their Web sites and believe the
results can be generalized to all customers. That is wrong.
The remainder of our external measurement techniques cover those in which we actively solicit feedback from our customers.
A structured interview is perhaps the simplest way to gather information from customers. These could be done through email exchanges, by phone or, preferably, in person. Richer contact media allow us to sense feelings beyond just the words. With phone, we get intonation. In person, we get body language.
Advantages: The key advantage of interviews is the richness of detail they provide. We learn not just the type of issue â€” positive or negative â€” but also the detail behind it. Thus, we can better develop corrective action plans. If we use a purposeful selection process to identify people for interviewing, then we are capturing feedback not just on the extremes, but covering the spectrum of customer feelings. However, the number of interviews will not likely lead to statistically significant findings. That's not the goal.
have their challenging requirements. Note the term “structured
interview.” For the data to have meaning we must collect it in at least
a semistructured approach. This means develop a set of questions to
guide the interview. Good interviewing skills are needed, as well as
coordination among multiple interviews. Clearly, this is a
labor-intensive process, especially for in-person interviews.
Also, known as “small group interviews,” focus groups share most of the strengths and weaknesses of personal interviews. Rather than interviewing each person individually, we interview them as a group over the course of an hour or two.
Advantages: The key advantage focus groups provide is the interaction among the participants, making for even richer detailed feedback than with individual interviews. We really get the “why behind the what” of customer issues. We also have some efficiency, since we're gathering feedback from a dozen or so people at once. .
key challenges are logistical. Planning and execution are vital. The
customers must be clustered geographically, unless you use an online
discussion group to serve as a Web-based focus group. The moderator's
ability to facilitate a good discussion is absolutely critical.
User Group Feedback
User group meetings provide an opportunity for a wealth of information collection. Gathering feedback at user group meetings through interviews or focus groups is just plain common sense. We have a group of people who have a strong commitment to our products. And they are likely to be very willing to tell you what they feel and what they'd like to see.
Advantage and Shortcoming: The
strength is also the weakness of the approach. They are current
customers and will drive product ideas toward improving their own use
of the product, rather than ideas that will appeal to an expanded
market. These customers may also now feel ownership for product
direction and will be disappointed if they don't see what they
proposed. Positioning is critical with user groups.
I suspect that most of you reading this article expected it to focus on customer surveys. Surveys are great at telling us how our customer base as a whole feels â€” if done properly. We gather structured data (hopefully) from a sample of our “population” and can then generalize the results from the sample to the population.
Advantages: The key benefit of the survey technique is that it provides an efficient means of generating a broad overview of how our customers feel, identifying strong and weak points in the service system. Incident surveys (also known as event or transactional surveys) can flag customers in need of a service recovery.
Shortcomings: Surveys are not good at generating the detail behind the feelings. (Hold that question…)
To generate meaningful data, surveys have strong requirements. The survey questionnaire must avoid introducing bias, the questions must reflect the true issues of concern to customers, and the resulting data must support the desired analysis. The survey administration has to avoid biasing results and we must get enough responses for statistically significant findings. Finally, the survey data analysis has to be executed correctly to give voice to the message buried in the numbers.
We've covered a lot of ground here. You may be asking, “So which measurement process is best?” The answer is: All of them and none of them. Every research technique has strengths and every research technique has weaknesses.
Re-read the above descriptions and you'll see that some techniques (e.g., focus groups and interviews) are good at generating detail-rich findings while others (e.g., surveys) are good at getting summary findings that represent our customer base as a whole. What's needed is a portfolio approach. Just as your financial portfolio should have a balance of investment instruments, so, too, should your service measurement approach.
The chart above makes this point. Proper interpretation and use of the findings from a research technique means understanding its place in the research spectrum. Rigorous application of a portfolio of complementary service effectiveness measurement techniques helps to ensure that we truly understand how our customers view us and how to address those issues. In short, our Balanced Scorecard becomes balanced properly.
Frederick Van Bennekom is Founder of Great Brook Consulting and author of Customer Surveying, A Guidebook for Service Managers. www.greatbrook.com.