Measuring Service Effectiveness in the Call Center

MeikahDelid
Meikah Delid Meikah, Meikah

Posted on February 25th 2010

Measuring Service Effectiveness in the Call Center
guest post by: Frederick Van Bennekom

From call monitoring and mystery shopping to customer surveys, focus groups and complaint solicitation — a portfolio of approaches is needed to provide comprehensive and balanced measures of service effectiveness.
 

“How are we doing?” It's a natural question for an individual or an organization. In some cases the answer is easy because the objective criteria for success are simple, obvious, easy to collect and easy to analyze. For example, a bricklayer is measured on the number of bricks laid in a day and the quality of the workmanship.

Few organizations are simple, obvious, or easy to evaluate. This is especially so when the organization has multiple objectives — which may be in conflict. This is the life of service organizations. Service organizations need to be:

• Efficient in their use of re­sources

• Effective in the eyes of the customer. (Just to complicate matters, many service organizations have multiple customer constituencies, presenting more opportunities for conflicting effectiveness objectives.)

Why are these in conflict? Typically, to be more effective means having a buffer of resources to deal with unexpected demand. To make a rational trade-off decision requires good measurement of both efficiency and effectiveness. Yet, efficiency metrics typically win out. Why? Because efficiency is easy to define and measure with the operational systems we have in place to manage the service process. For example,

• Efficiency: Calls per day per service agent. Piece of cake to measure

• Effectiveness: Quality of the service delivered. Hmmm…

The Balanced Scorecard concept drives home this need for internal measures of efficiency balanced with external measures of effectiveness to keep the service organization aligned with corporate goals. But how do we measure effectiveness? How do we measure how good we are in the eyes of the customer?

Two broad approaches exist, internal and external measurement techniques. We'll address each with their respective approaches.

Internal Measurement Techniques

By “internal” measurement techniques, we mean that we don't involve the customer directly in the measurement process. Rather, employees or contractors put themselves in the shoes of the customer. Two internal measurement techniques can be applied.

Call Monitoring

This technique is used in most call centers. Someone listens in on the service interactions and judges the quality of the service delivered along a pre-established score sheet. The monitor might do more than listen in; they may also be “shadow monitoring” by watching the agents computer screens remotely.

Advantages: The advantages of the technique are several-fold. First, it is unobtrusive. We don't burden the customer, and the service agents do not know when they are being monitored. Second, evaluation can be done in real time. Thus, identified process issues can be addressed and remedied quickly before more customers become dissatisfied. Third, every service agent is measured against established criteria.

Shortcomings: The technique also has shortcomings. First, the monitors are applying their own qualitative assessments of the agents' words and deeds. Multiple monitors are needed to ensure fairness. Second, the monitors must be trained and calibrated to be sure they're all measuring the same way. Most importantly, the criteria on the score sheet must reflect what is truly of concern to the customer. Some score sheets have arbitrary requirements that may not be important to customers, such as greeting the caller by name at least three times. Also, the score sheets typically derive an overall score by applying weights to the different sections. How were these weights derived? Typically, they are a managerial judgment, perhaps driven by short-term hot buttons, and not derived statistically.

Mystery Shopping

st of us have heard of this technique since it's used heavily in consumer industries and search engine ads that barrage us to “get paid while you shop or eat.” However, the concept is applicable in most any service environment. (I conducted one of these studies for a technology vendor to see how their warranty processes performed in comparison to chief competitors. It was perhaps the most fun project I have ever done.) With mystery shopping, a contracted person pretends to be a customer, and tries to “exercise” the various paths possible in a service interaction. The “shopper” has a score sheet similar in concept to call monitoring score sheet.

Advantages: The primary advantage of this measurement approach lies in the ability to explicitly test service scenarios to see how the agents — and the system as a whole — perform. In other measurement techniques, you take what you get. Also, while a shopper does waste a service agent's time, no real customers are directly affected.

Shortcomings: The shortcomings start with the cost. Clearly, this process is labor-intensive. It would be prohibitively expensive to generate several “shopping” experiences with each service agent. Thus, this technique is not suitable for quality control monitoring. And just as with the previously mentioned call monitors, the shoppers need to be trained and calibrated, the score sheet needs rigorous development, and summaries must be legitimate.

External Measurement Techniques

In contrast to the internal techniques where we simulate being a customer, with external measurement processes we capture feedback directly from the consumers of the service.

Unsolicited Comments

Our first external measurement technique is one that just appears. Complaints or compliments in which the customer took the initiative to tell us about his or her feelings. Few customers will take the time to send feedback (except people like me). So it behooves an organization to make it easy to deliver comments. Toll-free lines and survey forms off the company's Internet home page are a couple ways to promote this free research.

Advantages: The key advantage is cost. The book, A Complaint is a Gift, by Janelle Barlow, makes the argument that a complaint is free market research. It's not free if we create systems to encourage comments, but it's still very inexpensive for the value received. If someone took the time to complain, it probably has a germ of truth to it. They're telling us where our system failed or supported them. And some people will expound in great detail (like me).

Shortcomings: The shortcomings are high. Clearly, these comments represent extreme positions. They are not typical and need to be interpreted as such. Many companies post survey forms on their Web sites and believe the results can be generalized to all customers. That is wrong.
The remainder of our external measurement techniques cover those in which we actively solicit feedback from our customers.

Personal Interviews

A structured interview is perhaps the simplest way to gather information from customers. These could be done through email exchanges, by phone or, preferably, in person. Richer contact media allow us to sense feelings beyond just the words. With phone, we get intonation. In person, we get body language.

Advantages: The key advantage of interviews is the richness of detail they provide. We learn not just the type of issue — positive or negative — but also the detail behind it. Thus, we can better develop corrective action plans. If we use a purposeful selection process to identify people for interviewing, then we are capturing feedback not just on the extremes, but covering the spectrum of customer feelings. However, the number of interviews will not likely lead to statistically significant findings. That's not the goal.

Shortcomings: Interviews have their challenging requirements. Note the term “structured interview.” For the data to have meaning we must collect it in at least a semistructured approach. This means develop a set of questions to guide the interview. Good interviewing skills are needed, as well as coordination among multiple interviews. Clearly, this is a labor-intensive process, especially for in-person interviews.

Focus Groups

Also, known as “small group interviews,” focus groups share most of the strengths and weaknesses of personal interviews. Rather than interviewing each person individually, we interview them as a group over the course of an hour or two.

Advantages: The key advantage focus groups provide is the interaction among the participants, making for even richer detailed feedback than with individual interviews. We really get the “why behind the what” of customer issues. We also have some efficiency, since we're gathering feedback from a dozen or so people at once. .

Shortcomings: The key challenges are logistical. Planning and execution are vital. The customers must be clustered geographically, unless you use an online discussion group to serve as a Web-based focus group. The moderator's ability to facilitate a good discussion is absolutely critical.

User Group Feedback

User group meetings provide an opportunity for a wealth of information collection. Gathering feedback at user group meetings through interviews or focus groups is just plain common sense. We have a group of people who have a strong commitment to our products. And they are likely to be very willing to tell you what they feel and what they'd like to see.

Advantage and Shortcoming: The strength is also the weakness of the approach. They are current customers and will drive product ideas toward improving their own use of the product, rather than ideas that will appeal to an expanded market. These customers may also now feel ownership for product direction and will be disappointed if they don't see what they proposed. Positioning is critical with user groups.

Mass-Administered Surveys

I suspect that most of you reading this article expected it to focus on customer surveys. Surveys are great at telling us how our customer base as a whole feels — if done properly. We gather structured data (hopefully) from a sample of our “population” and can then generalize the results from the sample to the population.

Advantages: The key benefit of the survey technique is that it provides an efficient means of generating a broad overview of how our customers feel, identifying strong and weak points in the service system. Incident surveys (also known as event or transactional surveys) can flag customers in need of a service recovery.

Shortcomings: Surveys are not good at generating the detail behind the feelings. (Hold that question…)

To generate meaningful data, surveys have strong requirements. The survey questionnaire must avoid introducing bias, the questions must reflect the true issues of concern to customers, and the resulting data must support the desired analysis. The survey administration has to avoid biasing results and we must get enough responses for statistically significant findings. Finally, the survey data analysis has to be executed correctly to give voice to the message buried in the numbers.

We've covered a lot of ground here. You may be asking, “So which measurement process is best?” The answer is: All of them and none of them. Every research technique has strengths and every research technique has weaknesses.

Re-read the above descriptions and you'll see that some techniques (e.g., focus groups and interviews) are good at generating detail-rich findings while others (e.g., surveys) are good at getting summary findings that represent our customer base as a whole. What's needed is a portfolio approach. Just as your financial portfolio should have a balance of investment instruments, so, too, should your service measurement approach.

The chart above makes this point. Proper interpretation and use of the findings from a research technique means understanding its place in the research spectrum. Rigorous application of a portfolio of complementary service effectiveness measurement techniques helps to ensure that we truly understand how our customers view us and how to address those issues. In short, our Balanced Scorecard becomes balanced properly.

 Frederick Van Bennekom is Founder of Great Brook Consulting and author of Customer Surveying, A Guidebook for Service Managers. www.greatbrook.com.

Link to original post
MeikahDelid

Meikah Delid

Meikah, Meikah

Ma. Merdekah “Meikah” Ybañez-Delid majored in BA English at the University of the Philippines where she was president of The Language Society, an academic organization of the College of Arts and Letters. In 1996, after teaching in the tertiary level for four years, she took her Master of Arts in Journalism at UP and finished the coursework first semester of 1998. She was working on her thesis on “Peace Communication” when she was invited to join a Korean-owned website company. She joined the company as English Proficiency Specialist and went on to become Senior Content Developer. Sometime 2002-2003, she became part of a pool of writers for a textbook series for a leading publisher in the country. Since then, she had been writing and doing research both for the education-community and IT fields, particularly for websites of foreign organizations. She was working as admissions head at a foreign language training center, when she decided to do full-time blogging and writing electronic study guides. She blogs here at CustServ and also at Six Sigma. As of late, she has started actively maintaining two personal blogs: MamaIsWorking and On The Streets. Meikah’s latest preoccupation, which is taking most of her time, is project management. She is project managing GoingToMeet.com, a comprehensive online conference directory, and LocalPhilippines, a traveler’s ultimate guide to the Philippines. The latter project though is still in its dev stage.
See Full Profile >

Comments

Great info. More call center articles please :)