New Study, Same Results: Minority of Companies Do Effective Marketing Performance Measurement

The latest addition to my collection of surveys about marketing measurement is The Marketing Performance Advantage, a joint effort from strategic marketing consultants CMG Partners and market researcher Chadwick Martin Bailey. Based on 400 online interviews among CFOs, CEOs and marketing employees of companies with 100+ employees, this is one of the larger and more sophisticated studies on the topic.

One-Quarter of Companies Measure Marketing Performance Effectively

The main finding is that about one-quarter of marketers feel they do an adequate job of measurement. This matches other studies on the topic. The survey asked several questions along those lines:

- 20% say their company "excels" at measurement
- 22% "excel" at using measurement-based insights to drive improvement
- 24% see a positive impact from measurement, and
- 27% have fully integrated measurement into marketing planning

The study also resembled other research in showing that many more marketers list measurement as a top priority (44%) than actually do it.

Marketing VP's Are More Satisfied with Measurement Than Anyone Else

One intriguing detail was that senior marketers seem eerily "overconfident" (the authors' word) compared with those above and below them in the organization.

- 13% of marketing vice presidents consider marketing performance measurement a "huge challenge", compared with 34% to 38% of CEOs, marketing directors and marketing managers, and 61% of CFOs.

- 38% of marketing vice presidents felt that measurement has a "huge impact" on their business, compared with 15% to 29% of CEOs, marketing directors and marketing managers, and 7% of CFOs.

How well is your organization performing with respect to measuring the performance of marketing initiatives? How well are you using insights to improve the performance of marketing initiatives?

This is a huge challenge...	CEO	CFO	VP Marketing	Director Marketing	Manager Marketing
Measuring MP	36%	61%	13%	38%	34%
Improving MP	28%	61%	9%	38%	38%

To what extent, if at all, has measuring the performance of your marketing initiatives improved your business?

Impact of MP on your business	CEO	CFO	VP Marketing	Director Marketing	Manager Marketing
No impact	29%	52%	0%	21%	21%
Neutral	42%	41%	62%	64%	50%
Huge Impact	29%	7%	38%	15%	29%

Although the authors don't make the connection, these results help to explain why more money isn't invested in marketing measurement: the marketing vice presidents who control the purse strings are the least convinced they have a major problem.

Barriers to Measurement: Data, Technology and Process

The survey also asked about major barriers to marketing measurement. These include all the usual suspects. If anything, what was intriguing was that executive support is such a small issue compared with the others:

Barriers to improvement (% answers 1-4 on scale of 1-10):

- 40% collecting the right data
- 40% technology/systems
- 39% clear & effective processes
- 36% use of customer analytics
- 36% organizational alignment
- 26% skills sets
- 20% senior level buy-in

Effective Companies Have Clear Process to Apply Measurements, Invest in Measurement Capabilities and Hold Marketing Accountable for Results

Another set of questions covered adoption of best practices, and compared answers from companies reporting positive impact from marketing measurement with answers from the others. The biggest differences were in having clear processes to ensure that measurement-based insights are applied to decisions; the next tier included marketing targeted investments and holding marketing accountable for measured results. Senior level buy-in, strategic alignment and usage outside of marketing were less prominent.

Best practice adoption (index of use by companies reporting positive impact, where 100=average of all companies)

- 251 clear process to ensure measurements are applied to decisions
- 215 targeted investments in measurement technology/systems, skills and data
- 206 marketing held accountable on performance metrics
- 161 alignment of marketing activities to strategic business objectives
- 159 senior level buy-in
- 145 usage beyond marketing

eMarketer Report Details Next Steps for Online Brand Measurement

eMarketer recently released a deeply researched report on Online Brand Measurement. Since it touched on several topics I’ve been pondering recently (see Web Analytics Is Dead… on my Customer Experience Matrix blog) , I read it with particular care.

This is a long report (58 pages), so I won’t review it in detail. But here are what struck me as the critical points:

- Web measurement has largely focused on counting views and clicks, not measuring long-term brand impact. Counting is much easier but it doesn’t capture the full value of any Web advertisement. One result has been that marketers overspend on search ads, which are great at generating immediate response, and underspend on Web display ads which influence long term behavior even if they don’t generate as many click-throughs.

- Media buyers want Web publishers to provide the equivalent of Gross Rating Points (GRPs), so they can effectively compare Web ad buys with purchases in other media. That’s okay as far as it goes, but it’s still just about counting, not about measuring the quality or impact of the impressions. As the paper points out, even engagement measures such as time on site or mentions in social media, don’t necessarily equate to positive brand impact.

- Just about everyone agrees that the right way to measure brand impact is to tailor measurements to the goal of a particular marketing program. This may sound like a conflict with the desire for a standard GRP-like measure, but it really reflects the distinction between counting the audience and measuring impact. GRPs work fine for buying media but not for assessing results. Traditional media face precisely the same dichotomy, which is why marketing measurement is still a puzzle for them as well. And just as most offline brand measures are ultimately based on surveys and panels, I'd expect most online brand measures will be too.

- Meaningful impact measurement will integrate several data types, including online behaviors, visitor demographics, offline marketing activities and actual purchase behavior. These will come from a combination of direct online sources (i.e., traditional Web analytics), panel-based research and surveys (for audience and attitudinal information), and offline databases (for demographics and purchases). Ideally these would be meshed within marketing mix models and response attribution models that would estimate the incremental impact of each marketing program and allow optimization. But such sophisticated models won’t appear tomorrow.

To me, this final point is the most important because it points to a “grand unification theory” of marketing measurement that combines the existing distinct disciplines and sources. The paper cites numerous current efforts, including:

- multimedia databases being created (separately) by panel-based measurement firms including comScore, Nielsen, Quantcast and TNS Media Compete;

- Datatran Media’s Aperture, which combines email and postal addresses with Acxiom household data, IXI financial data, MindSet Marketing healthcare data and NextAction retail data;

- a joint effort between Omniture and WPP’s Kantar Group that combines data from email, search, display ads and traditional media;

- another Nielsen project combining TV ad effectiveness information from Nielsen IAG with panel purchase data from Nielsen Homescan.

These all reinforce the claim I made in last week’s blog post that individual data will increasingly be combined with panel- and survey-based information to provide community-level insights that are actually more valuable than individual data alone.

ANA Agency/Client Forum: Agency Performance Isn't Based on Results

I spent most of yesterday at the Association of National Advertisers (ANA)’s Agency/Client Forum. The agenda covered a range of timely topics including digital advertising and social media. But I think it’s fair to say that most of energy was focused on the pocketbook issue of agency compensation.

In particular, the question du jour was value-based compensation or its cousin, pay-for-performance. I would have thought those were pretty much the same thing, but as Coca Cola’s Director of Worldwide Media & Communication Operations Sarah Armstrong set out in a detailed description of Coke’s own process, value-based compensation works largely by estimating a reasonable cost in advance, while pay-for-performance is based on after-the-fact assessments. (Coke’s process incorporates both – a base fee that is intended cover agency costs, plus up to 30% bonus based on performance.)

However, as several speakers made clear during the day, most pay-for-performance measures are based on agency behaviors such as innovation, strategic thought and execution, rather than business results such as sales, market share or even communications activities such as media cost savings. Specifically, an ANA survey that will be formally released in mid-October found that 56% of agency performance measures were based on qualitative metrics, vs just 19% on business results and 25% on communications metrics.

My initial reaction to this survey was pretty dismissive – either you pay for results or you don’t. But the fundamental rationale, mentioned by several speakers through the day, is that business results are affected by many factors beyond the agency’s control, so it really wouldn’t be fair to penalized or reward them purely on that basis. The analyst in me says it’s still worth trying to isolate the agency's contribution to results, but this is definitely a valid point. So including the subjective measures does make more sense than I initially thought.

A related question that ran through several presentations was whether agencies are commodities. One presenter flashed a survey that showed 80% of respondents thought they are (I didn’t capture the details of that survey, but think it was an informal online poll, so it’s probably not very meaningful). But both the clients and agencies among conference speakers felt strongly that they are not.

What was interesting, though, was the sorts of distinguishing features that speakers cited – strategic insights, brand stewardship, creative genius, etc. Those are based largely on the skills and chemistry of the individuals working on an account. As the cliche says, those assets “go down the elevators every night” – that is, they are individuals rather than property of the agency itself. So it’s possible that the agencies themselves are pretty much commodities (i.e., have about the same processes and technology) even if their people are different.

And even when it comes to people, I find it hard to believe that any one agency can really have people who are on average much better than any other agency. There are, in fact, plenty of smart and creative people in the world. Yes, there are occasional true geniuses, and clients lucky enough to find them working on their accounts may indeed gain a strategic advantage. Perhaps some of those geniuses are even so clever that they can build an entire culture around themselves to leverage their skills. But I'd say that level of genius is very much the exception.

In general, then, I suspect that once a quality agency comes up to speed, it would produce roughly similar results to another quality agency. This doesn't mean that you could immediately switch from one to another. But over the long term, agencies probably are something close to a commodity.

This relates back to the performance measurement questions. The value of an agency really does lie in its strategic, creative, and execution contributions, plus its ability to work closely with the client. In theory, most agencies should be able to do these equally well. But in fact, there will be variations based on the individual team members as well as (to a lesser degree, I think) differences in agency processes and culture. So it makes sense for performance evaluations to focus on those factors, even though they’re subjective. Marketers must measure those factors to identify areas needing improvement, either by changing performance of their current agency partners or switching to new ones.

Mzinga Survey Shows Most Companies Don't Measure Social Media ROI

Toute le blogosphere is in love with social media, which of course means that some contrarians have to argue that it’s over-hyped. So it was interesting to see a survey (available here; registration required) show that social technologies are indeed widely adopted: 86% of 555 respondents said they are currently using them for business purposes, and 61% said it was an ongoing component of their business.

Caveat: the survey was sponsored by social technology vendor Mzinga in conjunction with the Babson Executive Education program, so they had a stake in the outcome. But I didn't see any obvious problems with it, and even allowing for some bias, the results still suggest wide social technology usage among a broad spectrum of businesses.

Probably the most interesting result from a marketing measurement perspective was that just 16% of respondents reported measuring ROI on their social media programs. No surprise, alas, but worrisome because programs that can’t prove ROI are subject to cancellation when money is tight.

Somewhat supporting this line of reasoning, the survey showed that just 40% of respondents had budget dedicated to social media and 57% had employees assigned to it. Perhaps many of those employees work for free, but a more likely explanation is that their costs are not part of project budgets because they're part of a vaguely fixed "overhead". This makes it easier to sustain a social media effort without formal economic justification. But it can’t be a permanent situation – managers will eventually realize that time spent on social media has a real cost. So justification of some sort will ultimately be needed.

Of course, that justification won’t necessarily be ROI. We all know that many traditional marketing investments are not justified on the basis of ROI, and marketing is by far the most common social media application (57%, vs 39% for internal collaboration, 31% other, 29% customer service & support, 25% sales, 21% human resources, 16% strategy and 14% product development). Marketing in social media could easily go unmeasured as well.

Indeed, just 8% of respondents said their social technology system could showcase ROI, vs. 41% who said it couldn’t. An impressively large 44% didn’t know, which I interpret to mean that they didn't care enough to find out. So I think it’s safe to say that ROI measurement hasn’t been a major priority.

The other intriguing figure in this survey was that 55% of respondents said there was no feature/function that they'd like added to their social media platform. REALLY? They can't be trying very hard: I mean, I can think of features I’d like added to a light bulb.*

If people are satisfied with their tools in such a rapidly evolving space, they probably aren’t using them for much. Or, to put it more charitably, maybe they recognize that they’re not taking advantage of what’s already available and feel they should master that before looking for anything more. Either way, this suggests that most deployments are quite immature.

One final factoid: 61% are integrating social media within their Web site or other sites, vs. 40% running standalone community sites and 39% deploying as social widgets in third party sites such as Facebook. I’m surprised that community sites and widgets are so popular. Maybe these are signs of experimentation. Anyway, it’s food for thought.

My general take, then: the survey shows wide testing of social technologies, but little deep engagement. Without a firm economic or other justification, there’s a good chance that the efforts won’t be sustained. So it’s up to social technology gurus, and vendors like Mzinga, to start demonstrating not just what social technology can do, but what makes it worth an investment.

__________________________________________
* How about an indicator that shows how long until it burns out? Preferably with a wireless Internet connection that alerts me when failure is imminent.

CMO Council Study: Customer Loyalty Is Fleeting

The CMO Council and Catalina Marketing’s Pointer Media Network recently released a major study on consumer loyalty in packaged goods brands. The study, Losing Loyalty: The Consumer Defection Dilemma™, draws on Catalina’s vast loyalty card transaction database to analyze the individual buying patterns of more than 32 million consumers in 2007 and 2008 across 685 leading CPG brands.

The bottom line is that “loyal” consumers are not as reliable as most of us might have guessed. “For the average brand in this study, 52% of highly loyal consumers in 2007 either reduced loyalty or completely defected from the brand in 2008.” You can download the 12 page report for details.

Not surprisingly, the report proposes to use individualized targeting services like Pointer Media Network to reduce churn by making carefully selected offers to at-risk consumers. Although the recommendation is obviously self-serving, I do think it’s correct.

But it seems to me that the implications are more fundamental. In the eternal debate about brand value, finding that loyalty evaporates more quickly than expected makes it even harder to justify marketing programs that don’t bring about an immediate, measurable return.

I’ve seen arguments (sorry, I can’t recall where) that the traditional buying model of awareness – interest – trial – purchase doesn’t correspond to reality. The survey results seem consistent with that position, in that they present consumer behavior as much less predictable than expected. This further reinforces the idea that investments with short-term results are more reliable than the long-term investments traditionally associated with brand building.

Pardon the cliche, but what we’re talking about here is a paradigm shift. If consumers don’t follow a predictable buying pattern, then brand value models based on such a pattern are not justifiable. Marketers need a fundamentally new framework to predict how their activities will affect consumer behavior. This framework may owe more to chaos theory than a linear process flow. I don’t know what they new model will look like, but recognizing that one is necessary is the first step towards creating it. If anybody out there has some candidates to offer, I’d love to hear about them.

Two More Surveys Confirm that Most Marketers Don't Track ROI

The Sales Lead Management Association and Velos Group published their annual lead management practices survey last week. (Read it here; free registration required.) The survy had a relatively small sample (just over 140 responses) and was weighted towards smaller companies (80% had fewer than 25 sales reps). But it still provides some insight into how many companies actually do business.

The key finding from a marketing measurement viewpoint was that 62.5% of respondents do not track ROI on marketing programs. This is not especially surprising; in fact, it’s better than the 76% reporting they do not use ROI another, larger study released last week by Lenskold Group. (Click here for the Lenskold study.) But it’s still bad news.

Perhaps even more distressing is that just 19.3% of the respondents listed their inability to track ROI as a major sales lead management concern. Subtracting those from the 62.5%, this means that more than 40% were not particularly concerned about their failure to track ROI. It MIGHT also mean that many of those 40% actually could track ROI if they wanted to, although it’s more likely that most don’t have the capability but don’t consider that a problem.

The other findings from the survey also generally confirm that dismal state of the art, at least among smaller firms.

- More than half the respondents (55.5%) said they do not qualify their marketing inquiries before sending them to sales. This implies a huge waste of time by salespeople who then do the qualifications themselves, or, more likely, cherry pick the leads that look superficially promising and ignore the rest. Unless a company has managed to staff its sales team with clairvoyants, this is a guarantee that it will discard some good leads and spend more than it should on some bad ones.

- One-third (33.8%) don’t use sales automation or customer relationship management systems. Again, this is fundamental efficiency-killer. The survey also found that companies using these systems were not terribly satisfied with the results: 54% rated their satisfaction at 5 or less on a scale of 1 to 10. Maybe the problem is with the software itself, but I suspect the issue is lack of training and other supporting investments.

- Half had no formal sales forecasting process (27.2%) or used Excel only (23.8%). Again, this shows very immature sales management at these companies.

I must say I find these results quite sad, given how long these tools have been available and how well their benefits are established.

But perhaps it’s best to adopt the more positive attitude of the survey authors and see this as an opportunity. As they put it, respondents “have a lot of room for improvement in their sales and marketing best practices. By spending time and resources in this critical business area, companies will be able to increase sales, allocate marketing resources more efficiently and will be able to forecast their sales more accurately. All of which will help them survive these difficult economic times.”

Whopper Freak-Out Wins Ad Effectiveness Award

I received a mailing with the agenda for the Association of National Advertisers' Marketing Accountability and Effectiveness Conference in New York on June 2. This looks like a good event, covering all the usual-but-useful bases: proving the value of marketing (Enterprise Rent-a-Car), earning a place at the "C-suite" table (panel led by Ernst & Young), advanced analytics (VG Corporation) and media optimization (Citizens Bank).

But my favorite is an "EFFIE" Award for Burger King, for its "Whopper Freak-out" campaign, which "explored deprivation to see what would happen if America's most beloved burger was removed from the menu forever without any announcement." Since I avoid both television and Burger King, this was news to me, but I gather a bunch of TV commercials were involved. Interesting.