eMarketer recently released a deeply researched report on Online Brand Measurement. Since it touched on several topics I’ve been pondering recently (see Web Analytics Is Dead… on my Customer Experience Matrix blog) , I read it with particular care.
This is a long report (58 pages), so I won’t review it in detail. But here are what struck me as the critical points:
- Web measurement has largely focused on counting views and clicks, not measuring long-term brand impact. Counting is much easier but it doesn’t capture the full value of any Web advertisement. One result has been that marketers overspend on search ads, which are great at generating immediate response, and underspend on Web display ads which influence long term behavior even if they don’t generate as many click-throughs.
- Media buyers want Web publishers to provide the equivalent of Gross Rating Points (GRPs), so they can effectively compare Web ad buys with purchases in other media. That’s okay as far as it goes, but it’s still just about counting, not about measuring the quality or impact of the impressions. As the paper points out, even engagement measures such as time on site or mentions in social media, don’t necessarily equate to positive brand impact.
- Just about everyone agrees that the right way to measure brand impact is to tailor measurements to the goal of a particular marketing program. This may sound like a conflict with the desire for a standard GRP-like measure, but it really reflects the distinction between counting the audience and measuring impact. GRPs work fine for buying media but not for assessing results. Traditional media face precisely the same dichotomy, which is why marketing measurement is still a puzzle for them as well. And just as most offline brand measures are ultimately based on surveys and panels, I'd expect most online brand measures will be too.
- Meaningful impact measurement will integrate several data types, including online behaviors, visitor demographics, offline marketing activities and actual purchase behavior. These will come from a combination of direct online sources (i.e., traditional Web analytics), panel-based research and surveys (for audience and attitudinal information), and offline databases (for demographics and purchases). Ideally these would be meshed within marketing mix models and response attribution models that would estimate the incremental impact of each marketing program and allow optimization. But such sophisticated models won’t appear tomorrow.
To me, this final point is the most important because it points to a “grand unification theory” of marketing measurement that combines the existing distinct disciplines and sources. The paper cites numerous current efforts, including:
- multimedia databases being created (separately) by panel-based measurement firms including comScore, Nielsen, Quantcast and TNS Media Compete;
- Datatran Media’s Aperture, which combines email and postal addresses with Acxiom household data, IXI financial data, MindSet Marketing healthcare data and NextAction retail data;
- a joint effort between Omniture and WPP’s Kantar Group that combines data from email, search, display ads and traditional media;
- another Nielsen project combining TV ad effectiveness information from Nielsen IAG with panel purchase data from Nielsen Homescan.
These all reinforce the claim I made in last week’s blog post that individual data will increasingly be combined with panel- and survey-based information to provide community-level insights that are actually more valuable than individual data alone.
Tuesday, September 29, 2009
Friday, September 25, 2009
ANA Agency/Client Forum: Agency Performance Isn't Based on Results
I spent most of yesterday at the Association of National Advertisers (ANA)’s Agency/Client Forum. The agenda covered a range of timely topics including digital advertising and social media. But I think it’s fair to say that most of energy was focused on the pocketbook issue of agency compensation.
In particular, the question du jour was value-based compensation or its cousin, pay-for-performance. I would have thought those were pretty much the same thing, but as Coca Cola’s Director of Worldwide Media & Communication Operations Sarah Armstrong set out in a detailed description of Coke’s own process, value-based compensation works largely by estimating a reasonable cost in advance, while pay-for-performance is based on after-the-fact assessments. (Coke’s process incorporates both – a base fee that is intended cover agency costs, plus up to 30% bonus based on performance.)
However, as several speakers made clear during the day, most pay-for-performance measures are based on agency behaviors such as innovation, strategic thought and execution, rather than business results such as sales, market share or even communications activities such as media cost savings. Specifically, an ANA survey that will be formally released in mid-October found that 56% of agency performance measures were based on qualitative metrics, vs just 19% on business results and 25% on communications metrics.
My initial reaction to this survey was pretty dismissive – either you pay for results or you don’t. But the fundamental rationale, mentioned by several speakers through the day, is that business results are affected by many factors beyond the agency’s control, so it really wouldn’t be fair to penalized or reward them purely on that basis. The analyst in me says it’s still worth trying to isolate the agency's contribution to results, but this is definitely a valid point. So including the subjective measures does make more sense than I initially thought.
A related question that ran through several presentations was whether agencies are commodities. One presenter flashed a survey that showed 80% of respondents thought they are (I didn’t capture the details of that survey, but think it was an informal online poll, so it’s probably not very meaningful). But both the clients and agencies among conference speakers felt strongly that they are not.
What was interesting, though, was the sorts of distinguishing features that speakers cited – strategic insights, brand stewardship, creative genius, etc. Those are based largely on the skills and chemistry of the individuals working on an account. As the cliche says, those assets “go down the elevators every night” – that is, they are individuals rather than property of the agency itself. So it’s possible that the agencies themselves are pretty much commodities (i.e., have about the same processes and technology) even if their people are different.
And even when it comes to people, I find it hard to believe that any one agency can really have people who are on average much better than any other agency. There are, in fact, plenty of smart and creative people in the world. Yes, there are occasional true geniuses, and clients lucky enough to find them working on their accounts may indeed gain a strategic advantage. Perhaps some of those geniuses are even so clever that they can build an entire culture around themselves to leverage their skills. But I'd say that level of genius is very much the exception.
In general, then, I suspect that once a quality agency comes up to speed, it would produce roughly similar results to another quality agency. This doesn't mean that you could immediately switch from one to another. But over the long term, agencies probably are something close to a commodity.
This relates back to the performance measurement questions. The value of an agency really does lie in its strategic, creative, and execution contributions, plus its ability to work closely with the client. In theory, most agencies should be able to do these equally well. But in fact, there will be variations based on the individual team members as well as (to a lesser degree, I think) differences in agency processes and culture. So it makes sense for performance evaluations to focus on those factors, even though they’re subjective. Marketers must measure those factors to identify areas needing improvement, either by changing performance of their current agency partners or switching to new ones.
In particular, the question du jour was value-based compensation or its cousin, pay-for-performance. I would have thought those were pretty much the same thing, but as Coca Cola’s Director of Worldwide Media & Communication Operations Sarah Armstrong set out in a detailed description of Coke’s own process, value-based compensation works largely by estimating a reasonable cost in advance, while pay-for-performance is based on after-the-fact assessments. (Coke’s process incorporates both – a base fee that is intended cover agency costs, plus up to 30% bonus based on performance.)
However, as several speakers made clear during the day, most pay-for-performance measures are based on agency behaviors such as innovation, strategic thought and execution, rather than business results such as sales, market share or even communications activities such as media cost savings. Specifically, an ANA survey that will be formally released in mid-October found that 56% of agency performance measures were based on qualitative metrics, vs just 19% on business results and 25% on communications metrics.
My initial reaction to this survey was pretty dismissive – either you pay for results or you don’t. But the fundamental rationale, mentioned by several speakers through the day, is that business results are affected by many factors beyond the agency’s control, so it really wouldn’t be fair to penalized or reward them purely on that basis. The analyst in me says it’s still worth trying to isolate the agency's contribution to results, but this is definitely a valid point. So including the subjective measures does make more sense than I initially thought.
A related question that ran through several presentations was whether agencies are commodities. One presenter flashed a survey that showed 80% of respondents thought they are (I didn’t capture the details of that survey, but think it was an informal online poll, so it’s probably not very meaningful). But both the clients and agencies among conference speakers felt strongly that they are not.
What was interesting, though, was the sorts of distinguishing features that speakers cited – strategic insights, brand stewardship, creative genius, etc. Those are based largely on the skills and chemistry of the individuals working on an account. As the cliche says, those assets “go down the elevators every night” – that is, they are individuals rather than property of the agency itself. So it’s possible that the agencies themselves are pretty much commodities (i.e., have about the same processes and technology) even if their people are different.
And even when it comes to people, I find it hard to believe that any one agency can really have people who are on average much better than any other agency. There are, in fact, plenty of smart and creative people in the world. Yes, there are occasional true geniuses, and clients lucky enough to find them working on their accounts may indeed gain a strategic advantage. Perhaps some of those geniuses are even so clever that they can build an entire culture around themselves to leverage their skills. But I'd say that level of genius is very much the exception.
In general, then, I suspect that once a quality agency comes up to speed, it would produce roughly similar results to another quality agency. This doesn't mean that you could immediately switch from one to another. But over the long term, agencies probably are something close to a commodity.
This relates back to the performance measurement questions. The value of an agency really does lie in its strategic, creative, and execution contributions, plus its ability to work closely with the client. In theory, most agencies should be able to do these equally well. But in fact, there will be variations based on the individual team members as well as (to a lesser degree, I think) differences in agency processes and culture. So it makes sense for performance evaluations to focus on those factors, even though they’re subjective. Marketers must measure those factors to identify areas needing improvement, either by changing performance of their current agency partners or switching to new ones.
Tuesday, September 1, 2009
Mzinga Survey Shows Most Companies Don't Measure Social Media ROI
Toute le blogosphere is in love with social media, which of course means that some contrarians have to argue that it’s over-hyped. So it was interesting to see a survey (available here; registration required) show that social technologies are indeed widely adopted: 86% of 555 respondents said they are currently using them for business purposes, and 61% said it was an ongoing component of their business.
Caveat: the survey was sponsored by social technology vendor Mzinga in conjunction with the Babson Executive Education program, so they had a stake in the outcome. But I didn't see any obvious problems with it, and even allowing for some bias, the results still suggest wide social technology usage among a broad spectrum of businesses.
Probably the most interesting result from a marketing measurement perspective was that just 16% of respondents reported measuring ROI on their social media programs. No surprise, alas, but worrisome because programs that can’t prove ROI are subject to cancellation when money is tight.
Somewhat supporting this line of reasoning, the survey showed that just 40% of respondents had budget dedicated to social media and 57% had employees assigned to it. Perhaps many of those employees work for free, but a more likely explanation is that their costs are not part of project budgets because they're part of a vaguely fixed "overhead". This makes it easier to sustain a social media effort without formal economic justification. But it can’t be a permanent situation – managers will eventually realize that time spent on social media has a real cost. So justification of some sort will ultimately be needed.
Of course, that justification won’t necessarily be ROI. We all know that many traditional marketing investments are not justified on the basis of ROI, and marketing is by far the most common social media application (57%, vs 39% for internal collaboration, 31% other, 29% customer service & support, 25% sales, 21% human resources, 16% strategy and 14% product development). Marketing in social media could easily go unmeasured as well.
Indeed, just 8% of respondents said their social technology system could showcase ROI, vs. 41% who said it couldn’t. An impressively large 44% didn’t know, which I interpret to mean that they didn't care enough to find out. So I think it’s safe to say that ROI measurement hasn’t been a major priority.
The other intriguing figure in this survey was that 55% of respondents said there was no feature/function that they'd like added to their social media platform. REALLY? They can't be trying very hard: I mean, I can think of features I’d like added to a light bulb.*
If people are satisfied with their tools in such a rapidly evolving space, they probably aren’t using them for much. Or, to put it more charitably, maybe they recognize that they’re not taking advantage of what’s already available and feel they should master that before looking for anything more. Either way, this suggests that most deployments are quite immature.
One final factoid: 61% are integrating social media within their Web site or other sites, vs. 40% running standalone community sites and 39% deploying as social widgets in third party sites such as Facebook. I’m surprised that community sites and widgets are so popular. Maybe these are signs of experimentation. Anyway, it’s food for thought.
My general take, then: the survey shows wide testing of social technologies, but little deep engagement. Without a firm economic or other justification, there’s a good chance that the efforts won’t be sustained. So it’s up to social technology gurus, and vendors like Mzinga, to start demonstrating not just what social technology can do, but what makes it worth an investment.
__________________________________________
* How about an indicator that shows how long until it burns out? Preferably with a wireless Internet connection that alerts me when failure is imminent.
Caveat: the survey was sponsored by social technology vendor Mzinga in conjunction with the Babson Executive Education program, so they had a stake in the outcome. But I didn't see any obvious problems with it, and even allowing for some bias, the results still suggest wide social technology usage among a broad spectrum of businesses.
Probably the most interesting result from a marketing measurement perspective was that just 16% of respondents reported measuring ROI on their social media programs. No surprise, alas, but worrisome because programs that can’t prove ROI are subject to cancellation when money is tight.
Somewhat supporting this line of reasoning, the survey showed that just 40% of respondents had budget dedicated to social media and 57% had employees assigned to it. Perhaps many of those employees work for free, but a more likely explanation is that their costs are not part of project budgets because they're part of a vaguely fixed "overhead". This makes it easier to sustain a social media effort without formal economic justification. But it can’t be a permanent situation – managers will eventually realize that time spent on social media has a real cost. So justification of some sort will ultimately be needed.
Of course, that justification won’t necessarily be ROI. We all know that many traditional marketing investments are not justified on the basis of ROI, and marketing is by far the most common social media application (57%, vs 39% for internal collaboration, 31% other, 29% customer service & support, 25% sales, 21% human resources, 16% strategy and 14% product development). Marketing in social media could easily go unmeasured as well.
Indeed, just 8% of respondents said their social technology system could showcase ROI, vs. 41% who said it couldn’t. An impressively large 44% didn’t know, which I interpret to mean that they didn't care enough to find out. So I think it’s safe to say that ROI measurement hasn’t been a major priority.
The other intriguing figure in this survey was that 55% of respondents said there was no feature/function that they'd like added to their social media platform. REALLY? They can't be trying very hard: I mean, I can think of features I’d like added to a light bulb.*
If people are satisfied with their tools in such a rapidly evolving space, they probably aren’t using them for much. Or, to put it more charitably, maybe they recognize that they’re not taking advantage of what’s already available and feel they should master that before looking for anything more. Either way, this suggests that most deployments are quite immature.
One final factoid: 61% are integrating social media within their Web site or other sites, vs. 40% running standalone community sites and 39% deploying as social widgets in third party sites such as Facebook. I’m surprised that community sites and widgets are so popular. Maybe these are signs of experimentation. Anyway, it’s food for thought.
My general take, then: the survey shows wide testing of social technologies, but little deep engagement. Without a firm economic or other justification, there’s a good chance that the efforts won’t be sustained. So it’s up to social technology gurus, and vendors like Mzinga, to start demonstrating not just what social technology can do, but what makes it worth an investment.
__________________________________________
* How about an indicator that shows how long until it burns out? Preferably with a wireless Internet connection that alerts me when failure is imminent.
Subscribe to:
Posts (Atom)
