Tuesday, November 18, 2008

Comparing Web Activity Measures for Demand Generation Vendors

I recently wanted to measure the relative popularity of several demand generation vendors, as part of a deciding how to expand the Raab Guide to Demand Generation Systems. This led to an interesting little journey which I think is worth sharing.

I started with a list of 23 marketing system vendors. A couple are fairly large but most are quite small. These were grouped into three categories: five demand generation vendors already in the Guide; four marketing automation vendors with significant demand generation market presence; and fourteen other demand generation vendors. (See http://www.raabguide.com/ for definitions of demand generation and marketing automation.)

My first thought was to look at their Web site traffic directly. The easiest way to do this is at Alexa.com, which tracks site visits of people who download its search toolbar. The number of users in this base is apparently a well-guarded secret, or at least well enough guarded that I would have had to look beyond the first Google search page for the answer. Alexa was originally classified by many experts as spyware, and is still somewhat controversial. But it was purchased by Amazon.com in 1999 and has since become more or less grudgingly accepted.

Be that as it may. I captured two statistics for each of my sites from Alexa: a ranking which basically reflects the number of pages viewed by unique visitors each month (the busiest site gets rank 1, next busiest gets rank 2, etc.); and a share figure that shows the percentage of total toolbar users who visit each site each month. (I think I have that correct; you can check the definitions at Alexa.com.) Ranking on either figure gives the same sequence (except for Pardot; I have no idea why). If you’re creating ratios or an index, the difference in the share figures is probably a better indicator of relative popularity, since a company with twice the share of another has twice as many visitors, but will not necessarily a rank number that is twice as low. (Lower rank means more traffic.)

Here are the ranks I came up with, broken into the three segments I mentioned earlier:

Alexa - 3 mo average
Already in Guide:rankshare
Eloqua20,2340.0070700
Silverpop29,0800.0030500
Marketo68,0880.0017000
Manticore Technology213,5460.0006100
Market2Lead235,2440.0004800
Vtrenz295,6360.0003600
Marketing Automation Vendors:
Unica / Affinium*126,2150.0008500
Alterian345,5430.0002500
Aprimo416,4460.0002200
Neolane566,9770.0001690
Other Demand Generation:
MarketBright167,3060.0005400
Pardot211,3090.0003600
Marqui *211,7670.0004400
ActiveConversion257,0580.0003400
Bulldog Solutions338,3370.0003200
OfficeAutoPilot509,8680.0002000
Lead Genesys557,1990.0001450
LoopFuse734,0980.0001090
PredictiveResponse2,313,8800.0000330
FirstWave Technologies2,872,7650.0000170
NurtureMyLeads4,157,3040.0000140
Customer Portfolios5,097,5250.0000090
Conversen*6,062,4620.0000070
FirstReef11,688,8170.0000010

These rankings were more or less as I expected. Within the first group, Eloqua is definitely the largest vendor, while Marketo is probably the most aggressive marketer at the moment. Vtrenz is the second-largest demand generation company, based on number of clients and almost certainly on revenue. But it is a subsidiary of Silverpop, so its traffic is split between Vtrenz.com and visits to Silverpop.com. This means that the Vtrenz.com ranking understates the company’s position, whie Silverpop ranking includes traffic unrelated to demand generation. I’ve therefore tracked both here. Manticore and Market2Lead get much less attention than the other three, so it makes sense that they have much less traffic.

Figures for the next group also seem to be ranked about correctly. Unica is certainly the most prominent of this group, with Alterian, Aprimo and Neolane trailing quite far behind. I would have expected a bit more traffic for Neolane, but it is definitely the new kid on this block and only entered the U.S. market about one year ago. The real surprise here is that this group as a whole ranks so far below the big demand generation vendors, even though the marketing automation firms are in fact larger and probably do more promotion. Perhaps the marketing automation vendors appeal to a smaller number of potential users (primarily, marketers in large companies with direct customer contact, such as financial services, retail, travel and telecommunications) and generate less traffic as a result.

I didn’t have much sense of the relative positions of the other demand generation vendors, although I would have guessed that MarketBright and Pardot were near the top. Marqui has had little attention recently, perhaps because they’ve been through financial difficulties culminating in the purchase of their assets by a private investor group this past August. ActiveConversions I do know, only because I’ve spoken with them, and they rank about where I expected given their number of clients. The other names were somewhat familiar but the only one I’d ever spoken with was OfficeAutoPilot, which I knew to be small. Since I had no fully formed expectations, the rankings couldn’t surprise me.

In other words, the rankings provided by Alexa seemed generally reasonable given my knowledge of the companies concerned.

But Web traffic is just one measure. Where else could I look to confirm or challenge these impressions?

Well, there is another Web traffic site that is somewhat similar to Alexa, called Compete.com. I actually hadn’t heard of them before but they came up in my research. They apparently use their own toolbar but also some other Web traffic measures such as volumes reported by Internet Service Providers (ISPs). You’d expect them to pretty much match the Alexa figures. But do they? Here is a chart comparing the two, with the Alexa multiplied by 10 ^7 to make them more legible.

Compete.comAlexa.com
unique visitors / monthshare x 10^7
Already in Guide:
Eloqua560,28870,700
Silverpop293,58030,500
Marketo34,24417,000
Manticore Technology15,7896,100
Market2Lead10,6894,800
Vtrenz5,3133,600
Marketing Automation Vendors:
Unica / Affinium*23,1388,500
Alterian4,4972,500
Aprimo5,1312,200
Neolane3,9271,690
Other Demand Generation:
MarketBright13,9935,400
Pardot7,3393,600
Marqui *3,2824,400
ActiveConversion1,5033,400
Bulldog Solutions6,4083,200
OfficeAutoPilot1,5672,000
Lead Genesys2,6301,450
LoopFuse1,9301,090
PredictiveResponse1,099330
FirstWave Technologies-170
NurtureMyLeads-140
Customer Portfolios-90
Conversen*-70
FirstReef-10

You don’t need Sherlock Holmes to spot the problem: the Compete.com figures for Eloqua and Silverpop seem much too high compared with the others. I could concoct a theory that this reflects the difference between counting unique visitors in Compete.com and counting page views in Alexa, and throw in the fact that Eloqua and Silverpop/Vtrenz host landing pages for their clients. But the other demand generation vendors also host their clients’ pages, so this shouldn’t really matter. I suspect what really happens is that Compete measures low volumes differently from higher volumes (remember, they use a combination of techniques), and thus the figures for high-volume Eloqua and Silverpop are inconsistent with figures for the other, much lower-volume domains.

Anyway, if we throw away those two, the rest of the Compete figures seem more or less in line with the Alexa figures, apart from some small exceptions (Bulldog in particular ranks higher). All told, it doesn’t seem that Compete adds much value to what I already got from Alexis.

So much for Web traffic. How about search volume? Google Keywords will give that to me. Again, we’ll compare to Alexa as a reference:

Google KeywordsAlexa
avg search volumeshare x 10^7
Already in Guide:
Eloqua

1,900

70,700

Silverpop

1,790

30,500

Marketo

839

17,000

Manticore Technology

113

6,100

Market2Lead

318

4,800

Vtrenz

752

3,600

Marketing Automation:

-

Unica / Affinium*

6,600

8,500

Alterian

861

2,500

Aprimo

1,600

2,200

Neolane

1,340

1,690

Other Demand Generation:

-

MarketBright

186

5,400

Pardot

210

3,600

Marqui *

1,300

4,400

ActiveConversion

46

3,400

Bulldog Solutions

442

3,200

OfficeAutoPilot

0

2,000

Lead Genesys

74

1,450

LoopFuse

260

1,090

PredictiveResponse

36

330

FirstWave Technologies

386

170

NurtureMyLeads

0

140

Customer Portfolios

0

90

Conversen*

170

70

FirstReef

12

10


If we limit ourselves to the first two groups, the search numbers look mostly plausible. The low figure for Manticore could have to do with checking specifically for “Manticore Technology”, since a looser “Manticore” would incorporate an unrelated company and references to the mythical beast. The high value for Unica probably reflects some unrelated uses of the word in other languages or as an acronym. I have no particular explanation for the relatively low value for Alterian or the substantial flattening of the range between Eloqua and its competitors. Perhaps Eloqua’s traffic is less search-driven than other vendors’. Or not. In any event, I think the implicit rankings here are about as plausible as the Alexa rankings.

But things get crazier in the Other Demand Generation vendor segment. I understand the Marqui number, which is high because Marqui can be a misspelling of other words (marquis, marque, marquee) and has some unrelated non-English meanings. Similarly, Conversen is a verb form in Spanish. I think that Bulldog Solutions, FirstWave and LoopFuse also gain some hits because of their component words, even though I tried to keep them out of the search results. The bottom line here is you have to throw away so many terms that the remaining rankings don’t signify much. So, in general, search keyword rankings need close consideration before you can accept them as a meaningful measure of importance.

How about Google hits? I’ll show them alongside the Google Keywords as well as Alexa rank.

Google hitsGoogle KeywordsAlexa
avg search volumeshare x 10^7
Already in Guide:
Eloqua

118,000

1,900

70,700

Silverpop

111,000

1,790

30,500

Marketo

103,000

839

17,000

Manticore Technology

9,620

113

6,100

Market2Lead

25,900

318

4,800

Vtrenz

35,200

752

3,600

Marketing Automation:

-

Unica / Affinium*

7,750

6,600

8,500

Alterian

262,000

861

2,500

Aprimo

161,000

1,600

2,200

Neolane

40,200

1,340

1,690

Other Demand Generation:

-

MarketBright

34,500

186

5,400

Pardot

27,600

210

3,600

Marqui *

1,370,000

1,300

4,400

ActiveConversion

16,800

46/p>

3,400

Bulldog Solutions

9,340

442

3,200

OfficeAutoPilot

777

0

2,000

Lead Genesys

5,880

74

1,450

LoopFuse

95,400

260

1,090

PredictiveResponse

21,800

36

330

FirstWave Technologies

13,400

386

170

NurtureMyLeads

1,050

0

140

Customer Portfolios

12,200

0

90

Conversen*

2,790

170

70

FirstReef

18,100

12

10


Here the impact of limiting Manticore to “Manticore Technology” shows up even more clearly (although Manticore truly doesn’t get much Web attention). I limited the Unica test to “Unica Affinium” since the number of hits is otherwise over 100 million; but this seems to excessively depress the results. Note that the low ranking for Alterian has now been reversed; in fact, Alterian has the most hits of all, and the marketing automation group in general shows more activity than the demand generation vendors. That could be true – those vendors have been around longer. Or it could be a fluke.

Once again, the Other Demand Generation group has a big problem with Marqui and perhaps smaller problems with LoopFuse and FirstReef. Even excluding those, the numbers jump around a great deal. As with keywords, these figures don’t seem to be a reliable measure of anything.

Let’s try one more measure: the blogosphere. Here I tried three different services: Technorati, BlogPulse and Ice Rocket.

TechnoratiBlogpulseIce RocketAlexa
blog postsblog postsall postsshare x 10^7
Already in Guide:
Eloqua

130

267

286

70,700

Silverpop

70

119

188

30,500

Marketo

3

179

229

17,000

Manticore Technology

0

12

56

6,100

Market2Lead

0

7

25

4,800

Vtrenz

0

30

53

3,600

Marketing Automation:

-

Unica / Affinium*

0

6

43

8,500

Alterian

8

119

145

2,500

Aprimo

0

118

139

2,200

Neolane

0

33

64

1,690

Other Demand Generation:

-

MarketBright

1

23

33

5,400

Pardot

0

32

33

3,600

Marqui software*

5

15

19

4,400

ActiveConversion

0

6

12

3,400

Bulldog Solutions

0

30

43

3,200

OfficeAutoPilot

0

5

5

2,000

Lead Genesys

0

1

5

1,450

LoopFuse

4

48

43

1,090

PredictiveResponse

0

0

0

330

FirstWave Technologies

0

5

11

170

NurtureMyLeads

0

1

5

140

Customer Portfolios

0

0

3

90

Conversen*

0

2

0

70

FirstReef

0

0

0

10



Results for all three services are roughly consistent, although Technorati gets many fewer hits and Ice Rocket finds a few more than Blogpulse. The major anomaly is the low value for Unica, but that happens because I actually searched on Unica Affinium, to avoid all the irrelevant hits on Unica alone. Similarly, I searched on Marqui Software to avoid unrelated hits on Marqui. The high values for Bulldog Solutions and Loopfuse are valid (I scanned the actual hits); these two vendors just managed to snag a relatively high number of blog mentions. Remember we are looking at very small numbers here: it doesn’t take much to get 40 blog mentions. Nor, if we trust the Alexa, do they translate into much Web traffic. However, the blog hits might explain the relatively high keyword search counts for those two vendors.

Well, I hope you enjoyed the trip. This is far from an exhaustive analysis of the issue, but based on the information available, I’d say that Alexa Web traffic is the most useful measure for assessing the market presence of different demand generation vendors, and blog mentions have at least some value. Google hits and keyword searches capture too many unrelated items to be reliable.

7 comments:

Dharmesh said...

Very interesting analysis. Thanks for taking the time.

You might also find http://website.grader.com to be a good metric to use as it blends a variety of factors into a weighted score.

David Raab said...

Thanks Dharmesh. I ran a test report and liked website.grader quite a bit.

Chad H said...

You can also measure traffic on Twitter with Tweet volume

Landon said...

Hey David,

Interesting obsession you've developed.. :)

All these traffic numbers are cool, but I'm not clear that they reflect anything that you might be looking for.

Is it market interest? Or number of clients?

Several things muck up your study. First, we get several times more traffic than you're reporting.. my guess is that's because we host landing pages on several subdomains that generate a lot of traffic which your tools aren't counting? Don't know. The same might be true for all the companies you've listed that host landing pages (which is only a few, I believe)

Also, raw traffic is much more a function of marketing spend than anything.. and is certainly not a proxy for number of clients. Marketo and Pardot, for example, publish their client counts and although we don't, I can promise that any disparity between theirs and ours is not reflected in any of the numbers you've got here.

Our sales primarily come from referrals through our partner program. While that doesn't generate a lot of traffic, it's certainly far more qualified and valuable than the traffic generated by a banner ad.

As an industry participant, I'd guess that you've nailed the top few players right.. also that Pardot is doing well. Beyond that though, my guess is that you've got your list pretty mixed up.

Instead of tooting our own horn, I'll point out that Bulldog Solutions (who's actually not a player in this industry.. they're more like an agency and they deploy Eloqua when marketing automation is called for) has gotten much more press and is far more 'talked about' than your list suggests.

Our own case shows that your blog research is way off.. you personally have written about us more times than your research has indicated as a total.

All in all, the conclusion I'd come to based on the research you've done is that it's real tough to figure out what's what based on very flawed tools like alexa etc.

For more on why Alexa is nearly useless:

http://www.e-consultancy.com/forum/101280-how-accurate-is-alexa-for-traffic-measurement.html

http://forums.seochat.com/alexa-ranking-49/how-accurate-is-alexa-4536.html

Am glad to see your interest in our market though!

Cheers,
Landon

David Raab said...

Thanks Chad. I took a closer look at this and published a separate post on the topic.

David Raab said...

Hi Landon,

Good to hear from you. I was looking for way to assess the presence of the vendors in the market, which basically boils down to the amount of attention they are receiving. Remember, the ultimate goal was to decide which vendors to add to the Guide, and I want to include the products that people are most interested in.

Customer counts are self-reported, which makes them inherently unreliable. More important, some customers are a lot bigger than others, so the customer counts don't correlate very well with revenue or market share. In any event, a measure of "market presence" should certainly include hot new vendors who are getting a lot of attention even if they don't have many clients yet.

I'm quite aware of the criticisms of Alexa. But as explained in the post, the Alexa figures came to closest to giving a plausible ranking. It's the "least worst" solution, nothing more.

David Raab said...

Incidentally, I had another interesting suggestion to consider measuring the number of keyword ads targeting a given term, and the price that people are willing to pay for them.