Rate This Neutral: Scout Labs Social Media Monitoring is Definitely Cool, Possibly Accurate

Sure I like flashing lights and buzzers: what technologist doesn’t? And if a product has all that plus a low price, it’s darn near irresistible. So I was quite excited when I saw Scout Labs, a very nicely packaged social media monitoring tool that combines automated search, sentiment identification, importance ranking, trend reporting, alerts, bookmarking, and collaboration for under $300 per month. What’s not to like?

A couple things, it turns out. But let’s look at the good stuff first. Scout Labs’ combines three of the five social media measures I proposed last week (tracking mentions, identifying mentioners, measuring influence, understanding sentiment and measuring impact). Specifically, it searches blogs, news feeds, video and photo sites, Twitter and some social network sites (although not yet the big ones); provides influence measures; and classifies blog posts by sentiment. It doesn’t attempt to identify mentioners (i.e., track multiple posts by the same individual), or to measure the impact of an item on its audience. But three out of five is pretty good.

More important, the things that Scout Labs does, it does well. The search feature lets users specify multiple terms and whether each term is required, relevant or excluded. Once a search is defined, the system will automatically scan the top 12 million blogs for qualified entries, rate their sentiments as positive, negative or neutral, and show them in a list. Each item on the list shows the blog headline and phrases with the search terms highlighted. A side box shows common words in all the entries, ranked by frequency. This by itself gives a quick view of what’s being said about the search target.

Users can drill into the listed items to see the full entry, details about where it came from, how many external links attach to the item and its source, and the sentiment rating. They can manually revise the rating, bookmark the item with keywords, attach a note for discussion, and email a link with a system-generated summary and the user’s own comments to anyone the user chooses. The system currently uses the link counts as an influence measure, and can rank the items by influence or date. Scout Labs is working to upgrade its influence metric by integrating Web traffic data and a measure of the source’s relevance to the search topic.

But there’s more. The system can prepare graphs showing trends in volume, sentiment, and share of total blog mentions. Graphs can compare statistics for up to four different searches. Users can specify the date ranges to report on, currently going back up to three months and soon extending to six months.

Things are a little less exciting once you move beyond the blogosphere. The system will list search results for photo sites, video sites and Twitter, but doesn’t offer sentiment tracking or graphs. Scout Labs is working on adding sentiment tracking to Twitter comments. I guess it's not fair to ask them to measure sentiment for photos or videos.

As to pricing, the smallest Scout Labs plan allows five saved searches for $99 per month, although the company thinks expects most businesses will take plans for 25 or more searches, which start at $249 per month. There are no limits on the number of users or search hits in any plan and the system continuously updates the results of the saved searches.

So far so good. There's a free 30 day trial, so I set up two test searches in Scout Labs, each for a demand generation software vendor I track closely. The system found many of the posts I expected, and it was definitely fun and convenient to dig into them. If I worked at one of those firms, I would gladly pay $249 per month for this.

But then I ran the same searchs in IceRocket, a free tool that also does searches of blogs and other sources. IceRocket found nearly twice as many hits during the same time period, and they looked legitimate. Ouch. But Scout Labs acknowledges that its 12 million blogs don’t cover the entire blogsphere (over 100 million blogs, last I heard), and it does let you add feeds if one you want is missing. Plus IceRocket doesn’t support saved searches or do any of the other cool stuff. So I’m a little worried about coverage but still willing to pay Scout Labs’ fee.

Next I took a closer look at the sentiment ratings in the Scout Labs results. I didn’t expect them to be perfect, but was seriously disappointed. On one search, 32 of 39 items were labeled as neutral. Some of those were actually pretty positive, but, as Scout Labs explains in a recent blog post, they try to be conservative by labeling items as neutral unless the tone is clear. Fair enough. But the seven positive items were all pretty much neutral too. For example, several were help wanted postings that simply specified experience with the products in question. There were no items classified as negative, although one or two of the posts arguably could have been.

In the blog post I just mentioned, Scout Labs offers a detailed discussion of its sentiment rating technique. The gist is that they don’t just count “happy” and “sad” words, but semantically analyze each entry to understand which words relate to the search topic. Sounds good in theory. They also say their automated ratings agree with college-educated humans about 75% of the time. In comparison, they say, college-educated humans agree with each other about 85% of the time. (Clearly they are not talking about married couples.)

But if the vast majority of items are neutral, that’s less useful than it sounds. Remember the basic statistics: if 80% of the items are neutral, then a system that blindly ranks everything as neutral will be correct 80% of the time. The ratings that really count are the positives and negatives, and I wonder how often a human would agree with those ratings in Scout Labs. I’d want to look at that much more closely before deciding whether to rely on Scout Labs' results.

I'd still pay for Scout Labs for the convenience of the searches, statistics and collaboration tools. As I say, it's a very nice interface. I might even find on closer examination that the sentiment ratings are useful even if they’re only somewhat accurate: after all, they might still get a trend right and call up useful samples. But much as I like the bells and whistles, I’m not as enthusiastic about Scout Labs as when I started.