Google Keyword Planner's Dirty Secrets

Posted by rjonesx.

Sometimes our best data sources aren't exactly up to par. While nearly every search marketer will rely on Google Keyword Planner data at one point or another, especially while doing keyword research, the reality is that the data is often untrustworthy and should be viewed with great skepticism. Whether you plan to use it to help build a paid search campaign or determine which content to write, there are huge caveats to the numbers presented as Average Search Volume. Today, I want to walk through a number of the "gotchas" in Google Keyword Planner data so you can do better keyword research and make smarter decisions for you or your clients' sites.

Dirty secret #1: Rounded averages

By far, the most-used piece of data from Google Keyword Planner is the "Average Monthly Search Volume" metric. This key data point is used in everything from basic decisions on what keywords to use in an ad campaign to complex traffic prediction curves. But can we trust it?

Suppose you run a sports website and two keywords pop up in the recommendations: baseball scores and basketball games. Google Keyword Planner lets us know that each of these keywords has an Average Monthly Search Volume of 201,000. At first glance, you should be able to choose either of these keywords and expect similar traffic results, right?

Wrong. The "Average Monthly Search Volume" is more than just an average; it's rounded to the nearest-volume-bucket (which I will describe later). We know this is the case because Google Keyword Planner also exposes the last 12 months of traffic data. If we average that data, we will see that baseball scores receives 217,275 visits per month, while basketball games averages only 205,750! That is a difference of over 10,000 searches per month, which is obscured by Google KWP's rounding algorithm.

When we took a sample of keywords at the 201,000 Average Monthly Search volume, the standard deviation was 14,621 in the "actual average." In some cases, it was off by over 40,000 monthly searches per month! If you don't look at the last 12 months of data, your annual traffic estimates will likely be off by tens of thousands of visits. What causes this anomaly?

Dirty secret #2: Traffic buckets

Google Keyword Planner uses "buckets" to group keywords by traffic volume. When a keyword returns a traffic volume of 201,000, it isn't because the keyword was actually visited that many times, or really that it was particularly close to the number 201,000, but just that it was closer to 201,000 than the next biggest bucket of 246,000. The next lower bucket is 165,000, which gives us a nice 80,000-searches-per-month wiggle room — within which a keyword might actually fall and still be categorized as 201,000 by Keyword Planner.

After analyzing a massive data set, we found that Google has around 85 different "buckets" for traffic, which are logarithmically proportioned. This means that long tail keywords might fall into buckets which only differ by 10–20 searches at a time, while head tail keywords might see gaps of hundreds of thousands of searches per month. The bigger the search volume, the less certain you can be about the accuracy of the Average Monthly Searches, especially relative to other terms that fall in the same group. In fact, the largest buckets have variances of of nearly a quarter million searches per month!

Google uses this rounding procedure for convenience and, likely, to take into account the real month-to-month variance which can be huge for these very popular terms.

Dirty secret #3: Hidden keywords

Rand had an excellent write up on this issue a while back if you want to read the full details or want a more in-depth look at the problem. However, I thought I'd just throw out some stats here to show you just how ridiculous the recommendation system can be relative to the reality of related words and phrases. Let's start with the phrase "football." In this example, we will start with using GrepWords data to find the most valuable words that contain "football" in them. Then, we simply ask Google what they recommend. How close do they match? What is missed?

The top 3 most-trafficked football-based keywords weren't recommended to us, and only 4 of Google's recommended made it into the top 10. In fact, when we analyzed dozens of Google keyword recommendation reports, we found that only 35% of the keywords were among the most trafficked terms.

It appears that Google Keyword Planner is simply trying to provide a diverse cross-section of terms, but for marketers it means you potentially miss out on huge opportunities unless you dig much deeper. You can battle back against this "feature" by choosing more short-tail terms to seed your searches and setting volume and CPC limits, as the recommendations get stronger and stronger the more specific you get. In the end, though, you're going to miss out on some great terms if you've restricted your research to only Google Keyword Planner.

Dirty secret #4: Combination inconsistencies

If you're like me and spelling isn't your forte, you have certainly seen Google give you the "showing results for {correct spelling}." This is very useful for the searcher, but throws a pretty big wrench into keyword volume metrics. What does Google do in these situations? Does it count all the traffic towards correctly spelled keyword (which is actually showing in the search results) or does it count the traffic toward the misspelling or variation? Well, it turns out it's a mixed bag. Let's take a look at a fairly popular term Texas A&M Football.

In the above picture we see several variations of how one might search for the concept Texas A&M Football.

Keyword	Corrected?	Distinct Volume
Texas A&M Football	No	Yes
Texas A and M Football	No	Yes
Texas AM Football	Yes	Yes
Texas A & M Football	No	Yes
Texas A& M Football	Yes	Yes

Notice that whether or not the keyword is mapped to the canonical spelling makes no difference, in this case, for the total search volume. Even though many keywords will show you Texas A&M results, Google's volume count is only for the correct spelling of the term.

Now here's where it starts to matter. Let's say that you run a site that sells football attire and you're deciding which schools to include. You look up Google's Keyword Planner data and see that "Texas A&M Football" and "FSU Football" are both searched 201,000 times a month. These keywords seem equal in terms of volume but, in reality, there are many more keywords that are mapped organically to the phrase "Texas A&M Football," which makes its combined search volume much higher. In this particular case, there are several thousand visitors a year that you might miss out on by choosing "FSU Football" over "Texas A&M Football" simply because Google doesn't combine the keywords in Keyword Planner despite doing so in organic search.

This might seem like a reasonable compromise. The Keyword Planner is giving you back the search counts for the keywords, regardless of whether those searches are redirected to a different phrase. This would be appropriate if it was consistent, but with certain punctuation in terms we see Google treat the case completely differently. Take the search terms facebook.com and facebook com. Google reports that both of these terms are searched 7.8 million times a month. Clearly these two variants are not searched an identical number of times; Google has simply mapped the keywords together BOTH in organic search results AND in volume. This forces keyword researchers to build huge keyword lists and go line-by-line removing the edge cases.

Here's a quick tip for you Excel experts out there: Look into using Jaro Winkler distance to find very similar terms that have identical search volume. Often these terms are mapped both in organic and in volume, and you can find those exclusions easily.

Dirty secret #5: Strange recommendations

Sometimes Google Keyword Planner gets the keyword recommendations completely wrong. Here are a couple of the examples that I was able to pull in just a few minutes of brainstorming:

Starting Keyword	Recommended Keyword
baseball glove	boxing glove
pigeon	cabins
calamari	pork chops
rap	country music

Because Google Keyword Planner uses more than just phrase matching to build their recommended keywords, you will regularly find some truly strange entries in your recommended keyword list, or connections that a computer might make but a human never would. Unfortunately, this means you have to be very careful about what you get back, going keyword by keyword if you want to start a paid search campaign based on what's been returned. You simply can't be confident in the relevancy of the results. Can you imagine how many webmasters just blindly added Google's recommendations to their advertising campaigns?

All is not lost

Luckily, there is more than one way to get at and improve the Keyword Planner data using clickstream data sources. For example, we know of two keyword data sources — ClickStre.am and SimilarWeb — which correlate nicely with Google Keyword Planner volumes.

While this data from SimilarWeb is very useful, building a more accurate prediction of search volume for a term requires that you build a regression model comparing the user data to Google's estimates. Moreover, demographic differences between the whole Google user base and those included in the user panels of SimilarWeb and ClickStre.am mean that building a ubiquitous regression model across all the keyword data might not be the best, as the users tracked by SimilarWeb and ClickStre.am might be biased towards different topics. The solution is to build models around topically-related keywords.

For example, instead of modeling all the keywords against one another, if Google Keyword Planner gave you 2 keywords on the same topic with the same keyword bucket (like 201,000 searches per month), you could build a regression model on the fly comparing a sample of topically-related keywords, using that to predict with greater granularity the performance of the two seemingly identical keywords.

While this user data helps you defeat issues of granularity, getting better (both more thorough and more accurate) recommendations for keywords can be a little more difficult. Your best bet here is to use keyword data aggregators like GREPWords, KeywordTool.io, or the upcoming Moz Keyword Explorer.

Keyword Planner is dead. Long live Keyword Planner

Unfortunately, despite all of the strange quirks and outright deceptions of Google Keyword Planner, it's the best thing we really have going for us in terms of getting search volume data out of Google. We can potentially refine some of the data with clickstream data, or get estimates by running Google Adwords campaigns and watching impression counts, or even looking in Google Search Console. But none of these are strong replacements for the Google Keyword Planner.

Instead of letting Google Keyword Planner's problems get in the way of your keyword research, use it to your advantage. Look for the edge cases where a keyword has a ton of misspellings mapped to the correct version, but not combined into the volume score. This could be a great win that your competitors are overlooking because the head term looks smaller than it really is. Wherever there's bad data, there's also money to be made in sweating the details. So, put your gloves on and get to scrubbing your Keyword Planner data. Somewhere beneath the rough is a diamond.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!