Are Keywords Really Dead? An Experiment

Posted by sam.nemzer

[Estimated read time: 6 minutes]

A quantitative analysis of the claim that topics are more important than keywords.

What’s more important: topics or keywords? This has been a major discussion point in SEO recently, nowhere more so than here on the Moz blog. Rand has given two Whiteboard Fridays in the last two months, and Moz’s new Related Topics feature in Moz Pro aims to help you to optimize your site for topics as well as keywords.

The idea under discussion is that, since the Hummingbird algorithm update in 2013, Google is getting really good at understanding natural language. So much so, in fact, that it's now able to identify similar terms, making it less important to worry about minor changes in the wording of your content in order to target specific keyword phrases. People are arguing that it’s more important to think about the concepts that Google will interpret, regardless of word choice.

While I agree that this is the direction that we're heading, I wanted to see how true this is now, in the present. So I designed an experiment.

The experiment

The question I wanted to answer was: “Do searches within the same topic (but with different keyword phrases) give the same result?” To this end, I put together 10 groups of 10 keywords each, with each group’s keywords signifying (as closely as possible) the same concept. These keywords were selected in order to represent a range of search volume, and across the spectrum of informational to transactional. For example, one group of keywords are all synonymous the phrase "cheapest flight times" (not-so-subtly lifted from Rand’s Whiteboard Friday):

• cheapest flight times
• cheapest time for flights
• cheapest times to fly
• cheap times for flights
• cheap times to fly
• fly at cheap times
• time of cheapest flights
• what time of day are flights cheapest
• what time of day to fly cheaply
• when are flights cheapest

I put the sample of 100 keywords through a rank-tracking tool, and extracted the top ten organic results for each keyword.

Then, for each keyword group, I measured two things.

1. The similarity of each topic’s SERPs, by position.
   • For example, if every keyword within a group has the same page ranking no. 2, that result will score 10. If 9 results are the same and one is different, nine results will get a score of 9, and the other will score 1.
   • This score is then averaged across all 100 (10 results * 10 keywords) results within each topic. The highest possible score (every SERP identical) is 10, the lowest possible (every result different) is 1.
2. The similarity of each topic’s SERPs, by all pages that rank (irrespective of position).
   • As above, but scoring each keyword’s results by the number of other keywords that contain that result anywhere in the top 10 results. If a result appears in the top 10 for all keywords in a topic group, it scores a 10, even if the results in the other keywords’ SERPs are in different positions.
   • Again, the score is averaged across all results in each topic, with 10 being the highest possible and 1 the lowest.

Results

The full analysis and results can be seen in this Google Sheet.

This chart shows the results of the experiment for the 10 topic groups. The blue bars represent the by position score, averaged across each topic group, and the red bars show the average all pages score.

The most striking thing about this is the wide range of results that can be seen. Topic group D’s keywords are 100% identical if you don’t take ordering into account, whereas group J only has 38% crossover of results between keywords.

We can see from this that targeting individual keywords is definitely not a thing of the past. For most of the topic groups, the pages that rank in the top 10 have little consistency across different wordings of the same concepts. From this we can assume that the primary thing making one page rank where another does not, is matching exact keywords.

Why is there such variation?

If we look into what factors might be affecting the varying similarities between the different topic groups, we could consider the following factors:

• Searcher intent: Informational (Know) vs Transactional (Do) topics.
• Topics with high competition levels.

Searcher intent

Although Google’s categorisation of searches into do, know and go can be seen as a false trichotomy, it can still be useful as a simplistic model to classify searcher intent. All of the keyword groups I used can be classed as either informational or transactional.

If we break up our topic groups in this way, we can see the following:

As you can see, there's no clear difference between the two types. In fact the highest and lowest groups (D and J) are both transactional.

This means that we can’t say — based on this data, at least — that there’s any link between the search intent of a topic and whether you should focus on topics over keywords.

Keyword Difficulty

Another factor that could be correlated with similarity of SERPs is keyword difficulty. As measured by Moz’s keyword difficulty tool, this is a proxy for how strong the sites that rank in a SERP are, based on their Page Authority and Domain Authority.

My hypothesis here is that, for searches where there are a lot of well-established, high-DA sites ranking, there will be less variation between similar keywords. If this is the case, we would expect to see a positive correlation in the data.

This is not borne out by the data. The higher the keyword difficulty is across the keywords in a topic group, the less similarity there is between SERPs within that topic group. This correlation is fairly weak (R2=0.28), so we can’t draw any conclusions from this data.

One other factor that could explain the lack of pattern in this result is that 100 keywords in 10 groups is a fairly small sample size, and is subject to variation in the selection of keywords to go into each group. It is impossible to perfectly control how "close" in definition the keywords in each group are.

Also, it may just be the case that Google simply understands some concepts better than others. This would mean it can see some synonyms as being very closely related, whereas for others it's still perplexed by the variations, so looks for specific words within the content of each page.

Conclusion

So does this mean that we should or shouldn't ignore Rand when he tells us to forget about keywords and focus on topics? Somewhat unsatisfyingly, the answer is a strong "maybe."

While for some search topics there's a lot of variation based on the exact wording of the keywords, for others we can see that Google understands what users mean when they search and sees variations as equivalent. The key takeaway from this? Both keywords and topics are important.

You should still do keyword research. Keyword research is always going to be essential. But you should also consider the bigger picture, and as more tools that allow you to use natural language processing become available, take advantage of that to understand the overall topics you should write about, too.

It may be a useful exercise to carry out this type of analysis within your own vertical, and see how well Google can tell apart the similar keywords you want to target. You can then use this to inform how exact your targeting should be.

Let me know what you think, and if you have any questions, in the comments.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!