The internet is full of advice to use Google's keyword tool to work out what search terms to optimise for. Here's why you shouldn't do this - and what you can use instead.

On the face of it, Google's tool looks promising. You put in a keyword or search term you're interested in (perhaps because you want to know what words to use in a headline or what tags or sub-category names to use in your navigation). You get a list of related keywords. And you can see how many searches there are for each one. You then know which are the most popular - and can target those.

Except you shouldn't do this. Here's why.

Karen Gillan

Let's start with Karen Gillan, who plays Amy Pond in Doctor Who.

Logged out

If you use the keyword tool when you're logged out, you see this - and a warning to "Sign in with your AdWords login info to see the full set of ideas for this search" (the tool is free - but you have to register and log in to see all the data):

Karen Gillan logged out

(All these screenshots are based on google.co.uk searches (though see the warning at the end of this post) with a phrase match type - ie any searches which includes the words shown in the quote marks in that order.)

Logged in

Log in, and you see this:

Karen Gillan logged in

Annoyingly with a phrase search, the straightforward term you entered ("Karen Gillan" here) isn't showing.

But more importantly, as you can see by comparing the two screenshots, the logged out version of the tool didn't show you the most popular search term with the phrase "Karen Gillan" in - namely "Karen Gillan underwear" (nice).

Logged out again ...

You can force the logged-out tool to show you the result for "karen gillan underwear" by searching for that specific term:

Karen Gillan underwear - logged out

Comparing logged in and logged out

Now, fair enough, there was a warning to log in to see all the relevant keywords. But if you've been paying attention, you'll have noticed something a bit odd about the search volumes

Logged in:

• Karen Gillan underwear - 1,900
• Karen Gillan pictures - 590

Logged out:

• Karen Gillan underwear - 9,900
• Karen Gillan pictures - 3,600

The numbers aren't the same. But I suppose they're at least out by about six times each, so the same order of magnitude ...

Only, that's not always the case.

Jennifer Aniston

Let's look at Jennifer Aniston.

Logged out

If you're logged out, you see these results:

Jennifer Aniston logged out

Logged in

If you log in, you see these results instead:

Jennifer Aniston logged in

You should notice two things. First, "Jennifer Aniston pregnant" is at number two when you're logged in, with 18,100 searches a month. This search term didn't show when we were logged out, but fair enough, there was that warning to log in to see the full set of results.

But, secondly, the numbers don't match again. When logged out, "Jennifer Aniston's hair" was shown as having 60,500 searches a month. When logged in, there are only apparently 12,100 searches a month.

Logged out again

Again, you can make the missing search terms appear when you're logged out if you know what they are by typing them in the box. If we do that, we get this for "Jennifer Aniston pregnant":

Jennifer Aniston pregnant - logged out

Comparing logged in and logged out

To sum up, we're looking at these numbers:

Logged in

• Jennifer Aniston pregnant - 18,100
• Jennifer Aniston hair - 12,100

Logged out

• Jennifer Aniston pregnant - 49,500
• Jennifer Aniston hair - 60,500

It's not just that the numbers are different, they are in a different order. When logged in, the tool says people search for pregnant 1.5 times as much as hair. When logged out, hair is 20% more popular than pregnant.

What Google News thinks

To make matters worse, if you go to Google News and start typing Jennifer Aniston's name, you see this in the Autosuggest feature - which is supposed to show you what people are searching for right now:

Jennifer Aniston - Google News autocomplete

There's no mention of hair, and "Jennifer Aniston and Brad Pitt" is more popular than pregnant - even though when logged in, the keywords tool told us that people were twice as likely to search for her pregnant (18,100 searches) as with Brad Pitt (9,900 searches).

Katie Price

Here's a similar story.

Logged in

This is what you see when you search on Katie Price's name logged in:

Katie Price logged in

Logged out

And here are the logged out figures for two search terms.

Katie Price and Peter Andre:

Katie Price and Peter Andre - logged out

Katie Price video:

Katie Price video - logged out

Comparing logged in and logged out

Compare the figures again:

Logged in - the same:

• Katie Price and Peter Andre - 14,800
• Katie Price video - 14,800

Logged out - one is 50% higher than the other

• Katie Price and Peter Andre - 40,500
• Katie Price video - 27,100

When you're logged in, the tool says the search volumes are the same. When you're logged out, it says there's 50% more searches for "Katie Price and Peter Andre" than there are for Katie Price video.

What Google News thinks

Go to Google News, and you can see that the current second most popular Google Autocomplete is Katie Price pregnant - behind "Katie Price and Peter Andre" and with Katie Price video nowhere to be seen.

Katie Price Autocomplete in Google News

When you're logged in, though, the number of searches for Katie Price pregnant is negligible:

Katie Price pregnant - logged in

Which means that the term people are searching for right now, according to Google News, isn't even worth looking at according to the keyword tool.

What all this means

You can't trust the logged-out version of the tool, as it doesn't show you everything.

The search volumes you see for a specific search term are different when logged in or logged out. I've seen a supposition that, when logged out, you see total search volumes but when logged in you see searches that triggered an Adwords ad. This would make the logged-in version of the tool useless for SEO, as what it's showing you is determined by PPC budgets (when the money's spend, there are no more Adwords ads, so those searches are ignored). Alternatively, it's just a bug as suggested here. Either way, the data is screwy.

However, you shouldn't trust this data anyway, even if it was unscrewy, as the Adwords tool doesn't show search volumes. It shows "the approximate 12-month average of user queries for the keyword on Google.co.uk and the Google Search Network". This basically means it's counting EG adwords panels on parked domains as "searches" - and inflating the search volumes.

Even if you did trust it, for the search terms above, which do tend to be fairly newsy, I'll give you, the Adwords keywords tool isn't reflecting what people are searching for now.

To sum up - the data's odd, inconsistent, and out of date. Don't use it.

Well, what shall I use?

To decide what to optimise pages for, I tend to use a combination of Google Insights, Google Autocomplete for web searches and Google Autocomplete for News. The latter have their issues (such as these).

But by plugging what you see from the Autocomplete data into Google Insights, I think you get a much better picture of what people are really searching for. If you're up to APIs and stuff, rather than copying what you see, you can automate the autocomplete discovery. Or you can use this awesome keyword discovery tool that does the same thing (hat-tip to @rishil).

The Sun decided last week to run a story about the false rumours circulating about Steven Gerrard:

SEX slurs claiming England's World Cup hopes were undermined by a player's fling with a teenage girl are a sick HOAX, The Sun can reveal.

Thousands of fans have received texts and emails saying captain Steven Gerrard had got wife Alex Curran's 16-year-old sister pregnant.

But the lies are blown apart by one crucial fact - model Alex, 27, does not even HAVE a sister.

(I love the justification here for running a story about a rumour: people are receiving texts and emails. Makes it sound link some sort of conspiracy as opposed to people just emailing their friends ...)

Anyway, I don't know what Gerrard's lawyers made of this story but they might want to have a word with Google. If you get as far as typing Steven Gerrard into Google News, the auto complete function throws up this list ...

Google's autocomplete for "Steven Gerrard"

Unlike the Sun's story, the list isn't peppered with provisos that none of this is true.

More evidence that Google has changed how it handles spelling errors. It is now fixing spelling in autocomplete (the list of suggestions it offers as you start to type in the web-based search form). (I think this is new, anyway ...)

Unlike yesterday's example, where Google was just deciding for itself which version of alternative, correct spellings you meant, I approve of this - partly because it's helpful, and partly because the user retains control (so they can search for wrong spellings if they want to).

Here are some examples - plus some where it ought to be fixing them but isn't. Note in each that the list of suggestions include the correct spellings (apart from millennium), even though I have spelled the word wrong.

Correcting common mis-spellings

Weird vs wierd

You type wierd, Google shows suggestions for weird

Acomodate vs accommodate

You type acomodate, Google shows suggestions for accommodate

Acceptable vs acceptible

You type acceptible, Google shows suggestions for acceptable

Greatful vs grateful

You type greatful, Google shows suggestions for grateful

Bellwether vs bellweather

You type bellweather, Google shows suggestions for bellwether

Please get these right however

Could of vs could have

This is not correct English - it should be could have.

There is no such construction as "could of". Please correct to could have.

Millennium - huh?

And millennium has two ls and two ns, however many results you've found for the wrong spelling ...

Right, I think it just can't spell millennium.

Conclusion

Looking at these, and the various other spelling examples that have emerged lately, it seems that Google must be using user behaviour in some way to deal with spelling errors.

For instance, where it observes people search for X immediately after Y (and X is a spelling variant of Y), it might decide to return X results when people search for Y. This probably explains the stationary / stationery issue from yesterday and the weird / wierd etc examples above.

Alternatively, it might be using its "did you mean" spelling correction data and, where people clicked this a lot, it's now just assuming they meant it all along.

I'm not sure it explains Google returning results for optimization when you search for optimisation, but there you go.

What do you think?

There are several ways to use Google to find out what people are searching for.

Having used three ways to check search terms related to The X Factor (ITV's talent show) - I reckon Google Insights is better than the Google Adwords Keywords tool for topical insight (and don't bother with the auto suggest feature).

Google Insights: four yeses

Google Insights lets you "compare search volume patterns across specific regions, categories, time frames and properties".

Here's what it tells us about The X Factor (the options were set to web search, UK, last 7 days). Unsurprisingly, X factor 2009 is top - followed by searches for youtube, auditions and ITV. Jamie, a contestant from a week ago tipped as a possible winner is in the top 10.

Google Insights: Search terms for The X Factor

As well as top search terms, you can also rising searches - those that have become particular popular lately, as this next table shows.

Google Insights: Rising searches related to The X Factor

The rising searches show that some of the contestants from saturday night, particularly the stunners (who weren't, and didn't make it through) are already being searched for.

Data like this on a weekly basis might be a clue as to the eventual finalists? Either way, it's all useful and interesting data for anyone doing SEO.

Google Adwords keyword tool: three yeses

The Google Adwords keyword tool lets you see global searches related to the term you choose or local searches for the most recent month - August in this case.

Google Adwords keyword tool: The X Factor results

I'm not sure these results (for a phrase match) look particularly accurate - were people really searching for the x factor 2007 and 2006 in august of 2009? 'X factor 2009' doesn't appear in the Adwords keywords tool list?

Also, they don't seem bear any relation to what Google Insights shows (I haven't done a screenshot) if you set a date range of August this year - where "X factor 2009" is shown as the top search term. This sounds much more likely.

The Google Insights results for August also show 'X factor Danyl' - the contestant from the first programme in August who has has 5 million people watch his debut on youtube - in the top 10, whereas the Adwords keywords result don't.

Google Auto Suggest: it's a no from me

Finally, there's the Google Auto Suggest function on the homepage.

Here's what it showed - I think these are so general as to be pretty useless for analysis of what people might be searching for.

Google Autosuggest and The X Factor

Conclusion

For topical subjects, Google Insights seems to give the best, er, insight into what people are searching for - its stats seem more convincing than the Adwords keywords tool, and give a better sense of current trends.

Google Insights: you're hired, Oh, wrong show.

There's a funny post on Google's autosuggest feature (where, when you start typing, Google suggests searches) on SEO Services group. I gave it a go with some less SEO terms. My god it's rude ...

Gordon Brown: sh*t, a moron, useless, incompetent

This is the prime minister. Shouldn't they check these?

This is what Google thinks of our Prime Minister

David Cameron: a sexy idiot, apparently

He gets a better press.

Google Autosuggest for David Cameron

Prince Charles: antichrist

I think he's a nutjob, waste of money and constitutional abhorrence. Apparently Google thinks he's the antichrist.

Google Autosuggest for Prince Charles

Jade Goody: is she dead, or not?

Google seems a bit conflicted.

Google Autosuggest for Jade Goody

Micheal Jackson: dead, back, freak and going to die ...

Yup, he's dead, back, a muslim, dying, a freak, innocent, ill, Latoya and going to die. Hmmm.

Google Autosuggest for Michael Jackson