Fast forward to September and the story is the same as earlier in the year - Guardian 1st, Telegraph 2nd and Mail 3rd. So what changed from June to September? To find out, I've compared the ABCe figures for UK and foreign visitors in June and September. The difference between the Guardian's performance and that of the Telegraph and Mail is revealing.

Guardian: strong growth here and abroad

Table: September unique visitors (millions) and percentage change since June

Analysis

The Guardian has seen significant growth in the UK AND abroad.

Its total visitor numbers grew 14% from June to September (up from 29m to 33m). There was a 17% increase in UK visitors and a 12% increase in visitors from abroad. This makes it the most popular online newspaper in the UK by some way (it's 2.4m ahead of the Mail in second place).

UK visitors accounted for 36% of the total in September (barely changed from 35% in June).

Telegraph: overseas growth only

Table: September unique visitors, percentage change since June

Analysis

The Telegraph has also seen a 14% increase in total visitors from June (27.2m) to September (31m).

However, the geographical breakdown is revealing - its UK unique visitor numbers are down 1% from June to August but its overseas visitors are up 22% (from 18m to 21.9m). It's now the most visited UK newspaper abroad - but only the 3rd most visited inside the UK.

As a result, the proportion of its visitors that comes from the UK has fallen from 34% to 29% - the lowest of any UK newspaper (the Mail held this honour back in June).

The Telegraph saw the biggest increase in overseas visitors of any newspaper - but because its UK traffic fell, the Guardian beat it into 2nd place.

Mail: UK growth only

Table: September unique visitors, percentage change since June

Analysis

The Mail's stood fairly still between June and September - it had 30m visitors last month, up just 2% on three months ago. But its story is the reverse of the Telegraph's.

The Mail saw strong UK growth - up 14% to 9.5m visitors in three months. Overseas visitors, however, fell by 2% to 20.6m. As a result, it now gets 32% of its visitors from the UK (up from 28% in June).

It got overtaken by the Guardian because it hasn't been able to match and sustain its overseas growth.

And the rest ...

As for the others:

• The Sun is down to 23m visitors in September, an 8% fall over 3 months. A 15% collapse in overseas visitors couldn't make up for a 3% increase in UK users.
• The Times is a story of decline - 13% down overall, with a 10% fall in the UK and a 14% fall from overseas.
• The same is true of the Mirror (down 5% overall) and the Independent (down 6% overall) but to a lesser extent.

This table has all the stats. If you can't see the iframe, you can see the full spreadsheet here.

The Express doesn't take part in the ABCes. The FT does only some months.

The BBC sends about 870,000 visitors a month to the Times website and 1.1 million to The Sun's, according to Alexa.com. It sends similar numbers to Sky's websites.

Percentage of traffic to News Corp's sites from the BBC

Alexa measures visitors' 'preceding sites' - the site they were on before they visit a particular site, and which is likely to have sent them there via a link.

The Alexa figures show how important the BBC is in sending traffic to the Times, Sun etc.

• Times Online: 4.1% of visits were preceded by bbc.co.uk - making the BBC the 3rd most important preceding site after google.co.uk (12.9%) and google.com (12.2%). (See the Alexa data - click clickstream).
• The Sun: 4.35% of visits are preceded by the BBC's site. This makes the BBC the 4th most important preceding site - after google.co.uk (9.5%), google.com (9.2%) and facebook (5.3%).(See Alexa data.)
• Sky.com: 4.0% of visits are preceded by bbc.co.uk, making the BBC the 5th biggest preceding site - after google.co.uk (14.9%), facebook.com (7.8%), google.com (7.7%) and yahoo.com (4.6%). (See Alexa data).
• Skysports.com: 6.6% of visits are preceded by bbc.co.uk, making the BBC the 3rd most important preceding site - after google.co.uk (10.0%) and facebook.com (9.0%). (See Alexa data).

(Reading by RSS? Click here to see the video version.)

Methodology

The figures are based on Alexa's clickstream data, which does have a slight American bias as my post on the accuracy of Alexa data found.

The Alexa data measures preceding sites - the order in which sites are visited. It doesn't measure how users got from one site to another. This is most likely to be because a user clicked a link - in which case the BBC is benefiting the next site to be visited. It could also be because they, say, clicked an existing bookmark, in which case the BBC is not responsible for that visit.

The Alexa figures are for visits, whereas the ABCe numbers are for visitors. However, I multiplyied the Alexa %s by the ABCe numbers to come up with the actual visitor numbers for the Times and Sun.

So, to find out, I've compared Alexa with the ABCe official audited data for UK newspaper sites - using the figure for the %age of each site's visitors from the UK.

As the table shows, Alexa is good but not brilliant.

Alexa vs ABCe

In particular, Alexa consistently underestimates the proportion of users who are from the UK (maybe reflecting its American roots?). However, the Mirror apart, the spread of errors is reasonably consistent.

Using the table:

• Clicking the first column takes you to the relevant Alexa page.
• The Alexa column shows the percentage of users Alexa thinks come from the UK. They don't give the timeframe.
• For the ABCe column, I dividing the June ABCe figures for total unique UK users into the overall total unique users. The FT isn't shown as it isn't audited every month (and wasn't in June). The Express's site isn't audited. The ABCe do some aggregating of data - for instance, the Sun figure includes the News of the World site. Alexa treats this separately.
• The average is a simple average - it isn't weighted in any way.

Using Alexa data

I'm going to use Alexa to make more comparisons about UK newspapers over the next few days.

You can read more about interpreting Alexa data on Alexa's own explanatory page - in particular note that they no longer just rely on people who have downloaded their toolbar, as they did in the past.

Some other information about the data (but much of it written when they did still rely on ONLY the toolbar data) includes:

• Impact of Museum Websites - a comparison using Alexa (see the comments)
• Please stop quoting Alexa data (although see the last comment where the author says things have improved).
• Alexa toolbar and the problems of experiment design.
• And Mark Pack's recent post (March 2010) on how the internet is changing British politics (well worth a read).

However it was only 4th in terms of UK visitors. Figures from Compete.com, which tracks Americans' internet use, show that, of the 4.7 million unique users the Mail added from May to June, 1.2 million were from the USA. American and other foreign visitors searching for Michael Jackson's kids - the Mail tops google.com for a search on this - drove this overseas growth.

US traffic to UK newspaper sites

Of the big three UK newspaper sites this is what happened to their US traffic from May to June:

• Guardian: up from 3.4m to 3.7m - a rise of 300,000 or 9%.
• Telegraph: up from 3.7m to 4.1m, a rise of 500,000 or 11%.
• Daily Mail: up from 4.0m to 5.2m, a rise of 1,200,000 or 30%.

This dramatic increase in traffic, compared to its rivals, from May to June helps explains how the Mail leapfrogged the Guardian and Telegraph.

The Mail's US traffic leapt from May to June

Google.com was the main referrer to the Mail - responsible for 22.7% of its traffic. More on this below. Next up was drudgereport.com (a large US news aggregation site), followed by Yahoo.com and Facebook.com.

What was behind this rise in US traffic?

So what led to this sudden increase for the Mail? Compete also shows you the main search terms that lead US visitors to sites.

Top 5 search terms that lead US visitors to the Guardian

• Guardian / the guardian: 2.6%
• Michael Jackson: 0.9%
• Swine flu symptoms: 0.6%
• Susan Boyle: 0.6%

Top 5 search terms that lead US visitors to the Telegraph

• Michael Jackson: 2.5%
• Susan Boyle: 0.8%
• Swine flu symptoms: 0.7%
• Daily Telegraph: 0.6%
• Michael Jackson children: 0.5%

Top 5 search terms that lead US visitors to the Daily Mail

• Daily Mail / Dailymail: 9.9%
• Michael Jackson (or Jackson's) children: 2.9%
• Michael Jackson's kids: 1.3%

What does this tell us?

The main keywords driving US search traffic to the Mail

The Guardian's top 5 search terms accounted for just 4.7% of its search traffic. The Telegraph's top 5 for 5.1%.

But the Mail's top 5 accounted for a massive 14.1% - split between searches for its brand name and for Michael Jackson's kids (and outside the top 5 there may have been many other MJ-related terms).

Its search traffic in June is heavily skewed to these two search terms in the USA - and elsewhere in the world, I think it's reasonable to presume.

Can this last?

Searches in the USA for 'Daily Mail' have been fairly consistent over the last few months according to Google Insights. I don't know why so many people do this compared to other newspapers.

But I do know that interest in Michael Jackson's kids is going to die down. This graph shows how there was a huge and sudden surge in searches for searches on his children and kids after he died. The graph shows just two search terms - there are likely to be many others, and so a significant proportion of the Mail's overseas traffic increase is down to search terms related to Jackson's offspring.

Searches for Michael Jackson and kids/children shot up

This increase in searches translates into traffic for the Mail because it is TOP for a search on 'Michael Jackson children' at google.com and 3rd for kids (it's also top in Google India for a search on his children, and India is the next most common source of traffic to the Mail after the UK and USA).

So all this data suggests that the Mail's top spot in June's ABCes is built on US and other worldwide search traffic around Jackson's children - the massive peak in late June and again around his funeral in early July.

Once people stop searching for these terms, this traffic will disappear. The Mail may still top July's ABCes on the back of this traffic - but it's hard to believe it will still be top in August.

Methodology

You can, of course, pick holes in this argument.

The 3 MJ's kids search terms account for 4.2% of Google traffic, which accounts for 22.7% of 5.2m visitors - so about 50,000 users.

But I think it's reasonable to assume that there are more search terms outside the top 5, there are other search engines, and that the other sources of traffic, such as people sharing links on Facebook and news aggregators, will also partially be about Jackson's children. Plus this is the only publicly available data that I'm aware of, and this is the story it seems to be telling.

The problem - the third result

ABCe: not found

The solution

The usual thing to do with known links that no longer work is to 301 redirect them to a different page (EG when you know a URL has moved). For all other URLs that don't work (EG when someone mistypes one), you return a 404 code.

These status codes help Google, for instance, work out what is going on. If it sees a 404 code it knows there's nothing there so it needn't return the page in search results.

I've written before about the awful abce site and the terrible abce redesign. It appears when they did the latest redesign, they decided to 301 redirect any broken links to this page: http://www.abce.co.uk/404.aspx.

However, that page, even though it it called 404.aspx and has a 'page not found' title, returns a 200 code - telling Google that it is a genuine URL.

The end result is that all links to old ABCe URLs are redirected to the new 404.aspx page. The 301 tells Google to count links to the old pages as if they belonged to the new 404.aspx page. Because this is a lot of links and a lot of pages, Google has ended up thinking the 404.aspx is the most important page on the abce site.

All of which means that if you type abce into google, you get the identical abc site first, and then you get the 404 page (with a 200 code) for the abce site. Obviously, this looks completely stupid.

If at any point the ABC wants to pay me a consultancy fee, for all this free advice, just leave me a comment to tell me how to receive my money ...

All the URLs have changed but there are no redirects

Search results for ABCe after redesign

They've had a redesign, but they haven't redirected the old URLs to new ones. So, for instance, if you click the second link shown in Google for a search on ABCe, you get page not found.

Lesson When relaunching a website, always 301 redirect your old pages to new ones (even if they're all just to your new home page). That way, external links still work and you keep the SEO benefit of any links.

They haven't sorted www vs non www

The more observant will have noticed that the title of the first result in that screenshot says 'To access IIS Help'. The ABC hasn't realised that abce.org.uk is not the same URL as www.abce.org.uk. And if you go to the ABC URLs without www, you get page not found or server errors.

Compare these pages:

• www.abc.org.uk
• abc.org.uk

and these ones:

• www.abce.org.uk
• abce.org.uk

Lesson When you set up your website, redirect yourdomain.co.uk/whatever to www.yourdomain.co.uk/whatever. And log in to your google webmaster account to set your preferred domain (www or non-www).

They're running two absolutely identical websites

The new ABC homepage. And also the new ABCe homepage.

You can access the entire website at www.abc.org.uk - or you can see an identical website at www.abce.org.uk.

The only difference for each and every page across the two sites is whether there is an e in the domain name or not. For instance, compare these two URLs:

• www.abc.org.uk/Corporate/AboutABCe/StandardsandMetrics.aspx
• www.abce.org.uk/Corporate/AboutABCe/StandardsandMetrics.aspx

The same is true for every URL. This will cause problems with duplicate content in search engines, plus splitting incoming links over two URLs is problematic for SEO.

They do have two brands - ABC for print, ABCe for online - but they don't need two websites. And even if they did, they wouldn't need two identical ones.

Lesson Have a website. Don't have another identical one.

The navigation is inconsistent

If you go, say, here, some related links appear in the right:

• If you click on the first one, JICwebs, then a jargon buster sudenly appears in the left hand menu, under the links that are for subscribers only.
• If you click on the second one - the jargon buster - you get page not found.
• If you click the others, some random stuff happens.

As far as I can work out, the top navigation bar controls the right hand related links box which in turn controls the left-hand menu.

Some pages have a right hand menu with related links, some are full-width with no right hand menu, and some have an empty right hand menu.

Lesson Make your navigation consistent. And keep things that are together, together. Having the top, left and right navigations interdependant AND inconsistent is really confusing. And use templates consistently.

Some minor points

• Login not clear To get to most pages, you need to login. It's not clear which pages require this. Although on this page it says "only members can access the following pages" and you can click on the links, so clearly they're confused at ABCe too ...
• Titles unclear Although the titles have improved since the last site, they still aren't ideal. For a start, capitalisation is random. And secondly, they don't show the site structure. Instead of going 'Page name - Section - ABCe', they just show the page name. This looks unclear in Google.
• Descriptions poor The descriptions aren't proper English. This isn't gramatically correct: "How to become a subscriber to ABCe and what are the benefits of doing so."
• Too many link styles Some links are blue italic. Some are black unerlined. Some are red not underlined. Some are grey. Make your mind up ...
• Mising images There's a contact list here, but all the photos are missing.

I could go on - there's more. But I won't.

Yet despite these basic problems, they're using SIFR to show fancy fonts. I think they should concentrate on these other things ...

The ABCe navigation frame -Â reached via one of the google sitelinks. None of the links on it work.

For a body in charge of standards, its own website is a disgrace. It's so bad, it's embarrassing. And hard to know where to start ... But here we go:

Stop using frames

Using frames is generally a bad idea. ABCe's use of frames causes the following problems:

• 

  ABCe sitelinks in google

  It confuses search engines - 1. The use of frames and poor information architecture of the site has confused google. The ABCe site is sufficiently 'important' in google's eyes that it has been given sitelinks when you search for ABCe (the extra layer of navigation, providing one-click access to bits of the site). One of these is for something called 'Navigation frame'. Clicking on this takes you to the blue stripy image, above. Clicking on the links on the stripy page does nothing. Note to ABCe webmaster: you can fix this with a google webmaster account.

• 

  ABCe - the blank page Google sees

  It confuses search engines - 2. If you look at the google cache of the ABCe homepage, it's empty. The use of frames and no content means that google sees an empty page. This screenshot illustrates what google sees - a completely empty space.

• It isn't very accessible to people who are using screen readers. You can make accessible frames, by following advice like this. ABCe fails to follow these guidelines. It uses confusing frame names (like lfsmenu), it doesn't declare an HTML document type, it doesn't use noframe tags for those can't see the frames etc. ABCe is sticking two fingers up at people with disabilities.
• Your URLs don't work. When you copy and paste a URL, it doesn't work. This make it impossible to bookmark pages or email them to a friend. This is really annoying. Stop using frames, it will fix it.

It looks terrible in Google

I've already covered the site links issue. On top of that, the site has the following problems, all of which can be easily fixed:

ABCe titles: all the same

• Loads of pages have the same title. Search for something ABCe-related in Google, and you're likely to get back a page with the title 'ABCe'. It's easy to show meaningful titles in Google - you just use the HTML title tag. Doing so will help people and Google understand what your page is about.
• Its description is incomprehensible. What on earth does "ABCe is the industry owned, tri-partite, not-for-profit organisation that works with and for media owners, advertisers and media buyers to help them better ..." mean to anyone? Help them better what? You can set your own description in Google using 'meta description'. It's easy to do.
• Its URLs are gibberish. How is anyone supposed to understand what on earth "http://www.abce.org.uk/cgi-bin/gen5?runprog=abce/abce&type=page&p=britinfo-case-study.html&menuid=about_abce|case_studies|britinfo-case-study" is about? Try to use so-called search-engine-friendly URLs so that humans can understand them.

Spelling, grammar and readability are appalling

It's an industry body, so I guess we can forgive it a fair amount of jargon. But it's a publisher's industry body ...

• 

  ABCe: FAQs doesn't have an apostrophe. Be consistent.

  FAQs either has an apostrophe or it doesn't. And it doesn't. Try not to use FAQs and FAQ's next to each other.

• Sentences need to be sentences. This doesn't actually make sense: "In addition, we offer marketing support, including publicity, stating that the subscriber and ABCe are working together, we also deliver newsletters, seminars and website subscriber listings."
• Try to write in a way that people understand. I ran the page (I can't link to it due to the URL problem, above) on the "Associate Subscriber Scheme" through a readabilty test. On the Flesch-Kincaid Reading Ease scale, it scored 26.7. This make it harder to understand than the Harvard Law Review according to Wikipedia. So instead of this: "participate in the ABCe technical group (ITG) meetings which are designed to ensure all clients and their agents may participate in developing and evolving the industry agreed rules and methodologies" ... why not say "take part in the group meetings that agree the industry's standards"? 11 words rather than 29.
• Sentences end in full stops. That's it really. They just do.
• Try not to randomly capitalise stuff. 'List of Clients' and 'Help and Links', but 'Where are we' and 'Case studies'. Be consistent.

It's impossible to find anything

The whole site is just impossible to use.

• 

  The ABCe search box - with meaningless label

  Labelling is poor. The search box says 'Data Search by URL' on it. What on earth does that mean?

• The 'ABout ABCe' page doesn't tell you anything useful.
• There it too much navigation and not enough content. Want to find their offices? Click 'About', 'Where are we?' and then 'Directions'. Couldn't you have given this info a bit earlier, given most of those pages are virtually empty? The 'Where are we' page has just 3 words on it! Couldn't the directions have gone there ...?
• There is an invisible 'Ask us a question' link. It's a big floating question mark in the left hand menu, and it took me about 50 pages before I accidentally moused over it and realised it did something. Try to label things so people know what they are ...
• Keep the site up to date. It's March. Maybe show the 2009 fees, rather than the 2008 ones?
• You published some data on regional newspaper site usage. I can't find it. Anywhere. Where is it?

• 

  ABCe search form - incomprehensible

  I don't understand the search form at all. What is a Class? What does Publisher mean in this context? What is the difference between 'Site/Company URL' and 'Site/Product name'. What is a product? Why company URL? Wouldn't the site's own URL be better? What does 'Site' mean at the front? How about some help text?