Sun blocks NewsNow from crawling its site

Sun blocks NewsNow from crawling its site 1

Posted on January 12, 2010 by Malcolm Coles

NewsNow revealed this morning that it's heard that News International will soon be blocking NewsNow from crawling The Sun's site - just as it has done with the Times.

This has now happened, as the Sun's robots.txt file shows:

The Sun's robots.txt file disallows NewsNow's crawler from indexing The Sun's content

The same is true of the News of the World site.

You can see NewsNow's views on the situation in comments on my previous posts here and here.

If you are going to report on this, please don't confuse linking and crawling (see previous posts) ...

All comments are moderated. Comments that include URLs and which just say 'Nice post' or similar won't be published, but thanks for the sentiment.

SD on iDosing: spot the difference between the Sun’s and the Mail’s stories:Cake, it’s a made up drug, but not as potentiall y...
Freelance Unbound on What does this Google Webmaster Tools data mean?:I saw it at the bottom of page 1. I was told by an online editor of my...
Freelance Unbound on iDosing: spot the difference between the Sun’s and the Mail’s stories:Let them eat Cake… ;
Jim Banks on What does this Google Webmaster Tools data mean?:Average position went from 7.0 to 9.9 and that might have thrown you off the cliff of...
Gareth on What does this Google Webmaster Tools data mean?:I agree with @rishil here, it’s just Google giving random nonsense data. Just like...

geschreven om te lezen on Behind the Times paywall: 46,154 readers a day
exclarotive on iDosing: spot the difference between the Sun’s and the Mail’s stories
The Times’ Paywall is Failing FirstFound Blog on Behind the Times paywall: 46,154 readers a day
Useful links from the last few days on Sites that ban you from linking to them. Still. In 2010
The Rot Within :: No Permission, No Link :: June :: 2010 on Sites that ban you from linking to them. Still. In 2010