Internet
How to
Newspapers
Twitter
SEO advice
Categories: How to, Internet, Newspapers. Tags: , ,

Sun blocks NewsNow from crawling its site

January 12, 2010 2 Comments

NewsNow revealed this morning that it's heard that News International will soon be blocking NewsNow from crawling The Sun's site - just as it has done with the Times.

This has now happened, as the Sun's robots.txt file shows:

The Sun's robots.txt file disallows NewsNow's crawler from indexing The Sun's content

The Sun's robots.txt file disallows NewsNow's crawler from indexing The Sun's content

The same is true of the News of the World site.

You can see NewsNow's views on the situation in comments on my previous posts here and here.

If you are going to report on this, please don't confuse linking and crawling (see previous posts) ...

You might also like
  1. NewsNow vs the Times: Right to crawl vs right to link
  2. NewsNow vs the Times: I’ve decided I’m not sympathetic
  3. Whisky hidden in robots.txt file
  4. Want an SEO job? Check out the Daily Mail’s robots.txt file …
  5. Ow.ly and Hootsuite in widespread breach of newspaper / other site T&Cs

Share this post

Follow me on Facebook or Twitter

2 Comments »

Leave a comment!

Add your comment below, or trackback from your own site. You can also subscribe to these comments via RSS.

Be nice. Keep it clean. Stay on topic. No spam.

You can use these tags:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

This is a Gravatar-enabled weblog. To get your own globally-recognized-avatar, please register at Gravatar.