Sun blocks NewsNow from crawling its site
January 12, 2010
2 Comments
NewsNow revealed this morning that it's heard that News International will soon be blocking NewsNow from crawling The Sun's site - just as it has done with the Times.
This has now happened, as the Sun's robots.txt file shows:
The Sun's robots.txt file disallows NewsNow's crawler from indexing The Sun's content
The same is true of the News of the World site.
You can see NewsNow's views on the situation in comments on my previous posts here and here.
If you are going to report on this, please don't confuse linking and crawling (see previous posts) ...
[...] Update: The Sun and News of the World are now also blocking NewsNow. [...]
Great articles & Nice a site?. Might want to add more graphics for this topic