Categories: How to, Internet, Newspapers. Tags: NewsNow, seo, Sun
Sun blocks NewsNow from crawling its site
NewsNow revealed this morning that it's heard that News International will soon be blocking NewsNow from crawling The Sun's site - just as it has done with the Times.
This has now happened, as the Sun's robots.txt file shows:
The Sun's robots.txt file disallows NewsNow's crawler from indexing The Sun's content
The same is true of the News of the World site.
You can see NewsNow's views on the situation in comments on my previous posts here and here.
If you are going to report on this, please don't confuse linking and crawling (see previous posts) ...
You might also like
- NewsNow vs the Times: Right to crawl vs right to link
- NewsNow vs the Times: I’ve decided I’m not sympathetic
- Whisky hidden in robots.txt file
- Want an SEO job? Check out the Daily Mail’s robots.txt file …
- Ow.ly and Hootsuite in widespread breach of newspaper / other site T&Cs