NewsNow revealed this morning that it's heard that News International will soon be blocking NewsNow from crawling The Sun's site - just as it has done with the Times.

This has now happened, as the Sun's robots.txt file shows:

The Sun's robots.txt file disallows NewsNow's crawler from indexing The Sun's content

The same is true of the News of the World site.

You can see NewsNow's views on the situation in comments on my previous posts here and here.

If you are going to report on this, please don't confuse linking and crawling (see previous posts) ...

I wrote yesterday that I was sympathetic towards NewsNow and its ongoing row with newspapers, especially the Times.

I've now decided that I'm not.

NewsNow crawls the Times site in order to gather information for its paid service to businesses as well as its free service to consumers:

Our unique filing engine is designed to match each new breaking news article, against any of thousands of complex search specifications, in seconds. Our subscription services harness the power of our filing engine for your organisation's specific purpose.

It charges from £95 a month for tailored feeds and from £75 a month for generic (pre-built) feeds.

So it basically charges organisations to show them news stories about a given subject or keyword.

Put like that, there's no real reason why the Times should let NewsNow crawl its site, unless it pays up to the Newspaper Licensing Authority.

Update: The Sun and News of the World are now also blocking NewsNow.

Original post: I am sympathetic not sympathetic towards NewsNow over the newspapers' attempts to prevent it linking to them. The site has now been blocked from showing Times Online stories (more background here). But its campaign is a bit odd as it - and most of the media reporting the story - are confusing linking and crawling / indexing:

Newsnow's response to the Times

The right to link

Right2Link (founding sponsor NewsNow) asserts that:

We, the supporters of the right to link, declare:

1. All should be free to create, forward and follow links — they are the signposts to content on the web.
2. Linking should require no permission nor charge.
3. The free circulation of publicly accessible information is threatened if individuals, organisations and search engines cannot continue to create, forward and follow links without undue restraint.

Crawling vs linking

What the Times has done, however, is block NewsNow from crawling its site using its robots.txt file. This is nothing to do with linking to its site.

In NewsNow's case, it requires crawling to do its linking, as it's an automatic news aggregator. So it needs to crawl / index the Times site to put its own site together.

But this afternoon's piece of pedantry from me is to point out that, for the rest of us, they are not the same. And nothing the Times has done today stops anyone from linking to them - the demand of the Right2Link campaign.

So a big red box about undermining access to public information seems irrelevant.

Update: What's more, NewsNow charges for some of its services so this has even less to do with public information ...

Oh wait, we are banned from linking, my mistake

Of course, if you look through the T&Cs of the Times site, it says:

Illegal and/or unauthorized use of the Services, including ... unauthorised framing of or linking to the Website is prohibited.

It's not very clear what unauthorised linking is, however. But I'd better not link to the T&Cs just in case! (This follows on from my posts last year in which I pointed out that most major newspapers and many other organisations forbade you from linking to them. SEO anyone?).