Comments on: NewsNow vs the Times: Right to crawl vs right to link http://www.malcolmcoles.co.uk/blog/times-newsnow-crawling-linking/ Where to find Malcolm Coles, reviews, and tips on how to do things I couldn't do. Mon, 25 Mar 2013 16:22:31 +0000 hourly 1 http://wordpress.org/?v=3.0.4 By: Julian Burgess http://www.malcolmcoles.co.uk/blog/times-newsnow-crawling-linking/#comment-8404 Julian Burgess Tue, 09 Mar 2010 16:12:02 +0000 http://www.malcolmcoles.co.uk/blog/?p=3779#comment-8404 Glad to see an article which has got this correct. In my spare time I work on website which crawls a number of websites, I always obey the robots.txt, however it is perfectly possible to ignore it, or to move your crawler to a different IP address which isn't banned. When someone is crawling in a way which you object to for any reason then robots.txt is good way to indicate they should stop. Glad to see an article which has got this correct. In my spare time I work on website which crawls a number of websites, I always obey the robots.txt, however it is perfectly possible to ignore it, or to move your crawler to a different IP address which isn't banned. When someone is crawling in a way which you object to for any reason then robots.txt is good way to indicate they should stop.

]]> By: Murdoch news block-out continues http://www.malcolmcoles.co.uk/blog/times-newsnow-crawling-linking/#comment-7135 Murdoch news block-out continues Tue, 12 Jan 2010 13:54:32 +0000 http://www.malcolmcoles.co.uk/blog/?p=3779#comment-7135 [...] like we missed the change going through, but blogger Malcolm Coles didn’t. As of this morning, NewsNow is locked out of Murdoch’s UK newspaper portfolio. [...] [...] like we missed the change going through, but blogger Malcolm Coles didn’t. As of this morning, NewsNow is locked out of Murdoch’s UK newspaper portfolio. [...]

]]> By: Adam Newby http://www.malcolmcoles.co.uk/blog/times-newsnow-crawling-linking/#comment-7104 Adam Newby Mon, 11 Jan 2010 15:20:58 +0000 http://www.malcolmcoles.co.uk/blog/?p=3779#comment-7104 Indeed, the right to crawl and the right to link are not the same, but are they so different? In order to publish a link, one must first obtain it. So even if one has a right to link freely, if one doesn't have the right to obtain those links, that right to link alone wouldn't be worth much. How does one obtain links? By starting somewhere on a site - probably the homepage - then navigating through the links. Looked at this way, crawling is just systematic linking. We recognise that with systematic linking, there is a risk of abusing a website's resources. Using robots.txt - a voluntary standard - to prevent this, is fair. However if sites start to use robots.txt in an arbitrary or discriminatory fashion, that's not so obviously fair. It's one thing NewsNow's user base not being able to reach Times Online content but if, for instance, as has been suggested, News Corp were to use robots.txt to block Google but allow Bing, that would be bad for users of the Internet in general. One of the strengths of any search engine is its neutrality: that the results have not been skewed to the commercial advantage of anyone to which it links. Also, although Times Online has imposed a crawler restriction via robots.txt, it should be emphasised that The Newspaper Licensing Agency Ltd's proposed 'licence' is about linking, not just crawling. This is shown by the existence of the "end user" licence, that intends to grant permission and impose charges on the customers of news monitoring organisations, for the apparent privilege of receiving or circulating links within their own organisations. Is this a slippery slope towards requiring all organisations pay a news tax for permission to "make commercial use" of newspaper content? On a separate note, the Times Online block applies only to NewsNow. Why would this be? If the argument is about paid-for services featuring links to The Times' content, then why has NI not targeted other organisations with paid-for services? Indeed, the right to crawl and the right to link are not the same, but are they so different? In order to publish a link, one must first obtain it. So even if one has a right to link freely, if one doesn't have the right to obtain those links, that right to link alone wouldn't be worth much. How does one obtain links? By starting somewhere on a site - probably the homepage - then navigating through the links. Looked at this way, crawling is just systematic linking.

We recognise that with systematic linking, there is a risk of abusing a website's resources. Using robots.txt - a voluntary standard - to prevent this, is fair. However if sites start to use robots.txt in an arbitrary or discriminatory fashion, that's not so obviously fair. It's one thing NewsNow's user base not being able to reach Times Online content but if, for instance, as has been suggested, News Corp were to use robots.txt to block Google but allow Bing, that would be bad for users of the Internet in general. One of the strengths of any search engine is its neutrality: that the results have not been skewed to the commercial advantage of anyone to which it links.

Also, although Times Online has imposed a crawler restriction via robots.txt, it should be emphasised that The Newspaper Licensing Agency Ltd's proposed 'licence' is about linking, not just crawling. This is shown by the existence of the "end user" licence, that intends to grant permission and impose charges on the customers of news monitoring organisations, for the apparent privilege of receiving or circulating links within their own organisations. Is this a slippery slope towards requiring all organisations pay a news tax for permission to "make commercial use" of newspaper content?

On a separate note, the Times Online block applies only to NewsNow. Why would this be? If the argument is about paid-for services featuring links to The Times' content, then why has NI not targeted other organisations with paid-for services?

]]> By: Recession, Decade of SEO, Big Media and Linking @ Webmaster Chronic Blog http://www.malcolmcoles.co.uk/blog/times-newsnow-crawling-linking/#comment-7076 Recession, Decade of SEO, Big Media and Linking @ Webmaster Chronic Blog Sun, 10 Jan 2010 08:21:15 +0000 http://www.malcolmcoles.co.uk/blog/?p=3779#comment-7076 [...] Right to crawl vs right to link » malcolm coles clueless newspapers say don’t link to us… [...] [...] Right to crawl vs right to link » malcolm coles clueless newspapers say don’t link to us… [...]

]]> By: Your Mum http://www.malcolmcoles.co.uk/blog/times-newsnow-crawling-linking/#comment-7045 Your Mum Fri, 08 Jan 2010 17:00:28 +0000 http://www.malcolmcoles.co.uk/blog/?p=3779#comment-7045 Very nice article Malcolm, i am so proud of you.. have you cleaned your room? Oh and one other thing... if all of these media sites keep denying me and potentially thousands of others from linking to them who is monitoring all the incoming links? Very nice article Malcolm, i am so proud of you.. have you cleaned your room? Oh and one other thing... if all of these media sites keep denying me and potentially thousands of others from linking to them who is monitoring all the incoming links?

]]>