Ed Bott blogged last week about a pay site for programmers that frequently shows up in Google search results and irritates him and others because they find themselves at a pay wall when they click through to results. His somewhat heavy-handed solution? To ask Google to remove all pay sites from its index.
Of course there are many subscription websites that provide very useful knowledge and expertise to their communities. There are many sites worth paying for because they provide unique, actionable and valuable information, frequently updated, from trusted sources. We know because at SubHub we provide the publishing platform for many of these sites and we see the communities that have grown around them.
There is no reason why these sites should not be findable via Google for those who would find them relevant. If Google were to remove them from its index then Google itself would be providing an incomplete picture of available information sources, and would be failing at its mission of indexing all of the world’s information — not just all of the world’s free information.
What is interesting is that despite the complaints Botts' commenters seem to have about the specific site in question, many of the comments then go on to explain how to game the system using caching or blocking of cookies in order to get the information from the site for free. So it’s not that the information is not valuable, it’s that some people don’t want to pay for it.
I will leave it to others to comment on the ethics of doing this, but it strikes me as a bit dubious. If you don’t believe in paying for the information, fine, but don’t steal it. Many of Botts' readers seem to be programmers who presumably would object if someone swiped their intellectual property — their code — so why would they be comfortable swiping someone else’s intellectual property — their content?
What they seem to be saying is that they agree the content on the pay site is indeed valuable to them professionally, but they are too cheap to pay for it. Which does not lead to the conclusion that Google should remove such sites from its index, but rather that pay sites use solutions that do not make it quite so easy to get at the paid content.
