I'm sure these guys have no compunction against ignoring robots.txt if it makes them money by doing so.

is there some written law that holds people to following robots.txt? if not, how is it even possible to call it a weakness?

This one [robotstxt.org].

On the other hand, that's an utterly asinine comment to have made (the one you quote, not yours). Of course they'll ignore it, why on Earth wouldn't they? It is in no way binding, and robots are free to ignore it, just as site owners are free to block connections from specific incoming IP addresses, the owners of those IPs are free to switch to new ones, and so on, ad infinitum.

News publishers using Attributor probably won't attack Slashdot for excerpting one paragraph from a ten-paragraph story any time soon. From the summary:

Are you kidding? ACTA's going to harmonise everything so closely to the US that they'll be able to prosecute anyone.

While they're at it, can they take down forum/mailinglist mirrors too?

It is extremely annoying when searching to find that the top 30 results all contain the exact same forum or blog post.

And in the process find all the commercial sites using my copyrighted Flickr photos for their own purposes without my permission or payment. I'm tired of sending invoices and dealing with companies who tell you that your photo wasn't worth the $300 you charge and instead send you $50 thinking that it will clear up the matter.

I love the hypocrisy of all of this. They are just as much at fault as any of those aggregation blogs. They just have more money to be a pain in the ass.

Crawlers which ignore robots.txt can be detected and blocked. Therefore, even though adhering to the robots exclusion standard isn't the law, it is a good idea.

On the other hand I don't see how robots.txt is even relevant here: Surely most of these sites depend on search engine traffic, so they can't hide the articles from crawlers.

Unless an article is very short, quoting 80% of it is not fair use. So for now, I think they have every right to take steps against sites making money from their content without compensation.

Yes, I am cynical enough to expect the reasonable 80% limit to be lowered over time until it reaches unreasonable levels. But let's hold the flames until they have actually crossed that line.

Since when did Slashdot ever use 80% of an article verbatim?

Sorry, no, any website doing *that* should be shut down. I hate those assholes. They're the reason why a search for a given term in Google pops up thousands of sites with the *exact same content*, just ripped from one another.

You bring up an interesting point. According to the Slashdot TOS you own that comment. Who should attributor pursue?

It's not just copyright. The slow but steady alignment of copyright holders, oppressive governments, legal changes, media pressure and surveillance technology has wound itself around the internet worldwide, and now the real pressure is being applied. This is a secular change, largely unobservable over smaller intervals, but the end result is that the web in 10 and 20 years time will be a noticeably less free place than it is today. Everything you do online will be monitored, everything will be logged, everything will be legally defined and controlled, and every infringement will be subject to criminal penalties.

The parties responsible have the support of the politicians, the censors, the press, the money men and most of the public. We used to have the support of the geeks and their creativity in bypassing censorship. But let's face it; geeks have not created a truly disruptive technology since BitTorrent almost ten years ago. While Geekdom slept, the likes of Cisco and the major Telcos have constructed a frightening array of technologies for surveillance and control of the internet, and the fruit of their efforts can be seen in China, Iran and now even countries like Australia. Soon it will be seen all over the world.

The Web has changed. Governments are no longer going to tolerate the freedom and anarchy that it grants to the population at large. They now have the means, method and opportunity to put this genie back in the bottle. This crackdown is the first offensive on what is going to be a wide front. Expect the free net to lose.

Wouldn't we have to see the actual takedown notice to be sure it was faulty? Maybe there was another copy of the Calculus book on Scribd and that one was MacMillan's, and perhaps the DMCA was a bit sloppily written, or perhaps Scribd just didn't read it right.

It could happen. I can look at Scribd right now and see several books with the title Calculus on them, only one can be yours, the others? Maybe they approve of their works being on Scribd, maybe not.

So do I think Attributor was particularly scummy just because you got an email from Scribd? No, not on its own. Everybody can make a mistake without it being malicious.

80% is a reasonable starting point. If they start lowering it, we'll have to express our righteous indignation then. Fair use, when interpreted, is generally considered a LOT lower than routinely cutting-and-pasting 80% of articles, so they have a long way to lower it before we can honestly call our indignation righteous.

Seriously, this really isn't a "slippery slope" situation. It seems to be a well-thought-out and sane set of guidelines. If anything, they are being a bit generous for now, and they can still tighten this quite a bit without coming close to busting "fair use" or even "reasonable use".

Basically they are saying, "if you routinely use 80%+ of our articles as your own content, we're asking you to stop. We won't sue you for any past uses, we just want to make it clear that this isn't cool any more."

A fair usage (not the lack of quotes, I am not talking about a legal doctrine) would be to use about 20% of the source article (properly attributed) with a link back to the original article. Give credit where it's due (and cite your sources). Then add your own thoughts, or don't. But don't take whole-cloth articles and post them on your own site with your own ads.

Every discussion board I've ever participated in has pretty much recommended some really close variant to this anyway. It usually reads something like "cite a paragraph or two at most and have a link to the source article plainly visible nearby".

It's one thing to make a mistake, and entirely another to invoke the law to enforce a mistake. You're right, it's entirely possible the takedown was poorly written, but therein lies the problem with the takedown mechanism - there's no standard that it must reach before it can be served. Thus mistakes, honest or otherwise threaten people with very real, very wide-ranging and scary/expensive actions - completely in error. As such, as reasonable people, we expect anyone taking action as serious as a takedown would apply a good dose of due diligence. Sadly though, not everyone views takedowns as "serious action", and so perhaps aren't taking the care with them that perhaps they should.

Click the link to read the first 21%

The first offending sites to be targeted will be those using 80% or more of copyrighted stories more than 10 times per month.

In the first stage of a multi-step process aimed at encouraging copyright compliance instead of punishing scofflaws, Pitkow said online publishers identified by his company will be sent a letter informing them of the violations and urging them to enter into license agreements with the publishers whose content appears on their sites.

If copyright pirates refuse to pay, Attributor will request the major search engines to remove offending pages from search results and will ask banner services to stop serving ads to pages containing unauthorized content. The search engines and ad services are required to immediately honor such requests by the federal Digital Millennium Copyright Act (DMCA).

If the above efforts fail, Attributor will ask hosting services to take down pirate sites. Because hosting services face legal liability under the DCMA if they do not comply, they will act quickly, said Pitkow.

"We are not going after past damages" from sites running unauthorized content said Pitkow. The emphasis, he said is "to engage with publishers to bring them into compliance" by getting them to agree to pay license fees to copyright holders in the future.

License fees, which are set by each of the individual organizations producing content, may range from token sums for a small publisher to several hundred dollars for yearlong rights to a piece from a major publisher, said Pitkow.

Attributor identifies copyright violators by scraping the web to find copyrighted content on unauthorized sites. A team of investigators will contact violators in an effort to bring them into compliance or, alternatively, begin taking action under DMCA.

Offshore sites will not be immune from the crackdown, said Pitkow, because almost all of them depend on banner ads served by U.S.-based services. Because the DMCA requires the ad service to act against any violator, Attributor says it can interdict the revenue lifeline at any offending site in the world.

Attributor already has been engaged by several major book publishers to get unauthorized eBooks off unauthorized sites. "And we have 99% success rate," he said.

Right... because a judge will find that offer, consideration, and acceptance of a contract took place between a webserver and a bot? The court case you cite is irrelevant to an automated program that has no understanding and cannot accept conditions presented online.

Awesome, so anyone can DoS a server, send mass spam or distribute a virus as long as a bot does it, because a judge will rule that the bot acted on its own and wasn't developed or set loose by anyone at all.

If the software wrote itself you might have a point, otherwise the people who wrote it are the ones responsible for how it acts.