On the Integrity of Hardware Review Sites 263
leathered writes "Charlie Demerjian of The Inquirer has posted an interesting article on the integrity of hardware review sites. Apparently the benefits of running such a site go far beyond advertising revenue with a fair amount of 'sweeteners' from the hardware manufacturers to say the least. All is not lost as Charlie informs us that there are a small number are flying the flag for trustworthy reviews, but the question of which sites we can trust remains." I like Daniel Rutter's (of Dan's Data) policy best.
sponsorship (Score:3, Interesting)
It seems like I remember Anand buying himself a Porsche for his 16th or 18th birthday, using the payola from his hardware review site.
While his business acumen is to be commended, I can imagine it would be difficult to remain 100% objective under such circumstances.
http://www.anandtech.com [anandtech.com]
Identification of sites he's accusing? (Score:3, Interesting)
My pick (Score:4, Interesting)
Re:Identification of sites he's accusing? (Score:4, Interesting)
Since the article he wrote was released today, I took a look at both Tom's Hardware and AnandTech. Both had "previews" of the Dual Core Intel chips...both major sites. That's just an educated guess, but seems correct.
Let's list them then folks... (Score:5, Interesting)
Anybody have any sites that they feel are bad or good (with respect to this article)? Please list a few reasons too, few examples if you can -- it makes it nice to see if these points are driven home over time by reading the reviews on different sites
Re:Identification of sites he's accusing? (Score:3, Interesting)
Now sites like Toms Hardware Guide and Anandtech I am not sure about.
At least you know you are getting a biased view at a site like AMDZone, heh.
Not Anandtech (Score:3, Interesting)
Re:Trust? On the net? (Score:4, Interesting)
the real problem with these hardware sites is that the writers are often clueless about the actual hardware and what it does - and make sometimes claims that are not even physically possible. trusting such guys to review something that supposedly does something is no good when they lack the knoweledge to make the decision if the product even works as advertised or not. a lot of the 'reviews' are just a "thank you for free hardware" pieces that are basically referates of the products description followed by a thumbs up icon(or whatever the particular site uses for 'editors choice' that every product they review happens to get).
Re:The Inquirer paid /. to run this article. (Score:2, Interesting)
I remember reading a comment of slashdot some time ago that the internet was "the last source of uncensored, unbiased information" and a +5 insightful reply saying "you're half right"
every site on the web is pushing some sort of agenda, whether it be for political motive, religious belief or big handfuls of cash.
Lets face it
Re:Two Words (Score:3, Interesting)
Among several review sites I look through, TH did not seem particularly bad, maybe above average on the integrity of review front. And they usually pick fairly representative sample - do not remember obvious omissions or mismatches in comparisons.
Though it is not my favorite site, indeed.
Sweeteners, sure, everybody needs to eat. It is what you do with them. I would not mind demanding something to place a story, especially about some fringe product nobody cares about - I would only mind outright lie in a story, or some really poor review from a technical point of view.
Learned my lesson well with deathstars (Score:1, Interesting)
Disaster strikes, times two. First drive fails with the infamous [techreport.com] screech of death [techreport.com]. Shut system down, try to figure out what to do, read online, buy third drive, plug in, start up, start to backup data from backup drive, lightening strikes twice. Lost years of work previously migrated from older systems.
My experience with trying to RMA the drives (just a few months old) was so bad that I made it my mission to ensure that no other newbies or small business owners or individuals went through what I did. Or do my best trying. So what did I do?
It became apparent in the weeks and months that followed that IBM GXP drives were so bad that they were failing by the hundreds at hosting providers, that many others were having such problems that many stories and threads were started on some hardware tech sites, and that even a class action lawsuit [techreport.com] was started over them.
So I started contacting hardware tech sites that were still glowing about the GXP drives, and asked them to revise their review or remove the review. Some didn't answer (CNet). One or two less famous sites actually removed a review, or added a disclaimer, maybe because I wrote or because there was simply too much bad press to ignore. Some other sites were using the drives in their computers for testing other hardware. And listing the GXP drives when describing what hardware they used to test. I saw this as an endorsement, so I asked them to stop using the GXP drives, explaining my position and providing links about the stories on the hard drives. Some site owners ignored me *cough* Tom's Hardware *cough* and continued using the drives even after repeated email requests that they don't, one actually emailed back that those were the drives he bought and couldn't afford to replace them, and others soon after stopped using the drives and switched to others as far as I could tell. Some sites (one that I recall, forget the name, haven't heard about it since then) actually featured the GXP drives on the top left of their front page, as a great drive (banner link to a review page), many months after the bad news on the drives came out and after the class action lawsuit announcement made it on slashdot (and after the news on the hosting provider losing hundreds of the drives). That site didn't even bother answering my emails about how wrong it was for them to push the drive in light of all the problems about the drives that everyone was shouting about.
That little episode was enlightening as to who I could trust with advice on purchasing decisions and who to avoid. Now, some 4 years later? Who can really be trusted for accurate reviews? Buyer beware, and spread the risk. Especially on hard drives, use raid, backup to optical media and buy more than one brand of drive and buy drives from more than one source. And then cross your fingers and pray.
Another way of finding good hardware... (Score:4, Interesting)
Not any more Re:Two Words (Score:4, Interesting)
Re:epinions (Score:2, Interesting)
You really think so? Personally I think it's absolutely terrible - apart from the "trying to make a living out of it" so-called-reviewers trying to pad up reviews with a bunch of copy/paste tripe, for serious hardware -- stereos, cameras, cars, whatever -- the reviews are overwhelmingly positive.
I attribute this to two probable causes - one is that when these reviewers actually own the product there is a natural tendency to defend what you bought (even if it turned out to be a dud). If someone saved up for months to buy their Kenwood Shittastic 5000, they're going to be damn sure in their head that it's the world's greatest stereo.
The second problem is that a lot of reviewers seem to correlate their own goals (making money) to epinions goals (making money). Epinion makes most of their money, I presume, from people getting SOLD on a product. Thus it is naturally in every reviewers best interest to shill every POS to unsuspecting visitors.
I've tried to use epinions a few times to help make purchasing decisions, however each time I found it absolutely terrible.
Re:What I do... (Score:3, Interesting)
I am one of those naive that really don't have a clue what hardware review sites to trust. My comfort is that I am probably far from alone, in this matter.
To assist me and other naives(sp?), please join this silly poll and review the following sites (regarding credibility) with a scale ranging from 1 to 10, where 1 is "No credibility at all" and 10 is "Perfect credibility, these guys wouldn't post a biased review for world domination":
About PC Hardware Reviews [about.com]
Ace's Hardware [aceshardware.com]
Anandtech [anandtech.com]
Ars Technica [arstechnica.com]
Beyond 3D [beyond3d.com]
Cnet Reviews [cnet.com]
Dan's Data [dansdata.com]
Dev Hardware [devhardware.com]
Extremetech [extremetech.com]
Firingsquad [firingsquad.com]
[H]ard|OCP [hardocp.com]
Hardware Analysis [hardwareanalysis.com]
Hardwarecentral [hardwarecentral.com]
Hardwarezone [hardwarezone.com]
IT Reviews [itreviews.co.uk]
OcPrices [ocprices.com]
Overclockers.com [overclockers.com]
ProCooling.com [procooling.com]
The Tech Report [tech-report.com]
The Tech Zone [thetechzone.com]
Tom's Hardware [tomshardware.com]
TrustedReviews [trustedreviews.com]
Viperlair [viperlair.com]
Xtreme Resources [xtremeresources.com]
If you know only a few of them, give your opinion on those.
Maybe someone with the right facilities could set up an independent poll?
Re:That's the truth (Score:4, Interesting)
Buy your own hardware at retail, using obfuscated identifying information.
Carry NO 3rd-party advertising of any kind.
Disallow manufacturers from using your name, excerpts from reviews, or any other acknowledgment that you exist in THEIR advertising.
Of course, that means your revenue model is going to have to change, either to a subscription site; privately funded, capped, and transparent donations; or through selling stuff (books?) of some kind.
Re:Two Words (Score:5, Interesting)
Tom's Hardware was the primary reason I stopped reading his site... er....
Let me clarify, A long time ago, during the BH6 heydays, I used to read Tom's Hardware almost religiously. I scarfed down every article with near fever, trying to stay on top of the latest hardware reviews. Then I got stupid and started dating, so I ignored Ol' Tom for a few years. When my other half and I broke up, I started digging into the hardware scene again, so I went back to Tom's to get up to speed.
A few articles into it, I realized the, "feel," of the articles changed. There didn't seem to be as much useful, practical, information in them anymore. They didn't have any interesting opinions on the hardware being reviewed. In fact, I don't think I found an article that was harshly negative in any way. Nor did I find any articles with useful grit-in-your-teeth information. Initially, I thought the, "neutralism," being presented in the articles were because of the close tolerances of the hardware being reviewed. I mean overclocking a 333MHz CPU to 450MHz had greater net results than overclocking the more modern CPU's now. But I put my faith in Tom, because it's supposed to be the best. After all, his site does say:
So who was I to argue with that?
I just can't put my finger on what, exactly, Tom is missing. As near as I can figure, Tom's Hardware reviews read too much like brochures. It's just enough to try and get a person interested. But whatever it is, it's enough to prevent me from utilizing his site for anything more than keeping track of the latest hardware. Nowadays, if I want real hard and honest opinions, I just hop on over to my favorite forum/BBS/IRC/whatever and sort through the flamefests to get a feel for a particular piece of hardwares viability.
Best reviews ever: IRC (Score:4, Interesting)
booradley> Buy me! I'm ever so sexy
booradley> ok. come home with me and we'll play among the stars
booradley> tee hee! I love you, boo!
booradley> I love you too, audigy
booradley>
booradley> there, you're all installed. how do you feel?
neshura> down in front!
booradley> audigy> LET JESUS FUCK YOU! VRAAAGH!
* audience gasps.
booradley> * audigy is putting noise across your PCI channels
booradley> hard drive> Mein leben!
booradley> * hard drive has died
booradley> audigy> Blaaah! blaaaugh! your mother sucks cocks in hell! graaagh!
booradley> modem> aaieee
booradley> *modem has died
booradley> and the new modem I got connects at 32k tops
Shendal> By far, that's the best one-act IRC play I've read this season. Do I smell a Tony award?
----
That's from bash.org. It's pretty much how it goes. I ask friends that have more disposable income and they tell me about products they've blown money on. heh. It's another reason that I chose Sirius over XM for my satellite radio - finding an objective review online was a waste of time because everyone was just spewing the same marketing drivel or Google results were flooded with e-stores. I just spent some time and talked to people I knew about both products and went from there.
The Internet is becoming increasingly *useless* for information that doesn't come straight out of some marketing droid.
Two words: PR (Score:3, Interesting)
PR is a neat invention. Just the idea of making it someone's responsibility and livelihood to ensure that products and company get presented in the media in a desired way and with desired frequency is brilliant. Those people are really good at it, the result is that currently technical journalism without corporate PR is hard to imagine.
Re:what if everyone's in on it? (Score:3, Interesting)
In cases where reviews are bad, it usually comes a fair bit after release to give the public a chance to waste their money on it before they are notified that the thing in question is a piece of shit. It's a win-win for the product companies and the reviewers.
Re:Trust? On the net? (Score:5, Interesting)
We also did marketing CDs for several equipment manufacturers, although our name never appeared in any credits. That part of the business was hush hush. We weren't to disclose those relationships.
We got lots of free equipment from these manufacturers, and others that wanted to be reviewed. We were specifically told to push certain products at the seminars and classes, and to give good reviews to "our" manufacturers products. We were also supposed to push product on the web site's forums, when people asked for advice. We never returned anything. In fact, the boss would give product to us employees as bonuses. I just checked the website, and they're still doing the same thing.
I openly questioned the ethics of this, and this and other factors led the boss to conclude that I wasn't a "team player", so I was let go. Honestly, I was glad, because there was a lot of other bullshit at that company as well. For a while, though, I was able to rationalize this because the products we were pushing were good products (for example, Canon DV cameras or Miller tripods). Still, when something is wrong, you know it's wrong, no matter how you try to justify it. If it wasn't wrong, why did we have to hide our business relationship with Canon?
Since this is pretty much par for the course, I think that if a magazine or web site really wanted to present unbiased, ethical reviews, they should state somewhere that they return the equipment, that their editorial is separate from advertising, and also disclose if the product manufacturer is an advertiser.
Of course, there still could be abuses, but when someone puts a policy in writing, it's a little harder to wiggle around basic ethics.
Reviewers are all liars, whores, and crooks!! (Score:2, Interesting)
I tested and wrote for a trade publication that did hardware and software reviews for three years. During that time, I learned the following:
Anand's Rebuttal (Score:2, Interesting)
$26
My dad came to this country with something like $26 in his pocket; $26 and a scholarship to UMASS. I didn't know about that until I was around 14, our family went to another family's house for dinner and it came up in after dinner conversation. I was honestly floored - all of the sudden everything in life made sense to me, I was given the drive that I needed to go anywhere and do anything in life. My dad took essentially nothing and raised a family out of it; we all helped, but one man's ability to do that is what I truly define as successful. We weren't rich, but he (along with the help of my mom) made sure that we could live in comfort as a family. My dad is what sparked my drive; he's the reason that if I get a product to review on Friday, and the review has to go up on Monday, I bust my ass all weekend to make sure it gets done. He instilled in me a true understanding of what hard work is really about, and that's a major cornerstone of who I am today.
My mom gave me an understanding of how to do something with myself and an understanding of ethics. She showed me what truly caring about something really meant, about what being selfless meant and gave me the foundation that allowed me to develop my own perspectives on the world. She didn't teach me right from wrong, she taught me how to figure out what's right and what's wrong. And I'll never forget that which she's taught me to this day. It wasn't until college that I really understood what she had done for me; she dropped out of college to take care of me, to raise me. She gave up her dreams of being a doctor, to live her new dream of having a son. She cared for me more than anyone ever could, and seeing and understanding that also helped shaped who I am today.
I started AnandTech almost exactly 8 years ago: April 26, 1997. I was a freshman in high school, 14 at the time, and completely into this stuff. I started AnandTech not as a business, but as something that I thought would be cool to do. I started it humble, and to this day I will never forget my beginnings. There's no room for big egos in writing, I hate reading it and I'm sure you all do too.
I started the website with nothing, it was a free site on Geocities and I had no hardware other than the scraps of my system. But I worked hard these past 8 years, AnandTech grew from nothing to where it is today - with over 6 million monthly unique readers. I've had one basic principle when it comes to how to deal with those readers, and it goes something like this:
Regardless of how many people come to the site, I look at it as each person coming to me with their money in hand, wanting to know what to purchase. Let's say the average hardware upgrade costs $150, that's 6 million people x $150. I don't have to let you know that that's an absolutely ridiculous amount of money. To trade the trust that you all are placing in me and my staff for any amount of anything, is just unfathomable. While I'm sure there are folks that do it, I am not one of them.
At the same time, if we didn't value your trust so highly, we'd be gone in an instant. AnandTech readers make their buying decisions based, in part, on our articles. If we gave some bad advice that resulted in a poor purchase, do you think we're going to keep those readers for long? Nope, common sense right?
Next let's talk about this myth of articles and exclusivity. To a journalist, an exclusive on an article is a huge deal, because it means that you'll get all the attention about this one topic. Yet another reason why I hate journalists, they are far too short sighted. One thing I learned very early on (and you'll notice this in the work I do) is that being first to break a story gives you a large influx of short term traffic, but does nothing for you long term. You can have all the exclusives in the world, but if your content is crap then they mean nothing. At the same time, you can be 3 weeks late to review something, but if it's the most thorou