On the Integrity of Hardware Review Sites

leathered writes "Charlie Demerjian of The Inquirer has posted an interesting article on the integrity of hardware review sites. Apparently the benefits of running such a site go far beyond advertising revenue with a fair amount of 'sweeteners' from the hardware manufacturers to say the least. All is not lost as Charlie informs us that there are a small number are flying the flag for trustworthy reviews, but the question of which sites we can trust remains." I like Daniel Rutter's (of Dan's Data) policy best.

Two Words

[stone2020]

Tom's Hardware

That's the truth

[Anonymous Coward]

We (voodoo networks) are in the process of starting up a review site for backend systems for server rooms, closets, etc. The reasons that drove us to do this are simple - it's almost impossible to find objective reviews of non-consumer gear, and when we do find them they are pretty much copies of manufacturers flyers.

The odd thing so far has been the reaction (or lack thereof) from manufacturers. I've lost track of the number of PR reps I've spoken with who asked my how much a listing or review cost. Most where surprised when I told them that serverroomstuff.com won't be charging for reviews - they are objective.

We are planning on accepting advertising, but the plan for that is simple - your ad dollars won't buy you a good review if your product doesn't stand up to it's claims, period.

epinions

[unk1911]

epinions.com remains a good across-the-board review site, not just for hardware. good detailed specs appear next to user writeups. plus you can make money by writing reviews, which i have done.

--
http://unk1911.blogspot.com [blogspot.com]

Re:sponsorship

[Erwos]

Anandtech is hardly the site I'd be holding up as an example of payola. The man has reasonable benchmarks, asks for user input on tests, and isn't afraid to say "hey, your product has no place in the market". They only gave a silver to the SN25P, even though it more or less shined in every conceivable way. Nobody buys 2nd place.

Anand's probably rolling in dough because not only is his hardware site one of the most popular, but he also has one of the most popular forums on the Internet in it. If you factor in that the ads on there are actually pretty well-targetted, I wouldn't be surprised to hear he has an excellent click-through rate.

It's probably difficult to remain objective, but that's what professionals do. Anand and his crew are obviously professionals.

-Erwos

Re:Two Words

[Anonymous Coward]

Two more. Bull shit. As someone who used to work as a technical contact within a hardware marketing team, I can personally guarantee that Tom's Hardware demanded sweeteners, more than any other review site we worked with. And they were arrogant jackasses to boot.

Re:Im not sure if he understands multithreading

[krf]

It depends upon what you are doing. Most games probably aren't written to take advantage of multiple threads, but things like renderers or image processors most definately are. The difference is that people (legitimately) running high-end software are more likely to have higher-end machines, so the programmers can aim their system requirements higher to get the job done.

Also, some kinds of tasks are just more amenable to parallelization than others. For example, if you are trying to ray trace an animation, you can usually process two frames independantly from each other. Running convolution filters on large images can also be split into convenient chunks.

In a game, you could do things like run your AI code in a separate thread, and it would run faster on a multiple cpu system. But it's harder to write. If dual-core cpus become more cheaper and more popular, you'll probably see more games written to take advantage of them. Game developers just haven't bothered yet, because the extra complexity involved with writing everything to be thread-safe isn't worth the speed improvement for just a small number of users.

viperlair did a good writeup

[enrico_suave]

viperlair did a good write up on what i like to call hardware review site payola [viperlair.com] a while back.

e.

Re:Trust? On the net?

[ZephyrXero]

"...a lot of the 'reviews' are just a "thank you for free hardware" pieces..."

Hmm...that's funny. I used to work for a review site and we always had to return the hardware after we reviewed it. Now software, that's a whole other story ;)

Bribed, greedy and/or imcopentent

[9mm Censor]

Having written a few reviews for www.overclockers.com myself, I have looked into this and found a few things, while reading other reviews.

Reviews tend to fall into a few catagories.

Bribed: The supplier has give something to the review as incentive for a good review. This is the supplier doing the dirty deed, the reviewer just goes along with it because they are profiting.

Greedy: The reviewer hands out praise and awards like candy, to keep the goods flowing. The reviewer is just keeps pumping out favourable reviews, to keep getting products for the review to play with or sell. This is the reviewer doing the dirty deed and the supplier just goes along with it, because its good PR.

Incompetance: Some reviewers just suck. They simply dont know how to get the numbers right, or their testing done properly. I just call them idiots.

Lastly and most rare, is the competant, unbiased reviewer. They know what they are doing and dont pull any BS.

I find www.overclockers.com and www.procooling.com to be good, www.xtremeresources.com and www.anandtech.com are also pretty good.

HardOCP is ok.

www.madshrimps.de and Toms Hardware both suck.

Re:what if everyone's in on it?

[Anonymous Coward]

Let's not forget that most of these guys litter their sites with advertisements for the very product they are reviewing, too.

Reminds me of the time, many years ago, when I read a "shootout" style review of several network cards. Throughout the review, two things became clear: they really liked the Linksys offering (please no fanboy flames here; I'm not giving an opinion of the cards, just relating what was written) and didn't care too much for the card from 3Com.

Imagine my surprise when the 3Com card was declared the winner of the shootout in the conclusion. I reread the review, and yep, it still sounded the same; didn't like the 3Com, liked the Linksys. I pondered this for a few seconds, but all became clear when I noticed the 3Com banner ad at the top of the page.

The fact is, people are stupid and greedy. They always have been, they always will be. Keep it in mind whenever you read anything written for public consumption and you will find you get disappointed (and misled) far less often.

Re:sponsorship

[cheinonen]

First, I recall him getting a BMW convertable back then, not a Porsche. Second, that article definately takes a stab at Anand when it talks about how one site came up with a way to benchmark when people are running multiple apps at once (which Anand did), but then attacks them for not benching games. Had the person read Anand's blog on the site to keep up with what he was working on, he could have seen that Anand lacked the time to get all the benchmarks done over the weekend, did the ones he thought most important, and the gaming results went up yesterday or today.

I think it was well assumed that games would show no advantage on standard benchmarks since they don't spawn multiple threads, and every site that ran conventional benchmarks showed that on Monday. Today, however, Anand has those results, and then also tests games with thinks like Norton Antivirus and MS Antispyware running in the background, which many of us do have running when we game. On those, the dual core showed better performance.

Anand is the one site I always trust, it's too bad the article didn't have the guts to come out and name who they thought was being bought.

Re:sponsorship

[ShieldW0lf]

I can't give you any examples, as I haven't gone there in a couple of years, but I found his site to be one of the worse ones. Toms too.

If you want accuracy in your information, best is to check out a support site for the product that isn't run by the manufacturer. amdmb.com for example I've found to be great for finding out about unstable hardware. I've seen critical posts disappear in manufacturer run forums before, so I wouldn't assume that a lack of people with problems there means there aren't any.

Re:"I take goods you send me for free & review

[ergo98]

They weight reliability and depreciation quite high in their rankings, and the reality is that the guy who buys a Toyota/Honda will, on average, have far fewer problems and will recoup a lot of his initial expense than the guy with a Ford for instance.

In any case their top ranked small car for a while has been the Ford Focus. Is it possible that you were a little jaded that they didn't pick your pet car/minivan as the best?

Re:Let's list them then folks...

[hawkear]

Why would they sell harware they use daily?

They don't do comprehensive motherboard or processor reviews like other sites. You're not going to find a table with 15 different combinations of hardware compared. Usually it's just one product.

The good, the bad, and the ugly

[Sivar]

Slashdotters are far more likely than the average joe to recognize a good review from a bad one.

Please post your good/bad/ugly (badly written but probably not corrupt) website opinions here.

I have been following hardware reviews since about 1995, have written a few myself, and am fairly knowledgeable on the inner-workings of hardware and of English, so I have become fairly critical of reviews. Here's my brief list:

The good (Quality, non-biased, well-written reviews with reproduceable results and non-marketfluff technical explanations):

- Ace's Hardware (on the odd occasion they review something)

- StorageReview.com (I am a volunteer moderator on the forums there, so I may be considered biased, but I doubt anyone can point out a website with better hard drive reviews)
- xbitlabs.com
- Anandtech (usually, Anand usually writes well but some of the staff writes rather awkwardly, and some of their reviews, like that of the Addtronics W8500 case (and most of their other case reviews) are staggeringly awful)
- Ars Technica, though I wouldn't classify them as a hardware review site.
- Maybe HardOCP. I don't read them much, but the odd occasion I have read them didn't set off any BS detectors. Perhaps a bit unprofessional.
- Maybe DansData.com. I rarely read them, but I have always been impressed with what I read.

The bad:
- tomshardware.com (These guys were my favorite review site years and years ago, but their quality and either integrity or competence (or both) dropped staggeringly years ago. Their hard drive reviews in particular are awful beyond belief, though that may be because I am more familiar with that type of review than others).
-

The ugly:
- Just about any review site whose URL contains the word "Game." These websites tend to be full of reviewers that seem to be random average highschool kids with no real understanding of what they are reviewing, how to write a review, or how to test or rate the product they are reviewing. The word "arbitrary" comes to mind. Note that I do not put Gamespot.com in this category (even considering their owner).
- CNet. I can't tell whether the reviewers are incompetent or bribed, but the reviews (usually, but not always) are quite lame.

Any opinions on other sites?