There has been some discussion recently about how we rate products on Overclockers.com. We typically publish three or more reviews per week currently, from a number of different reviewers. People read our reviews for several reasons, but most people have one aim in mind: is it any good? A vast number of different types of ratings are used across review sites, ranging from simple stars and points rankings (n out of 3, 5, 10 …) to percentages, broken down into 10, 5 or even 1% graduations. Other sites take this a step further and have an overall rating derived from a number of sub-categories on which the product is judged. This article takes a look at the different systems, and sets out exactly how we think about reviews and ratings at Overclockers.com.
How Many Categories Are There?
Percentage systems are good for some things, not so good for others. Sites like Metacritic work well with percentages, as they are collating an average score from a number of sites. If Site A rates something as 3/5 stars, Site B rates it as 4/9 points and Site B rates something as 80%, then percentages are a nice easy way to harmonize these; the average would be 61.5%, for example. Or 6.15/10 stars. Or 1.45 dancing antelopes per purple zebra. You get the point, I’m sure.
However, from the point of view of a reviewer: how precise can you get? We can all understand that a “78%” graphics card is better than a “45%” graphics card, but what about versus an “81%” graphics card? Is the extra three percent because the reviewer got a couple of extra MHz out of it, or because the PCB was an attractive color of purple. Having such a finely grained rating system is only workable if there is an incredibly precise set of metrics that each and every product is assessed against.
How Many Reviewers Are There?
Overclockers.com doesn’t employ paid writers. Only one member of Overclockers.com staff, Matt B. (a.k.a. I.M.O.G.), gets a wage from iNet Interactive (who own this site and the associated forums) and it’s not for writing. We source our reviews from a number of people, such as mdcomp, hokie, splat and mudd have contributed a number. Others write one or two articles. It’s a community site, so standardizing review formats is difficult. Each of the editors has spent a number of hours changing article formatting, language and layout to try and make sure we’re as consistent as possible. There will always be noticeable differences between articles – everyone is different, after all – but we try to present as consistent a format as possible. A finely grained rating system makes this a lot harder, as it introduces too many subjective elements to the rating.
The Rating System We Use
With all these things in mind, we’ve stuck to using three ratings. Below is a picture of each badge, plus a description of what it means. If these were to be compared to a star-type rating system, these would most likely correspond to 1, 2 and 3+ stars.
We want people to read our reviews; our reviewers put a serious amount of work into each and every review they write. Please read what they’ve written – don’t rely on an often quite arbitrary points, percentages, or stars system. One of the most important parts of any given review is where the pro’s and con’s are compared, and this is where you will get a good feeling for whether this is a product you want to own or not.
Thanks go to a number of forum staff for their input into this discussion: Jay_III, muddocktor, splat, hokiealumnus, I.M.O.G., splat, MattNo5ss, Thideras, macklin01 and the rest.
I give this article a hearty 4.3 dancing pink antelopes per watermelon-eating brown cardboard tiger-chewed bucket of sugar!