If I were a nice normal reviewer, I’d just run the benchmarks one or three or however many times and come up with a single number you could refer to.
But I’m not.
I come up with all different kinds of weird angles and approaches. I don’t just wonder about the equipment, I wonder about the benchmarks,
and sometimes wonder about the benchmarkers.
I think the benchmarking companies are on to me. They’ve been moving towards one single number, and not letting you figure out how they came up with it.
In this series, you’re going to see why they don’t want to let you figure out how they came up with it.
All I did was change one little parameter in my testing. All I did was change the monitor resolution.
(All tests shown here are run at 16-bit resolution)
|SYSmark ® 2000 Rating|
|Internet Content Creation|
|Elastic Reality ® 3.1|
|NaturallySpeaking ® Pref 4.0|
|Netscape ® Communicator|
|Paradox ® 9.0|
|Photoshop ® 5.5|
|PowerPoint ® 2000|
|Premiere ® 5.1|
|Windows Media Encoder 4.0|
Let’s get graphical:
Hmmm. Pretty big jump, isn’t it? Let’s see what caused it.
No, not Internet applications.
Hmmm, Office Apps.
Actually, only four applications got a big boost from this.
Why Should I Care?
A few reasons, actually.
A couple adjustments you might not normally think of can yield improvements in some applications, or
may not even though you’d think they might.
Here, the information is fairly useless in real life, since any computing performance gain you’d get from a smaller screen in these
activities would be more than offset by human productivity loss scrolling around all the time.
But this demonstrates what I’ve been seeing in more serious tests, you just don’t get increases (or decreases) in performance
across the board, or even for most applications, when you change a (more serious) parameter. That means if you just look at a single number
like the overall Sysmark, you will get the wrong idea on what you can expect from a change.
For instance, later on this series, we’ll look at what extra RAM does for you. If you just look at the single SysMark2000 number, it shows
about a 4% increase in performance, but most of that 4% comes from ONE application. (We’ll also point out why you can’t necessarily
rely even on the benchmark for your application, depending on what you do with it.
The second, more immediate reason why you should care is that if you don’t know what the monitor resolution is in a review, and don’t realize
it’s important, you might think one board performs better than another when the real difference is just a monitor setting. There was a 7% difference
in scores due to just that, which is a lot bigger performance difference than we’re seeing between motherboards.
For instance, the Epox EP-8K7A has been getting a good deal of attention lately. I’ve been able to more or less equalize testing conditions between one review
of it and the K7Master, and I’m fairly sure the K7Master is a tiny bit slower than the Epox (need to do one other thing to make pretty sure). But if I (or someone less scrupulous) wanted the MSI to “win,”
all I’d have to do is change the monitor settings, and just not mention it. I’ve seen one case where I suspect the reviewer ran Sysmark at 640X480, which might make some sense for games, but not for most office apps.
The idea is to explore what changes do or don’t do for you in various apps, which will hopefully advise you on what will help you and what won’t in your own machine and activities.
What does color depth do to performance? We’ll discuss that tomorrow.