A Different Angle–Part I

If I were a nice normal reviewer, I’d just run the benchmarks one or three or however many times and come up with a single number you could refer to.

But I’m not.

I come up with all different kinds of weird angles and approaches. I don’t just wonder about the equipment, I wonder about the benchmarks,
and sometimes wonder about the benchmarkers.

I think the benchmarking companies are on to me. They’ve been moving towards one single number, and not letting you figure out how they came up with it.

In this series, you’re going to see why they don’t want to let you figure out how they came up with it.

The Setup

  • Windows 98SE
  • 1.13GHz 266MHz FSB Athlon overclocked to 1.425Ghz (9.5X150)
  • MSI K7Master ver 1.1 board, BIOS 1.1,
  • 5126MB Crucial PC2100 running at CAS2, Setting 8-8-7-5-2-2-2-2
  • 45GB IBM Deskstar ATA100 7200RMP Hard Drive
  • Matrox G450 Dualhead 32Mb, driver version 6.51, Dualhead disabled
  • Swap drive optimized with Norton Speed Disk
  • All I did was change one little parameter in my testing. All I did was change the monitor resolution.

    (All tests shown here are run at 16-bit resolution)

    TEST RESULTS
    Application

    1600X1200

    1024X768

    800X600

    640X480

    SYSmark ® 2000 Rating

    253

    263

    267

    270

    Internet Content Creation

    266

    268

    268

    270

    Office Productivity

    244

    257

    267

    270

    Bryce 4

    339

    345

    349

    337

    CorelDraw(TM) 9

    404

    455

    475

    480

    Elastic Reality ® 3.1

    340

    341

    335

    343

    Excel 2000

    278

    291

    303

    326

    NaturallySpeaking ® Pref 4.0

    240

    244

    245

    247

    Netscape ® Communicator

    214

    246

    261

    253

    Paradox ® 9.0

    225

    225

    231

    232

    Photoshop ® 5.5

    145

    149

    151

    154

    PowerPoint ® 2000

    276

    307

    313

    309

    Premiere ® 5.1

    290

    282

    283

    291

    Word 2000

    143

    148

    147

    149

    Windows Media Encoder 4.0

    276

    278

    277

    276

    Let’s get graphical:

    S1

    Hmmm. Pretty big jump, isn’t it? Let’s see what caused it.

    S2

    No, not Internet applications.

    S3

    Hmmm, Office Apps.

    Actually, only four applications got a big boost from this.

    S4

    Why Should I Care?

    A few reasons, actually.

    A couple adjustments you might not normally think of can yield improvements in some applications, or
    may not even though you’d think they might.

    Here, the information is fairly useless in real life, since any computing performance gain you’d get from a smaller screen in these
    activities would be more than offset by human productivity loss scrolling around all the time.

    But this demonstrates what I’ve been seeing in more serious tests, you just don’t get increases (or decreases) in performance
    across the board, or even for most applications, when you change a (more serious) parameter. That means if you just look at a single number
    like the overall Sysmark, you will get the wrong idea on what you can expect from a change.

    For instance, later on this series, we’ll look at what extra RAM does for you. If you just look at the single SysMark2000 number, it shows
    about a 4% increase in performance, but most of that 4% comes from ONE application. (We’ll also point out why you can’t necessarily
    rely even on the benchmark for your application, depending on what you do with it.

    The second, more immediate reason why you should care is that if you don’t know what the monitor resolution is in a review, and don’t realize
    it’s important, you might think one board performs better than another when the real difference is just a monitor setting. There was a 7% difference
    in scores due to just that, which is a lot bigger performance difference than we’re seeing between motherboards.

    For instance, the Epox EP-8K7A has been getting a good deal of attention lately. I’ve been able to more or less equalize testing conditions between one review
    of it and the K7Master, and I’m fairly sure the K7Master is a tiny bit slower than the Epox (need to do one other thing to make pretty sure). But if I (or someone less scrupulous) wanted the MSI to “win,”
    all I’d have to do is change the monitor settings, and just not mention it. I’ve seen one case where I suspect the reviewer ran Sysmark at 640X480, which might make some sense for games, but not for most office apps.

    The idea is to explore what changes do or don’t do for you in various apps, which will hopefully advise you on what will help you and what won’t in your own machine and activities.

    What does color depth do to performance? We’ll discuss that tomorrow.

    Email Ed

  • Be the first to comment

    Leave a Reply