There’s some argument about this AMD patch for SysMark2001. Unfortunately, somebody made a mistake, and no one else has noticed, until now.
Here’s the problem:
1) Windows Media Encoder 7 only looks for Intel processors when it decides whether it can use SSE or not. It doesn’t turn on
when there’s an Athlon MP or XP in the machine.
2) SysMark2001 uses Windows Media Encoder 7, so its scores get affected by this.
3) Microsoft will fix this in the next version, but in the meantime, AMD has come up with a patch for reviewers which fixes the problem.
4) How much so? Here’s the problem. Anandtech discusses the SysMark2001 in some detail here.
Now what should have been done at this point was to run SysMark2001 with and without the AMD patch. Anandtech didn’t. What they did instead was run SysMark2001 with and without SSE. The Internet Content Creation score went up 18%.
Anandtech then went back to discussing the SysMark/WME patch as if that were the only reason why the score went up 18%. So did others commenting on this.
That’s not true.
The overall effect of SSE is not the issue here; the effect of the patch is.
The Internet Content Creation portion of SysMark2001 uses five programs:
Photoshop 6
Premiere 6
Dreamweaver 4
Flash 5
Windows Media Encoder 7
Photoshop, Premiere and Flash also use SSE optimizations; it’s not just Windows Media Encoder. As we reported back when the MP came out, SSE optimization helps Photoshop a whole lot, and Premiere significantly (it also helped Windows Media Encoder 4). No patches, no nothing.
So that 18% improvement in Sysmark2001’s Internet Content Creation score isn’t entirely or even mostly due to that patch; it’s due to the other SSE-optimized applications doing their thing without any extra help from AMD.
Now if I had a Palomino, SysMark2001 and the AMD patch, I’d just run the test and throw up the numbers. Since I don’t have any of the three at the moment (though if anybody does have the AMD patch, please send me a copy so I can run some alternate tests),
I can only estimate that the improvement due solely to the AMD patch is probably more like 5% rather than 18% for SysMark2001 Internet Content Creation, and more like 2-3% for the overall SysMark2001 rather than 8%.
So, until somebody runs the numbers, if you look at the chart here, you won’t be more than a couple points off if you just reduce the Athlon XP scores in the first chart by 5%, and those on the second chart by about 3%.
But if you do have all three of these items immediately available, try this:
Run Internet Content Creation without the patch (or if you’ve already patched this, put back the old wmvdmoe.dll file).
Then run Internet Content Creation with the patch.
Report the difference, either to me or on your website.
Let’s get this mistake fixed ASAP.
To Patch Or Not To Patch?
Now that we’ve better established the likely degree of the problem, the question remains, “Should reviewers patch or not?”
There seems to be two issues being argued about on this point:
Is BAPCo evil for not doing the patch itself?
If not evil, is BAPCo’s position reasonable?
First, we have to see what BAPCo’s position is, which we can find here:
“After BAPCo releases a benchmark the applications are not updated and this is done in order to ensure consistent and comparable benchmark results throughout the useful life of the benchmark.”
Now if BAPCo were truly evil, in Intel’s pocket, or whatever else your favorite conspiracy theorist tells you, what they would have done is add application patches which enable SSE2. For instance, Adobe’s done that for Photoshop.
Did they do that? No.
If you look at the SysMark2001 patches, you’ll see that BapCo uses patches to fix the benchmark, not update the applications.
So the policy is even-handed. Nobody gets to tweak the benchmark after it is set. No sign of bias here.
Nor is this a unique position to take. The ZDNet benchmarks do the same thing.
If It’s Not Biased, Is It The Best Approach?
This is a point upon which honorable men can disagree.
Whether it’s BAPCo or ZdNet, the reason why they freeze the application code is to ensure consistency throughout the life of the benchmark (which usually gets updated yearly).
There’s something to be said for that, especially if you value consistency highly.
On the other hand, the world changes quickly. If I have a PIV, I’m sure as hell going to download and install anything that gives me SSE2 enhancements. If Microsoft came out with a Windows Media Encoder patch to enable SSE for my Athlon, I would
sure use that, too.
Any benchmark which doesn’t use the latest and greatest is going to be outdated and thus somewhat misleading for those interested in what they could get NOW from their particular computers.
So we have two audiences: one that wants consistency, and another which wants current accuracy.
Can you possibly satisfy both? I think so.
I think the benchmarking companies could set up two sets of patches. The first set of patches would be the same as they are now, fixes to the benchmarking program.
The second set of patches would simply install any enhancements to the current program set. These should come out at least several times a year. Anybody who valued consistency over current accuracy could get just that by not installing the second set of patches.
Well, Should We Patch For WME Or Not?
My answer to that is “No” for two reasons:
In this particular case, the patch is not publicly available, so the average person looking at the benchmark couldn’t get any benefit from it.
Even if MS provided a patch tomorrow, though, the answer would still be “No” unless all other enhancements to programs made since the the benchmark was released were included.
Regardless of whether you want consistency or current accuracy, you want a level playing field. Either you use all the programs as they stood when the benchmark was released, or you use all the programs as they stand now.
That means if you want to use a patched WME to enable SSE, you have to use a patched Photoshop that includes SSE2, too. Either none of them, or all of them.
Unless, of course, you couldn’t care less about accuracy so long as your side “wins.”
Be the first to comment