No Black Box Here
There’s been a lot of wailing and gnashing of teeth about benchmarks being biased, and the solution to all that is some sort of open-source benchmarch.
This discussion has not been helped by a great deal of ignorance and confusion about these programs do and how. It has been claimed that programs like SysMark are “black box” programs where you can’t tell what’s being run. Because of that, we need open-source programs.
I’m sorry, but if “open” in this context means “being able to figure out what the program is doing,” SysMark2000 is an “open” program.
These programs run scripts. Whatever it does is shown on the screen. Even without a script, you can more-or-less figure out with
some time, patience, and a much slowed down machine, just what is being run.
However, at least with SysMark2000, you can just look at the script. Just open up the relevant PCD file with Notepad, and at the bottom of the file, there it is. Just what’s under the hood. You see what the program does, in English.
Maybe not the easiest thing in the world to figure out, but certainly a lot easier than reading “open-source” C++ code.
This is no “black box.”
If you think the choices Sysmark made stunk, fine. You then have the burden of proof of showing why they stink (and that’s what I’m looking at right now).
But it’s not like it’s impossible to figure out what the choices are. That’s no reason to say you need open source programs that run completely different activities.
I don’t have SysMark2001, but I bet a little digging around will find the same thing. That will be a more complicated script, with applications being run concurrently, but there should still be the same kind of script.
A Better Solution
Let’s assume, for argument’s sake, that BapCo or ZdNet or whomever are spawn of Satan or Santa (Clara, whichever spelling you prefer), and even Rambus finds what’s going on lower than whale feces.
What’s the solution?
It sounds to me some want to say, “We must not use their biased program, instead, we must make up our own (biased-the-“right”-way) programs.”
Bias is in the eyes of the beholder. If I want to know how Photoshop typically does, the answer to a biased script is not some program that has nothing to do with what I want to test and which may well have biases buried inside code that 99.999% of any users will never comprehend. That kind of “open-source” makes Windows looks like Linux in comparison.
No, what I need is an better Photoshop script.
We don’t need open source programs to solve this particular problem; we need open-source scripts.
I see two ways of doing that, the comprehensive approach, and the down-and-dirty.
The comprehensive approach would be to create a program with plug-ins that would incorporate open scripts, in any combination you like. A problem with that would be the (practical) requirement that you’d have the programs being tested, but that might not be so bad for most people.
Actually CSA Research requires just that with their benchmark program. It benchmarks using Office, and you need a copy of Office for it to work.
The down-and-dirty approach would be to simply modify the existing scripts to run a more appropriate series of actions. There may be some legal problems associated with modifying and then publicly displaying the results, in which case the fallback is option one, but at least it could allow for some quick experimentation and tests.
If there are those among us who can modify BIOSes and crack software protection, surely this would not be more difficult.
Either way, this strikes me as being a far better and simpler approach to solve the problem of bias, plus give the fringe benefit of much more testing flexibility more appropriate to those who want to test using applications.
I don’t see how anyone could reasonably object to this approach.
Be the first to comment