Number worshippers pray to the wrong god. –Ed
We’ve talked about AMD’s PR rating, and the Dimwit Dilemna.
Let’s go to the other end of the spectrum. Unfortunately, like politics, groups at opposite
ends of this spectrum start looking alike, too.
Just how does AMD come up with this equivalent? AMD has a real problem here.
The reality is an overall performance comparison is just a technical fiction. If 3D Studio Max is my life, for instance, I want an Athlon. I don’t care about no stinkin’ stats; the Athlon wipes the PIV there.
On the other hand, if Quake is my life, the PIV is very, very good to me.
That being said, any single number purporting to compare two processors is a muddled average that can’t be taken too precisely or exactly as a measure of reality.
It’s like an IQ test. IQ tests are excellent in separating idiots from the normal, or geniuses from the normal. They are not at all good at figuring out the difference when one person is a little smarter than the other.
Due to these variations in results, no matter what standard AMD uses, somebody is going to have a problem with it. Since those paying a lot of attention of benchmarks tend at the very least to be anal-compulsively inclined, an unbelievably impassioned argument will break out, and go on and on and on and on . . . .
I guess if you have no other outlet for your passion, it’s got to go someplace.
The Potter Stewart Standard
Potter Stewart was a Supreme Court justice. In 1964, the Supreme Court heard a case which involved determining whether or not a certain movie was obscene. Here’s what he had to say about it:
“I shall not today attempt further to define . . . {obscenity}; and perhaps I could never succeed in intelligibly doing so. But I know it when I see it, and the motion picture involved in this case is not that.”
To the number-worshipper, this is a terrible way to approach the matter. It’s not objective or scientific at all! We should take calipers onto the human body, and if you expose 91% of your body, that’s not obscene, but 92% is.
Potter Stewart was wiser than that.
There are some things you can measure, and taking that exact measurement literally is useful. There are other things you can measure exactly, but using that measurement anal-retentively is useless or even detrimental. Just because you can measure something exactly doesn’t make that measurement valuable.
Have you ever seen a beautiful member of the opposite sex? An ugly one? How could you tell? I bet you don’t get out your ruler or tape measure for an objective, exact measurement. It’s probably a lot closer to Potter Stewart’s “I know it when I see it.”
That doesn’t mean you (or Potter Stewart) didn’t use some form of benchmarking, but I bet it’s a pretty crude (in more than one sense of the word) benchmark. Probably arbitrary, and way too flexible for consistent readings.
Despite this horrible state of affairs, mankind has somehow managed not to go extinct. This inexact, unscientific tool somehow gets the job done.
I would suggest to you that comparing computers should be conducted more in the spirit of judging a member of the opposite sex than a physics experiment.
That approach doesn’t make benchmarks completely worthless; it just puts them in their proper place. Let’s go back to the IQ test. The bigger the gap, the more useful and accurate the information is. The smaller the gap, the less useful and accurate it is.
When I looked at the proposed AMD PR ratings, they looked fine, even conservative to me. How did I come up with that? In the course of my working, I’ve looked at a whole bunch of benchmarks over a whole range of applications.
Based on that, I have no problem with AMD saying a 1.4Ghz Palomino is the equivalent of a 1.6Ghz PIV. Probably wouldn’t have a problem with them calling it the equivalent of a 1.7GHz PIV. If they went higher than that, then I start looking more closely.
What did I do? I acted like Potter Stewart; I’ll know a bogus rating when I see it.
I’m sure AMD will use some “official” measurement, but in all honesty, if they said instead, “We just looked at some benchmarking, too, and came up with a rough, conservative approximation,” this would not blow my circuits, nor break my heart.
The lack of scientific methodology wouldn’t bother me at all. So long as the number seems reasonable compared to real life, I really don’t care how they came up with it. The second it becomes unreasonable compared to real life, it goes out the window, and I don’t care if it took fifty guys in lab coats fifty days to come up with that number.
A benchmark is an approximate reflection of reality, it is not reality itself. It isn’t and can’t be reality, reality is too diverse to be measured exactly in a single number.
It’s like saying the answer to the Universe is 42. Exact, but useless.
Making The Rubble Bounce
Number-worshippers love trees. Hell, they love the veins on the leafs. What forest?
You’ll see people argue and fighting over a couple percentage points, never considering for a second that the difference they’re treating as life-or-death matters makes no difference in real life.
There’s a word for this: futile.
During the Cold War; huge technological and intellectual resources got thrown into a nuclear competition. To effectively destroy the other side, each side needed to deliver about four hundred weapons to the other side. When that’s the reality of the matter, it doesn’t matter a whole lot if one side has nine thousand and the other side has “only” eighty-five hundred.
Numbers Don’t Count The Uncountable
There are two countries that could effectively destroy the United States with nuclear weapons today, since they can deliver four hundred weapons to the United States, no matter what.
You know one of them. Who’s the other?
Wrong. It’s Great Britain.
Now that you Americans know, are you going to stay up at night worrying about this? Do you think anyone even the most paranoid, Dr. Strangelove guy in the Pentagon worries about this? No, of course not, other factors that can’t be enumerated are more important.
In both cases, the single number can be used to say something, but reality says something else.
Numbers Don’t Kill, People Kill
Benchmarks are like a screwdriver. Used properly as a tool, they can be very hand. Used as a weapon, they can be deadly.
I’m not talking about the benchmarks themselves, that’s a subject for another day.
Rather, I’m talking about the abuse of benchmarks by number-worshippers.
If you take a benchmark as a fairly rough approximation, and realize that small differences make no real difference, that’s the right way to go about it.
It’s regarding the rough approximation of reality as exact Holy Writ that’s abuse. Seeking a perfect, exact general benchmark is like hunting for a snark; you’re looking for something that doesn’t and can’t exist. Anyone saying their benchmark does the trick is claiming to have found the snark.
Estimophobia
Why do people do this?
Some have a political agenda, but I’m not talking about them this time around.
Rather, it’s those who sincerely believe an exact number always has to be better than any approximation or estimate.
Wrong.
An exact number interpreted exactly is important only when two conditions are met:
Neither condition is met by computer benchmarking when interpreted as a precise rule. Let’s go back to the 3DStudio Max/Quake users. Reality gives different results as to which CPU is better. How can you have one number give two different results?
Now that doesn’t make them completely useless, they can be valuable as approximations in situations less extreme than that. They are only useless when considered perfect representations of reality. They aren’t; they just aren’t.
A lot of people, though, take the attitude “Perfect or nothing.” PR has been in the past, and no doubt will be again a real problem for many techies.
I think it’s important to note that the techies aren’t complaining that AMD’s estimates are BS. No, they’re aghast at the notion of some kind of estimate rather than something exact and concrete like MHz.
Well, the concrete exact number doesn’t work here. It can’t work comparing the Athlon apple to the PIV orange. You need an inexact estimate to more exactly measure the phenomena. The phenomena is what’s important, not the ritual.
Do you know what I think the underlying problem for such people is?
They’re afraid. Of themselves. Of their abilities to think about and interpret inexact phenomena in an inexact world. They want technical training wheels.
You see, if you have a holy number, you don’t have to think about it. You don’t have to interpret it. You can end discussion by citing it just like some religious fanatics cited chapter and verse to back themselves up, no matter how out of context.
(The number-worshippers of today are the atheistic descendants of those religious fanatics. Actually, the problem isn’t with religion or computers, they’re just different channels for the same fanaticism. “Convert or die” and “Your side sucks, mine rules” are the same sentiments expressed at different times.)
If you have to think and interpret, you then have to take on the risk of being wrong. All men err, but few accept it.
Numbers are tools, not rules. Mere ritual gives no number authority, only meaningfulness does.
Be the first to comment