Got quite a bit of comment about the video quality. Over 90% would prefer seeing
measurements of video quality over performance.
Some at least wondered about why it had to be either/or. There’s a very simple reason for putting the question that way:
If I gave you a choice between giving you a scoop of vanilla, a scoop of chocolate, or a scoop each of both, how much would that tell me about the number of vanilla lovers vs. chocolate lovers?
Nothing. Since it doesn’t cost you any more to get both, just about everyone outside of rabid vanilla or chocolate haters would have said “both.”
By making it either/or, I imposed a “cost” on the choice. People would have to “give up” performance data to see quality data.
You might point out that this was hardly a sacrifice since performance data can be found at plenty of other places, and that’s absolutely true.
What I really wanted to see was the number of people who were interested enough to send me an email about it and indicating they wanted to see visual quality data. Certainly the response justified doing this.
Meaningful Data, Not Gratuitous Graphs
When we do this, we aren’t going to refuse to do any performance testing. What we’re going to refuse to do is throw up page after page of meaningless graphs just to fill up space.
Let me give you a few examples:
If Video Card A does 207fps at some low Quake setting, and Video Card B does 210fps, we don’t care. That’s meaningless, and if you think that’s important, you are a numeric fundamentalist and should seek immediate treatment.
If Video Card A does 207fps at some low Quake setting, and Video Card B does 407fps, that may well mean a lot more. Would we mention it? Probably. Would we necessarily give you a graph? Probably not. What’s wrong with saying, “Video Card B is about twice as fast?”
If there’s some new, visually complex game, and Video Card A does 20fps and Video Card B does 45fps, that is significant. If this were a case of 20fps vs. 25fps, that would be significant, too. Playability is always significant (and we’ll have a very liberal definition of what that means).
If I test, or see from other reviews that Video Card A runs 5-10% faster than Video Card B in everything, why do I need fifteen pages of graphs to show that one fact? Why not just say “Video Card A is generally 5-10% faster than Video Card B?”
If you have a problem with that, could you please tell me what insight you’re going to get staring at those fifteen pages of graphs other than “Video Card A is generally 5-10% faster than Video Card B?”
Doubting Thomases Are Alive And Well
Now Thomas . . . was not with them when Jesus came.
So the other disciples told him, “We have seen the Lord.” But he said to them, “Unless I see in his hands the print of the nails, and place my finger in the mark of the nails, and place my hand in his side, I will not believe.”
Eight days later, his disciples were again in the house, and Thomas was with them. The doors were shut, but Jesus came and stood among them, and said, “Peace be with you.”
Then he said to Thomas, “Put your finger here, and see my hands; and put out your hand, and place it in my side; do not be faithless, but believing.”
Thomas answered him, “My Lord and my God!”
Jesus said to him, “Have you believed because you have seen me? Blessed are those who have not seen and yet believe.”—John 20:24-29
Thomas must have had kids. I see his descendants all the time on the Internet. Of course, what they say instead is “Unless I see the exact FPS graphed for every card under every setting for every game placed on the website, I will not believe.”
At least Thomas had an understandable excuse. Believing that somebody rose from the dead after three days is a bit of a stretch.
But video card FPS?
Numeric Fundamentalism
Another way to express the same idea is “numeric fundamentalism.”
If only numbers will do, no matter how insignificant or meaningless or even misleading they may be, and you think that’s always “better,” you are a numeric fundamentalist.
You should shake hands with those Islamic fundamentalists who think that the closer they get to recreating every little detail of the seventh century, the holier they are. You are two birds of a feather, the same belief applied to different beliefs.
If two video cards keep swapping 2% leads, isn’t saying “they’re about the same” good enough?
Let’s say there is a difference. Let’s say Video Card A does 29.7 FPS in an action game and Video Card B does 45.4 FPS. If I said, “Video Card A does 30fps, and Video Card B does 45”, isn’t that good enough?
If your answer is “No,” you are a member of the Digitaliban and need to go elsewhere for your fix.
We are pro-meaningful statistics. We are also pro-, not postliteracy. We don’t write the text just for the fingerobic benefit. We’ll graph when there are significant, important differences. We won’t graph every number we happen to crank out.
How Are We Going To Measure Visual Quality?
Good question. We have some ideas about that, and I’ll talk more about that early next week after I go through all the emails and see what ideas you have on the subject.
Unless we’re talking about huge, even-a-blind-man-could-see differences, we’re inclined to seek objective vs. subjective measurements, but as we said earlier, there’s not a lot of testing software out there.
We’re still in the idea-gathering stage, so if you know of something helpful, please tell us.
Be the first to comment