Update 3/28/01: Most of the criticisms found here are statistical in nature. Due to the nature of what’s being done here, this opens up the possibility of more accurate analysis that simply has not been possible with a user-based database.
So the criticisms are not, “Boy, this really sucks,” but rather, “You can take this to a higher level if you do the following.”
Here’s a letter I received from the principle tester from the Overclockerz Store:
“Hey man I just read your column on the little stepping article we did…
I appreciate your criticism but you didn’t lead to any conclusion to your arguments which were perceived by me at criticizing our testing procedures at OCZ. I for one submitted the material to be published since it merely happened by accident when we received a delivery of CPUS on a Friday afternoon.
I noticed that we had a whole score of chips that were topping out at certain speeds and they were all consistant in behaviors. There is really no way to really explain what behavior until you start unlocking cpus with a conductive pen and testing 50-75 a day. You become very, very sensitive to the booting procedures, multitasking procedures, etc. Passing Prime 95 is one thing but it really doesnt do much if the chip boots slow or has poor tasking. I have seen massive amounts of chips that pass Prime 95, Q3, and 3DMark, yet are barely stable. This is where experience comes in and brings me to conlcuding that every single one of the AXIA and AVIA (the AVIA’s are faster BTW) produced effortless benchmarks and anything else I could throw at it, yet we still took about 50mhz off the top end speeds published in the stepping article.
I just thought I would share my little excursion on chip testing since I do most all of it or at least 99% of it.
-Matthew Kean Magill
The Computer Zone Inc.
“The Overclockerz Store”
Editorial comment: This confirms just who did the testing and how it was conducted.
Per AVIAs being faster than AXIAs, this illustrates the problem we have. The impression I’ve gotten so far is that the AVIAs aren’t quite as good, but it’s close.
However, it’s close enough that I can’t definitively say he’s wrong, and I can’t definitively claim to be right. There’s a ton of overlap here, and there’s certainly cases where one particular AVIA will do better than one particular AXIA.
The closer the “real” difference is, the more samples you need to try to establish that there is a real difference, and the less that difference really means when selecting a single CPU. For instance, (and I’m just making up numbers here) if 40% of AVIAs get over 1450Mhz and 50% of AXIAs get over 1450Mhz, the AXIAs are overall “better” but that’s a fairly poor predictor when you’re buying just one CPU.
Per how good the testing mechanism is, probably the best way to determine those who have bought pretested CPUs whether or not the product met expectations. We’ve set up a thread in our Overclockers Forum under Vendor Discussions so you can comment on this.
AMD is tossing all sorts of combos at us. Figuring out what’s what is going to be a more challenging task than, say Intel sspecs.
The latest attempt at this can be found here.
Give Credit Where Credit Is Due
Here’s a great case of technical accuracy in action. Technical accuracy is true but misleading information.
Here’s what was said (errors corrected):
After seeing the huge amount of speculation about AMD step-codes and what they mean to overclockers we decided to bring some light on to the subject. We will draw our conclusions based on a database of over 1000 AMD CPUs and their stepcodes and their overclocked speeds. The database has been compiled over several months. This we believe is the most accurate manner which to derive results.
The website has an overclocking database, so you might think that’s where the information came from. Nope.
Here’s what would have been a more accurate statement:
After seeing the huge amount of speculation about AMD step-codes and what they mean to overclockers we decided to bring some light on to the subject. We will draw our conclusions based on the Overclockerz Store’s database (and/or some other related reseller(s)) of over 1000 AMD CPUs and their stepcodes and their overclocked speeds. The Overclockerz Store’s (and/or whomever) database has been compiled over several months. This we believe is the most accurate manner which to derive results.
Do you know why I’m so sure about this? Because the AthlonOC.com database doesn’t ask for this information. Checked their whole database, not a mention of codes. Clicked “Add an AMD CPU,” not a word about codes.
To add to that, they state, “The testing platform for all of the chips enter in the database is uniform across the range of CPUs.”
There’s no way any database composed of user results could make that claim.
Why Not Be Honest?
Athlonoc.com has had at least an extraordinarily close relationship with one or more retailers.
In some ways, that can be bad. In some instances, that can be very good. This is one of those instances where it could be very good.
So why deny the obvious? If some reseller offered me the same chance, I’d take it in a flash. I just wouldn’t try to take credit for it.
Advantages and Disadvantages
A retailer can test a number of CPUs under controlled conditions. That can be a big advantage for some purposes.
However, there is much less of an advantage when one does not know what those controlled conditions are. Athlonoc.com doesn’t mention what they are, but the Overclockerz Store states that they use Abit KT7As.
It would be very useful to find out the other details on this testing. What fan/heatsink is being used? What memory? What hard drive? What video card? What operating system do they use? Do they test overclocking by increasing the multiplier, increasing the FSB, or both?
The Overclockerz Store says that the CPUs have been “modified.” Does that just mean a pencil job (I’m pretty sure that’s the case), or is it something else?
When they choose components for testing, do they handpick components that have been proven to work at high PCI speeds, or not?
Just what constitutes a “pass” in these tests? I recall reading some months ago from one of their representatives that a “pass” was running Prime95 for five minutes, then certifying it at a slightly slower speed. Are these scores upon which the averages are based on the highest speed reached?
What is the range of speeds reached by a particular range of CPUs? Do they all clump around the same speed, or is there a significant difference between one and another CPU?
All this information has to be available to those who put this together. The information provided would be much more useful if this additional information were also provided.
If you compare one type of CPU to another on a more-or-less standardized platform, there’s two important pieces of information you need:
1. The number of CPUs used to make the comparison and
2. The range of results
If I’m comparing A to B, and I have fifty processors each, the odds are any conclusions I draw from that comparison will be more accurate and valid than if I only had ten processors each.
Secondly, an average may or may not be a good indicator. If I compare the results of A and B, and in both cases, most of the results come very close to the average, that is very useful. However, if the results from processor A range fairly equally from 1300-1400Mhz, and processor B ranges from 1300-1450Mhz, processor B may come out a little better on average, but that average means a lot less.
These reasons are why I’ve been fairly general about my conclusions. I don’t have results from a standarized platform, and there is a lot of overlap between results from different processors. Some of it is due to differing conditions, some of it is due to inherent differences in the CPUs themselves.
Athlonoc.com can use the Overclockerz Store database to give us a clearer picture of this from a particular angle. I don’t begrudge them that advantage at all, but for the sake of all overclockers, use it. Provide the number of processors tested. Provide ranges so we can see what the level of variation is. This would benefit everybody.
Mighty Mouse Syndrome
This may be a quibble, but calling earlier work on this speculations and acting like you’re Mighty Mouse saving the day is just silly. I’ve been seeing a lot of that lately. 🙂
Well, I’m not Mighty Mouse.
I don’t know whether or not we were the first website to point out the value of reading codes, or noticing the value of AXIAs. I really doubt I was the first individual to notice this, I’m sure somebody else somewhere probably noticed before it ever dawned on me.
Even if we or I were, it wasn’t because of any great genius on our part, it’s because you told us and put us on alert, and we brought it to people’s attention.
From Joe Citarella’s creation of the CPU database, this website has leaned towards mass analysis. Time and circumstances have led us to lean more and more heavily in that direction.
All that we or anybody else are doing is this area is pattern and trend recognition. We may have been the first to apply it a lot to this field, but we didn’t invent the concept, and we certainly didn’t patent it.
We don’t live in a vacuum, not at all. A lot of what we do is gather information from a lot of different sources, and try to put the isolated pieces together. Again, we didn’t invent thinking, and have no patent on this.
If others want to jump on the bandwagon and try to do this better than us, go for it. The more the merrier. Let the best win.
We all build on the work of others. We all build upon the experience of users. If you come up with something that gets the job done better, great.
Using a overclocking reseller database in some ways represents a decided improvement over current methodologies. Not all ways, there are some limitations.
Even as is, this is a bit of a step forward. It can become much more of a step forward if the suggestions we make get adopted, and from there, others and ourselves can move onto the next level.
Competitive cooperation will get us a lot further a lot faster than any other method. Look at science, that’s how they do it. People build upon the discoveries of others, and others build on those discoveries.
Their Conclusions, Our Question
The AthlonOC.com statement claims that the initial coding on the third line of the processor makes a difference in overclocking.
We’ve looked at this, and have real doubts about this based on the information presented so far. We don’t know whether they’re right or wrong on this, and won’t until we get the kind of detail about the number of processors tested and the range of results.
Once we see that, we may well agree with their conclusion. Or we may not, but at least we’ll say why.
What We’ve Been Working On
When Joe Citarella started the CPU database, times were relatively simple back then (or at least we thought they were). It’s gotten a lot more complicated since then. More variables to consider, more variance with methodologies, more variance in the level of skills by the participants submitting data.
At least in our minds, we have become increasingly aware of the level of variance even between identical units.
On the other hand, we know that a lot of you want a simple answer. You don’t want to hear how complicated this is. You don’t want to get a headache poring through tons of raw data.
We know our CPU Database is getting old and creaky. We know others have put together competing databases, and at least one points towards the right general direction. Our challenge is to come up with something better than what we or anybody else has without giving you carpal tunnel in the process.
Trying to reconcile these contrary trends is not so easy, but we have most of the details down, and should have something up in the intermediate future.
This Is Not The Last Word
Using the results for the Overclockerz database, even if all the additional information we suggest be included is, will not be the final word in this field.
The reason for this is that while it’s useful to look at this from one standardized angle, there’s more than one angle at which you can look at this.
Just for one example, does the IWill KK266 (or any other mobo) do better at overclocking than the KT7A? This database won’t tell you that.
Is the IWill (or anything else) a more stable platform than the KT7A? This database won’t answer that, either. Doesn’t even really answer how stable the KT7A is, either, outside of whatever overclocking tests they happen to run.
The point here is not to trash this database, just to point out some limitations to it, limitations future attempts at this need to address.
I’ll be the first to admit that if the improvements I suggest are implemented, the AthlonOC.com database will be very useful, and for a couple purposes, probably will be more useful than anything we or anybody else reliant on user feedback could come up with, never mind have.
However, it doesn’t and can’t address other issues.
Two Tools Are Better Than One
User-furnished and reseller-furnished databases are mirror images of each other. The weaknesses of one are the strengths of the other.
We will have to see how this develops over the next few months. Right now, what’s being presented is a bit useful. It could be a lot more useful if our suggestions were implemented. You should
write them and suggest this. You can send an email here and suggest that they adopt these suggestions if what I said made sense.