The Effects of Benchmarking Bias

Bubba-Hotepp said:

I'll work on that tonight.

Some have questioned MaxxMem so I just ran it. Confusing thing about that one, running it with all the regular ID's the results all appear to be within the normal deviations. But when I ran it under the Bubba Hotepp Vendor ID all the results were within normal deviation except latency which suddenly dropped. That has me scratching my head.

Click to expand...

I have no idea how the vendor strings and whatnot you are looking into work, but I do know that the MaxxMEM score does not seem to reflect the real data bandwidth (throughput) when I compare the results from it to the results from other memory benchmark programs. These programs do not show all of the same data, so it is hard to do a full apple to apples comparison. But both AIDA64 and Sandra show some big increases in the results that can be compared directly to the MaxxMEM ones. Below you can see that MaxxMEM says I have 12 Gb/s of throughput, and Sandra says 19 GB/s. In the second screen it has the Sandra results for latency(can't open both benches at once), and you can see that Sandra and AIDA agree that my memory latency is around ~47 ns, and MaxxMEM says 54 ns. Aida says 20,000 MB/s memory copy rate and MaxxMEM says 16,000. Too bad AIDA does not have a bandwidth test, and too bad Sandra does not show Copy/Read/Write speeds. You can get all three of these for free at MajorGeeks, although the free version of AIDA blanks out some of the fields. It would be interesting to see what some Intel rigs will get when they run these same three side by side like I did here. From what I know a decent Intel rig should score around ~20-25 Gb/s on MaxxMEM, which if the same ratio (12gbs/19gbs= ~50% increase)were applied as what it is with an AMD that Intel rig should score 30-37 Gb/s on Sandra, and I know that can't be.




I became aware of this about a year ago when somebody brought it up on OC.Nuthouse. I did see some posts of Intel rigs scoring about the same on all 3 tests, and for AMD rigs you can see for yourself the difference between the scores of these benchmark programs above. I am sure that particular thread was shoved down the memory hole as is the case with anything that might show AMD products in a good light on that site that happens to be plastered with Intel ads from top to bottom. Knowing what I know now about OCN I have to wonder if the "MaxxMEM clubs" and seeing MaxxMEM always being pushed as the memory benchmark program to use by the "helpful" staff is just a coincidence.

Aida64 is showing massive differences in results for memory by as much as 50+%. The "CPUID Vendor String" is the portion of the CPUID that gives a value for the manufactuerer, family and model number" Think of it as the "Make and Model" portion of the CPUID. So it will give a value in binary for "GenuineIntel" or "AuthenticAMD" and family and model number. So for example An AMD "Bobcat" with family of 14 and model 1 stepping 0 in the 0EEX0FMS Hex format (before converting to binary) would be 00500F10. Agners program takes that hex string and put's it into the open registries (in binary) in the Nano CPU for the vendor string. Via was having difficulties with Win XP years ago (before SP3 I believe) where if the CPU wasn't either an Intel or AMD XP would crash. They decided to move the "Vendor String" portion of the CPUID over to a modifiable section of registers in order to get around that and have XP run on their Nano CPU's. Agner Fog wrote a program that uses that ability in the Nano that you can enter a Hex value for a certain CPU and it will load that in the Nano. That allows you to "spoof" the Vendor make and model number of the CPU so that any software doing a query to the CPUID will get a return showing the value that you want. You just have to keep in mind that when the computer is rebooted though it's wiped and it reverts back to the default (Via Nano). Now, the actual capabilities of the Nano remain the same of course. As well, the "instruction set flags" portion of the CPUID which tells programs which "instruction sets" that the CPU supports (i.e. SSE, AVX etc.) remains unchanged as that is "hard coded" in the CPU. So if a program is checking to see who the manufacturer/model number of a CPU it is highly questionable and more likely than not means the program was compiled using Intel's compiler and libraries with the biased CPU dispatcher.

Basically what this allows you to do is to check and see if a program is doing a "check" for who manufactures the CPU. There is nothing in the vendor ID portion of the CPUID that any program needs to know in order to run code. The ONLY programs that have a legitimate reason to query the vendor string are "hardware info" programs such as CPU-Z. Otherwise, EVERYTHING needed is in the portion of the CPUID containing the "instruction set flags", which are the "yes/no" values that tell a program which instruction sets it's capable of running.

What would instruction sets have to do with memory performance though?

EarthDog said:

What would instruction sets have to do with memory performance though?

Click to expand...

I wasn't addressing that part of his question. I was explaining to him about the "Vendor String".

As for your question, it all depends on what code they're using to measure memory performance. You'd have to check out the documentation or ask the developers of MaxxMem if they're using any accelerators in the code. Or decompile and examine the code. I would think that the programmer of a memory benchmark would use just standard x86/x87 code to test memory but then I'm not a programmer and I don't know if there is a good reason to use other instruction sets.

I will say this about MaxxMem2 though, I was completely ready to place that one in the column of programs that have no change no matter which Vendor String is used. And then I watched the latency drop from 149ns to 140ns when using the generic "Bubba Hotepp" Vendor ID. That needs further explaining. Fluke that happened 3 times? Evidence of bias? (but why would it only show up then and not in the other name changes and why just latency and not the other results?) meaningless anamoly? I have no idea at this point but I'd like to have an answer to that one as it has me scratching my head in confusion.

Thanks for the thorough explanation of what "vendor strings" are and how they could relate to performance bias. I don't know why MaxxMEM scores lower for the same tests with AMD CPUS when compared to other benchmarking programs but there is no doubt that it does. I do find MaxxMEM to be a useful tuning tool nonetheless because it gives consistent results and it does this very quickly, making it a good tuning check when you are dialing in your memory.

I must say I really like the tone on this site, I have posted a few MaxxMEM screens to show tuning results, and not a single time did someone jump in and exclaim "Blargh! My Intel scores double that on MaxxMEM, You are an Idiot for buying AMD!" like is the standard procedure on OCN. Not that there aren't folks who are proud of their Intel rigs here and like to let people know , but it isn't the same as the seething Trollpit Bub and I and all of the other AMD owners have put up with at OCiNtell.

I cruised scores for all three of the benchmark programs I used in the example above, and it appears that MaxxMEM does indeed give shooped speed readings for AMD CPUs. The average Intel setup scores around 20 GB/s on both MaxxMEM and Sandra, and an average AMD FX rig will score around 12 Gb/s on MAxxMEM, and up around 18 Gb/s on Sandra. Same thing with the AIDA results that can be directly compared.This is from looking at scores for similar rigs that I could find to compare, but is not the same as having a test showing all three side by side from the same rig. Can someone here running a current Intel rig please run these so we can have a side by side comparison of how these three programs rate Intel setups? I have a P4 out in the Shop, but I don't know if testing on that would have any relevance. I may do it anyways.


Sisoft Sandra Download
MaxxMEM Download
AIDA64 Download Note that the free version of AIDA blanks out some fields. There is a new beta version out, I think it may show all results, not sure on this.

Bubba-Hotepp said:

I wasn't addressing that part of his question. I was explaining to him about the "Vendor String".

As for your question, it all depends on what code they're using to measure memory performance. You'd have to check out the documentation or ask the developers of MaxxMem if they're using any accelerators in the code. Or decompile and examine the code. I would think that the programmer of a memory benchmark would use just standard x86/x87 code to test memory but then I'm not a programmer and I don't know if there is a good reason to use other instruction sets.

I will say this about MaxxMem2 though, I was completely ready to place that one in the column of programs that have no change no matter which Vendor String is used. And then I watched the latency drop from 149ns to 140ns when using the generic "Bubba Hotepp" Vendor ID. That needs further explaining. Fluke that happened 3 times? Evidence of bias? (but why would it only show up then and not in the other name changes and why just latency and not the other results?) meaningless anamoly? I have no idea at this point but I'd like to have an answer to that one as it has me scratching my head in confusion.

Click to expand...

Nor was I addressing that reply directly. Just asking a generic question.

I didnt know the memory speeds would use instruction sets like that to crunch its data. I have no idea the path memory takes, but for some reason, in my completely uneducated thought, it doesnt use those instruction sets to do it.

Perhaps if you emailed the vendor they can let you know?

Jagged_Steel said:

Thanks for the thorough explanation of what "vendor strings" are and how they could relate to performance bias. I don't know why MaxxMEM scores lower for the same tests with AMD CPUS when compared to other benchmarking programs but there is no doubt that it does. I do find MaxxMEM to be a useful tuning tool nonetheless because it gives consistent results and it does this very quickly, making it a good tuning check when you are dialing in your memory.

I must say I really like the tone on this site, I have posted a few MaxxMEM screens to show tuning results, and not a single time did someone jump in and exclaim "Blargh! My Intel scores double that on MaxxMEM, You are an Idiot for buying AMD!" like is the standard procedure on OCN. Not that there aren't folks who are proud of their Intel rigs here and like to let people know , but it isn't the same as the seething Trollpit Bub and I and all of the other AMD owners have put up with at OCiNtell.

I cruised scores for all three of the benchmark programs I used in the example above, and it appears that MaxxMEM does indeed give shooped speed readings for AMD CPUs. The average Intel setup scores around 20 GB/s on both MaxxMEM and Sandra, and an average AMD FX rig will score around 12 Gb/s on MAxxMEM, and up around 18 Gb/s on Sandra. Same thing with the AIDA results that can be directly compared.This is from looking at scores for similar rigs that I could find to compare, but is not the same as having a test showing all three side by side from the same rig. Can someone here running a current Intel rig please run these so we can have a side by side comparison of how these three programs rate Intel setups? I have a P4 out in the Shop, but I don't know if testing on that would have any relevance. I may do it anyways.


Sisoft Sandra Download
MaxxMEM Download
AIDA64 Download Note that the free version of AIDA blanks out some fields. There is a new beta version out, I think it may show all results, not sure on this.

Click to expand...

Glad you like it We grew up round here and the moderation round here does a good job of acting on reported threads. We keep it clean but make no mistake there is plenty of trash talking but at a whole other level. We try to use facts instead of opinion and FanBoiIsm.

Hey my Intel is faster than my AMD! My AMD smokes my A9 tablet.

Other than the tablet I really cant tell! Wait my tab is faster than my ATOM Netbook! Wait I cant prove it; excuse me I need to go beat the crap out of myself to settle this. Any good Android benchmarks out there?

EarthDog said:

Perhaps if you emailed the vendor they can let you know?

Click to expand...

Might be a good idea to Email them and ask them why it is that there software's AMD result are lower than they are on others while there Intel results match those others. And also ask them what compiler they are using.

EarthDog said:

Nor was I addressing that reply directly. Just asking a generic question.

I didnt know the memory speeds would use instruction sets like that to crunch its data. I have no idea the path memory takes, but for some reason, in my completely uneducated thought, it doesnt use those instruction sets to do it.

Perhaps if you emailed the vendor they can let you know?

Click to expand...

Sorry, didn't mean it to come across that way. I was just trying to clear up what I meant. I really don't know if it's using accelerators in the code. It may or may not. I don't know why they would if they are. But I'm sure if it is being used there must be some reason for that I would hope. I'm just very troubled by the fact that all results seem to all be within a very small margin and then at the end the latency drops. Weird and is setting off red flags.

Jagged, prime example. Archer and I were debating for days. Yet I don't think either one of us disrespected the other (as you typically see on OCN, especially from moderators like pioneerisloud and duckieho). And mods like E.D. tend to crack down on people that get too far out of line.

Edit - I have to say that I have seen a few instances of Intel fanboyism, however it wasn't "out of control" and nothing like the attack atmosphere that we were experiencing over there Jagged.

Bubba-Hotepp said:

Sorry, didn't mean it to come across that way. I was just trying to clear up what I meant. I really don't know if it's using accelerators in the code. It may or may not. I don't know why they would if they are. But I'm sure if it is being used there must be some reason for that I would hope. I'm just very troubled by the fact that all results seem to all be within a very small margin and then at the end the latency drops. Weird and is setting off red flags.

Jagged, prime example. Archer and I were debating for days. Yet I don't think either one of us disrespected the other (as you typically see on OCN, especially from moderators like pioneerisloud and duckieho). And mods like E.D. tend to crack down on people that get too far out of line.

Edit - I have to say that I have seen a few instances of Intel fanboyism, however it wasn't "out of control" and nothing like the attack atmosphere that we were experiencing over there Jagged.

Click to expand...

You missed all that fun

Archer0915 said:

You missed all that fun

Click to expand...

Good. I had enough of it on OCN . I have yet to see anyone here use the terms "AMD losers", "AMDfags" "AMD idiots" etc. etc. here. And I'm glad the mods don't let the atmosphere here get to the point where you see such garbage over and over and over.

But anyways, I do want to reiterate this thread isn't about who's CPU's are better or worse. IMHO there are excellent CPU's from BOTH companies at their respective price points. This is more about benchmarks and their reliability.

Bubba-Hotepp said:

Good. I had enough of it on OCN . I have yet to see anyone here use the terms "AMD losers", "AMDfags" "AMD idiots" etc. etc. here. And I'm glad the mods don't let the atmosphere here get to the point where you see such garbage over and over and over.

But anyways, I do want to reiterate this thread isn't about who's CPU's are better or worse. IMHO there are excellent CPU's from BOTH companies at their respective price points. This is more about benchmarks and their reliability.

Click to expand...

Could not agree more about benchmark reliability.

As far as other forums? Many have those issues because they lack content and must turn into a reality TV show to keep up their stats.

Archer0915 said:

Could not agree more about benchmark reliability.

As far as other forums? Many have those issues because they lack content and must turn into a reality TV show to keep up their stats.

Click to expand...

Or have one of them such as OCN with Intel as a sponsor so have an interest in creating such divisiveness.

Bubba-Hotepp said:

Or have one of them such as OCN with Intel as a sponsor so have an interest in creating such divisiveness.

Click to expand...

Yeah but I think we should stop the direct naming as Overclockers has relationships with many of the other forums on the net. We do not want to burn bridges.

Archer0915 said:

Yeah but I think we should stop the direct naming as Overclockers has relationships with many of the other forums on the net. We do not want to burn bridges.

Click to expand...

Jagged and I are just a little jaded after the bad experiences we and many others have had over there recently. Supposedly the forum manager over there is going to look into it after the long message I sent him but we'll see.

Just added the triple run Sisoft Sandra 2012 results to the first post.