The Effects of Benchmarking Bias

[EarthDog]

No worries here bubba. I didn't mean smearing under the guise of intel vs amd. Just intel by itself. I tried to relay the tongue and cheek manner in which I meant that 'smearing' comment with that smiley but ut seems to have not been taken that way. Sorry about that.

Anyway, I ask us all to think about the potential repercussions of this when looking at reviews. Would there be a way to even the playing field? Why hasn't this come out from amd post 09? I can't imagine there isn't at least one soul in amd that isn't constantly checking up on this. I mean this would ne BILLIONS more in fines, no?

 

[Neuromancer]

No I reserved one slot so I'll use the first two posts to list all of the results. Don't take this the wrong way or as hostile or anything but I fail to see the relevance of that.

Edit - I could be wrong but as far as I knew Aida64 was multi-threaded. As for the rest, understand what's going on. I'm using the same hardware, same settings with the ONLY thing changing is the vendor string in CPUID. The rest of the CPUID remains unchanged (i.e the section with the flags that tell the program which accelerators the CPU is capable of using and which ones it can't). A benchmark shouldn't care whether a CPU is a "genuineIntel" or "AuthenticAMD" etc. and that should have no bearing on the results whatsoever. It should be looking at the CPUID flags to determine which accelerators (i.e. SSE etc.) to use. The mere fact that simply changing the vendor ID causes different results should be troubling to all of us. Think about what that means for a second and you'll realize why agencies like the FTC included it in the anti-trust filings. This was supposed to have been fixed but with the first program I test as you can see there are already differences.

I misinterpreted your post and thought you were comparing different systems. Poor reading comprehension on my part.

The only software that uses a multithreaded memory benchmark is sisofts sandra. (MAxxmem2 also has an option for multithreading but it is not the default application used at hwbot)

AIDA64 version 3 will be out this year and will have an updated engine that included multithreaded memory tests.

Looking forward to more results.

No worries here bubba. I didn't mean smearing under the guise of intel vs amd. Just intel by itself. I tried to relay the tongue and cheek manner in which I meant that 'smearing' comment with that smiley but ut seems to have not been taken that way. Sorry about that.

Anyway, I ask us all to think about the potential repercussions of this when looking at reviews. Would there be a way to even the playing field? Why hasn't this come out from amd post 09? I can't imagine there isn't at least one soul in amd that isn't constantly checking up on this. I mean this would ne BILLIONS more in fines, no?

I don't think its smearing if its truth is it? Asking how many chinese kids died making your iPod would be a smear.

Archers post shows the FTC is looking into and could provide private firms with litigation possibilities. I think the issue is not that it "works better" on Intel CPUs, but the fact that it disables extensions like SSE2 and above when a Intel Vendor ID is not found.

Trying to find some more info on it, it looks like the coding change was done after Intel lost an attempt to prevent AMD instituting SSE2 in their clawhammer chips. Since they could not prevent the extensions from being installed because AMD has a license for x86 and extensions they sabotaged the software to prevent them from being used.

WOW Intel Compiler issues back in 2002

Eds article on fairness

 

[Bubba-Hotepp]

Interesting, so with the CPUID set to Intel Atom the PCMark 7 computational score is 464. Interesting indeed.
Using the Intel Xeon (Pentium 4 variety) it comes up as 465.

 

[Bubba-Hotepp]

So I started to try Passmark PT 7 but everytime it hits the 3D section I get an error. So, I opened up my PC's case and took out the 5870 and slapped it onto the Nano's board (talk about overkill). There should be no doubt that this CPU will bottleneck that GPU .

PT 7: As Nano= 316.7, as AMD= Instant BSOD when PT7 is opened, as Intel Atom=333.6. I'll examine the subscores later but that equals a +5% increase in score and is very very close to what you normally see when running PT7 multiple times so hard to call on that one unless I see larger differences in the subscores (I'll go through the screenshots later).

 

[Bubba-Hotepp]

On my To Do List I have, Sisoft Sandra, Cinebench, 3DMark 11, Unengine Heaven, Fritz Chess Benchmark, WPrime & SuperPi (in theory should be zero difference on those since they don't use optimizations). Open to suggestions on anything else....

 

[JigPu]

Very interesting -- I remember the hubbub from when this was going on, and its good to see somebody doing a "check up".

That's where you're wrong. We'll need a programmer to weigh in on this but if I remember correctly (searching in the dim recesses) the software that you run on your computer doesn't actually make specific system or register calls. That's handled by the OS and the BIOS/UEFI. When Intel states "compatible processors" they mean x86, x87, SSE, SSE2, AVX etc. NOT that it's architecturally the same or "compatible".

Figure I should clear this up Executables contain machine code that runs directly on the CPU (ignoring things like DOS, Java, and .NET programs that actually run inside a VM). They contain raw instructions like "add eax, 5" that the CPU loads from memory and executes. The OS is responsible for setting up an environment for executables to run in but aside from those moments the kernel is running (or has handed control over to some other piece of code) the binary is in "full" control of the CPU. (I use quotes because a CPU in Protected mode does impose some limitations, both in what the program can do and how it sees memory).

JigPu

 

[Bubba-Hotepp]

Very interesting -- I remember the hubbub from when this was going on, and its good to see somebody doing a "check up".


Figure I should clear this up Executables contain machine code that runs directly on the CPU (ignoring things like DOS, Java, and .NET programs that actually run inside a VM). They contain raw instructions like "add eax, 5" that the CPU loads from memory and executes. The OS is responsible for setting up an environment for executables to run in but aside from those moments the kernel is running (or has handed control over to some other piece of code) the binary is in "full" control of the CPU. (I use quotes because a CPU in Protected mode does impose some limitations, both in what the program can do and how it sees memory).

JigPu

Thanks, I'm a comms guy not a programmer Can you explain to those that have questions how a "compiler" works? (as in how all of this is relevant)
I've tried my best to complain but (unsurprisingly) am getting hit with "but intel's compiler is for intel cpu's" etc.

 

[Bubba-Hotepp]

Cinebench 11.5:
Via=.27, AMD=.28, Intel=.27, Equal performance and unaffected.

 

[JigPu]

Well, I do kinda agree with the thought that Intel writes their compiler with their own chips in mind. They're "for Intel CPUs" in the sense that Intel knows all the cool tricks to eke out every bit of performance in their chips. Intel shouldn't be penalized for writing a compiler that includes code which says "Oh! This chip is GenuineIntel, Family 6, Model 9, Stepping 2? Jump to this bit of code over here for the best performance". Nor should they be expected to know the inner workings of their competitor's products to the point that their generated code can say "Oh! This chip is an AuthenticAMD, Family 10, Model 3, Stepping 2? Jump to this bit of code over here for the best performance".

That said, ever since the 486, x86 CPUs have had an instruction to identify themselves and the features they support (the 386 can be determined in its own special way). Because of this, even if Intel doesn't know the optimal way to organize code for some chip, they can at least send it off to run along a codepath that uses the same features. However, instead of doing this, the Intel compiler would just punt and tell any unrecognized CPUs to run 386-compatible code that literally any 32-bit x86 chip can handle.

Now, that's poor form on Intel's part (it wouldn't take much to fall back to checking for supported features), but not anything I'd call them out on. What does strike me as a little unfair though is the fact thanks to the 'extended family' field, there's no such thing as an unrecognized Intel CPU. A binary made for Prescott (the first chip with SSE3) will continue to use SSE3 on Ivy Bridge (and ancient x87 FPU on Bulldozer) even though that chip was made eight years after the binary was compiled. Keeping forward-compatible performance is awesome, but using feature detection would also do that and give reasonable performance to their competitors as well.

JigPu

 

[Bubba-Hotepp]

Well, I do kinda agree with the thought that Intel writes their compiler with their own chips in mind. They're "for Intel CPUs" in the sense that Intel knows all the cool tricks to eke out every bit of performance in their chips. Intel shouldn't be penalized for writing a compiler that includes code which says "Oh! This chip is GenuineIntel, Family 6, Model 9, Stepping 2? Jump to this bit of code over here for the best performance". Nor should they be expected to know the inner workings of their competitor's products to the point that their generated code can say "Oh! This chip is an AuthenticAMD, Family 10, Model 3, Stepping 2? Jump to this bit of code over here for the best performance".

That said, ever since the 486, x86 CPUs have had an instruction to identify themselves and the features they support (the 386 can be determined in its own special way). Because of this, even if Intel doesn't know the optimal way to organize code for some chip, they can at least send it off to run along a codepath that uses the same features. However, instead of doing this, the Intel compiler would just punt and tell any unrecognized CPUs to run 386-compatible code that literally any 32-bit x86 chip can handle.

Now, that's poor form on Intel's part (it wouldn't take much to fall back to checking for supported features), but not anything I'd call them out on. What does strike me as a little unfair though is the fact thanks to the 'extended family' field, there's no such thing as an unrecognized Intel CPU. A binary made for Prescott (the first chip with SSE3) will continue to use SSE3 on Ivy Bridge (and ancient x87 FPU on Bulldozer) even though that chip was made eight years after the binary was compiled. Keeping forward-compatible performance is awesome, but using feature detection would also do that and give reasonable performance to their competitors as well.

JigPu

Just finished Sandra and the overall score is almost the same between Nano and Intel. Using AMD caused instant BSOD when benchmark ran.

Jinpu, I agree with you completely. Although where I ding Intel is not in the fact that they optimize the best they can for their own CPU's but how it treated (and in some ways as it looks still treats) their competitors. It would have been a completely different and justifiable story if they had written the compiler and said, "Hey, if you use our compiler it will optimize the best for an Intel CPU. What got them into trouble with the regulators is that they used their position to flood the market with their compilers and libraries, and then when a developer wrote code and chose the option for example "SSE2" it would optimize the code for Intel and then when a "non-Intel" CPU was detected would revert back to x87 etc even IF the "non-Intel" CPU had the CPUID flag signaling it was compatible with SSE2. That's alot different then compiling for SSE2 but for the intel CPU the code path is more efficient or "better". And all of this was done without anyone knowing anything about it but Intel (until they were caught). To me it's kind of like a drug dealer getting people hooked on top grade "crack" and after their hooked switches it out for the cheap cut stuff without telling anyone. (probably a bad analogy but you get the point)

Edit - I'm off to bed. I'll run more tomorrow. I'll probably start drafting up the graphs on Sunday with what I have so far and post those after with the screenshots.

 

[Frakk]

Cinebench 11.5:
Via=.27, AMD=.28, Intel=.27, Equal performance and unaffected.

0.28, my that's a lightning fast CPU your working with there, you should have everything done in as little as 2 weeks.....

Thank's for all your hard work

 

[Archer0915]

Well, I do kinda agree with the thought that Intel writes their compiler with their own chips in mind. They're "for Intel CPUs" in the sense that Intel knows all the cool tricks to eke out every bit of performance in their chips. Intel shouldn't be penalized for writing a compiler that includes code which says "Oh! This chip is GenuineIntel, Family 6, Model 9, Stepping 2? Jump to this bit of code over here for the best performance". Nor should they be expected to know the inner workings of their competitor's products to the point that their generated code can say "Oh! This chip is an AuthenticAMD, Family 10, Model 3, Stepping 2? Jump to this bit of code over here for the best performance".

That said, ever since the 486, x86 CPUs have had an instruction to identify themselves and the features they support (the 386 can be determined in its own special way). Because of this, even if Intel doesn't know the optimal way to organize code for some chip, they can at least send it off to run along a codepath that uses the same features. However, instead of doing this, the Intel compiler would just punt and tell any unrecognized CPUs to run 386-compatible code that literally any 32-bit x86 chip can handle.

Now, that's poor form on Intel's part (it wouldn't take much to fall back to checking for supported features), but not anything I'd call them out on. What does strike me as a little unfair though is the fact thanks to the 'extended family' field, there's no such thing as an unrecognized Intel CPU. A binary made for Prescott (the first chip with SSE3) will continue to use SSE3 on Ivy Bridge (and ancient x87 FPU on Bulldozer) even though that chip was made eight years after the binary was compiled. Keeping forward-compatible performance is awesome, but using feature detection would also do that and give reasonable performance to their competitors as well.

JigPu

Thanks for the replies. I did not feel it was right for me to answer for a couple of reasons and first and foremost it was because I did not want it to turn into an argument.

Again thanks.

Bubba I was joking about the compiler thing. Do you really not know what a compiler does? Irrelevant. ON WITH THE TESTING!

I will say the biggest issues I had in programming classes was getting addresses. Professors spend so much time doing one thing they dont know how to apply everything. IDK how many after class discussions I had.

 

[Neuromancer]

Agners blog was an interesting read.

Didnt get through all of it yet but I bookmarked it for future reading.

 

[Bubba-Hotepp]

I'll get back to testing this afternoon. The wife has been complaining about the around the house "to do" list so I'll be fixing things today

 

[Archer0915]

I'll get back to testing this afternoon. The wife has been complaining about the around the house "to do" list so I'll be fixing things today

Understood! I dont have that issue at my home. I am the man and I run things! I run the Vacuum, Washer, dryer, saws and whatever else she tells me to! I also wear the pants in my family! When she lets me

 

[Bubba-Hotepp]

I'm the KING of my house....no woman tells ME what to do...*looking nervously over shoulder to make sure wife isn't within earshot*

 

[Archer0915]

I'm the KING of my house....no woman tells ME what to do...*looking nervously over shoulder to make sure wife isn't within earshot*

Yeah exactly right!! No I have to go do something! It is my choice! I can do it or hear mouth. I will go!

 

[Frakk]

Every time i treat myself to a new toy mine wants new carpets, some ridiculously expensive shoes, a new kitchen appliance or a whole new kitchen!!!! the more i spend on myself the more expensive her wants are..... no my pockets are not that deep, i wish they were, i would not complain.

Anyway, sorry, back on topic. i just thought i would involve myself as we are moaning about our wives.

 

[freeagent]

And the battle wages on

 

[Archer0915]

And the battle wages on

??? Battle??? We capitulate. We men who let the women rule usually get more with less headache. I rarely say anything or argue or resist much but when my foot goes down that is it. We understand that she is the boss until it is about to hit the fan.