[Bubba-Hotepp]

Let me begin by saying this, this thread is not meant to be an Intel "bashing" or "smearing" thread. I won't be getting into the merit's of one companies CPU's over another. This thread is about the benchmarks that all of us use all the time to measure the performance of our CPU's and if there is any "bias" in the results. I think that it's in all of our best interests to answer that question either way. The following is merely an explanation of past events and how they may or may not be affecting current programs.

It's been well documented and gone over about how Intel's compiler and Libraries have a biased CPU dispatcher which negatively affects Intel's competitors largely brought to everyone's attention by a man named Agner Fog. That same dispatcher resulted in being part of AMD's lawsuit against Intel as well as being part of the numerous anti-trust filings and fines levied against intel in the U.S., the E.U. as well as other countries. As part of the Federal Trade Commission's suit against and settlement with Intel it was required that Intel post a "disclaimer" for it's compiler as well as reimburse any developers who used the program for the cost as well as their costs in recoding their programs with a different compiler etc. Part of AMD's settlement with Intel was that they remove the compromised code and replace it with one that does not discriminate based on the vendor string in the CPUID (i.e. AuthenticAMD etc.). Intel later stated that their compiler no longer discriminates (yet the CPU dispatcher code is still there) but has remained silent as to all of their libraries that had the same CPU dispatcher coded in as well. You can read about all of it here at agner's blog as well as by doing a google search for the lawsuit and anti-trust actions. http://www.agner.org/optimize/blog/read.php?i=49

With all of that in mind and a good amount of curiosity I set out to try and see if benchmark programs are still affected by this. At first I tried using VMWare and altering the CPUID in the config file as others in the past have done with some success but unfortunately nothing I tried worked (that function was removed maybe?). So I broke down and purchased a Via Nano Motherboard, downloaded Agner's program and started to get to work to see if there is any bias in the programs we trust to give us an idea of our CPU's capabilities as well as the programs and games we use all the time. With that in mind let's take a look at the results. All results within 3% of each other should be considered equal as that is within the margin for error.

I'm still in the process of testing so these results aren't final. I'll update the charts as I go and screenshots when I'm done with everything as I have literally hundreds of screenshots already to go through and upload.

*NOTE - If the results do not show any difference no matter what the "Vendor ID" is set to (normal 1 point or small variations less than 3% etc) I will only run the tests once as there is no need to do triple runs and average the results. I will also, in the interests of saving bandwidth not upload and post the screenshots for those tests although I'm thinking of a way to have a bulk link to everything instead of posting individual screenshots.

Aida 64 Extreme Edition

*with Vendor ID set to Intel and "Bubba Hotep Aida 64 would not run when selecting memory tests.





Cinebench 11.5




Conclusion: All results within normal deviation between runs.



Passmark Performance Test 7

*With Vendor String set to AMD, PT7 instantly crashes when opened.





PCMark 7

*Note all results, higher = better





PC Wizard 2012 Benchmark

This test could not be completed due to consistent random BSOD crashes when changing the Vendor string.



NovaBench





Sisoft Sandra 2012




MaxxMem^2




3DMark 11

[Bubba-Hotepp]

Reserved

[Neuromancer]

AIDA64 is a single threaded benchmark which means (for some reason) it does not score past dual channel ram.

If you need me to delete this post for reserved I will.

[Bubba-Hotepp]

Originally Posted by Neuromancer

AIDA64 is a single threaded benchmark which means (for some reason) it does not score past dual channel ram.

If you need me to delete this post for reserved I will.

No I reserved one slot so I'll use the first two posts to list all of the results. Don't take this the wrong way or as hostile or anything but I fail to see the relevance of that.

Edit - I could be wrong but as far as I knew Aida64 was multi-threaded. As for the rest, understand what's going on. I'm using the same hardware, same settings with the ONLY thing changing is the vendor string in CPUID. The rest of the CPUID remains unchanged (i.e the section with the flags that tell the program which accelerators the CPU is capable of using and which ones it can't). A benchmark shouldn't care whether a CPU is a "genuineIntel" or "AuthenticAMD" etc. and that should have no bearing on the results whatsoever. It should be looking at the CPUID flags to determine which accelerators (i.e. SSE etc.) to use. The mere fact that simply changing the vendor ID causes different results should be troubling to all of us. Think about what that means for a second and you'll realize why agencies like the FTC included it in the anti-trust filings. This was supposed to have been fixed but with the first program I test as you can see there are already differences.

[EarthDog]

[Bubba-Hotepp]

Originally Posted by EarthDog

Earthdog, give me tips on how to post pics so that I can have multiple pics in a link and not a very very long post showing each and ever pic. Or is there a spoiler function in this forum?

[EarthDog]

You can use the spoiler function...otherwise, Im not sure, sorry.

[Bubba-Hotepp]

Maybe I'm missing something obvious but I don't see the spoiler function button in the editor.

[EarthDog]

No button, need to add manually.

[spoiler]YADA![*/spoiler] (remove the asterisk)

Spoiler:

YADA!

[Archer0915]

Originally Posted by Bubba-Hotepp

Edit - I could be wrong but as far as I knew Aida64 was multi-threaded. As for the rest, understand what's going on. I'm using the same hardware, same settings with the ONLY thing changing is the vendor string in CPUID. The rest of the CPUID remains unchanged (i.e the section with the flags that tell the program which accelerators the CPU is capable of using and which ones it can't). A benchmark shouldn't care whether a CPU is a "genuineIntel" or "AuthenticAMD" etc. and that should have no bearing on the results whatsoever. It should be looking at the CPUID flags to determine which accelerators (i.e. SSE etc.) to use. The mere fact that simply changing the vendor ID causes different results should be troubling to all of us. Think about what that means for a second and you'll realize why agencies like the FTC included it in the anti-trust filings. This was supposed to have been fixed but with the first program I test as you can see there are already differences.

Now if I tell a program that is is working with one CPU instead of the other and I am using the same CPU for all it does make a difference as it pertains to registers and system calls does it not?

Perhaps I am missing something here but to tell a program that an Intel is and AMD and an AMD is an Intel is not a legitimate test because they are different. Registers are not the same and how the program approaches functions would differ would they not? I see what you are doing but I just do not see it working out the way you plan.

[EarthDog]

I know you probably dont want to be answering this right now bubba... but Im just tossing this out there to chew on eventually.

Regarding the AIDA64 (memory) testing. I would also imagine that due to the architectures being so much different within the CPU's and the IMC's, that you cant compare cross platform and confidently say that it was the coding in the application that caused the large discrepancy... can you? Why wouldn't the method that each maker uses to access/process memory, both R and W, be the difference?

I guess my question or what I dont understand, is how does anyone know that the code is the problem when your testing two completely things that handle the same thing (memory for example) in different ways to begin with? CPUID may be the answer, but that wasnt done (right?). Is this what the VIA chip is allowing you to do? You are picking a 'neutral' CPU and changing the ID with it?

[Bubba-Hotepp]

Originally Posted by Archer0915

Now if I tell a program that is is working with one CPU instead of the other and I am using the same CPU for all it does make a difference as it pertains to registers and system calls does it not?

Perhaps I am missing something here but to tell a program that an Intel is and AMD and an AMD is an Intel is not a legitimate test because they are different. Registers are not the same and how the program approaches functions would differ would they not? I see what you are doing but I just do not see it working out the way you plan.

That's actually incorrect. You're confusing the "vendor ID" portion of the CPUID with the rest which tells the OS and other programs how the CPU functions. The actual "name" of the manufacturer shouldn't matter at all.

Quote from Agner on his blog

However, the Intel CPU dispatcher does not only check which instruction set is supported by the CPU, it also checks the vendor ID string. If the vendor string says "GenuineIntel" then it uses the optimal code path. If the CPU is not from Intel then, in most cases, it will run the slowest possible version of the code, even if the CPU is fully compatible with a better version.

The vendor ID string is the ONLY portion of the CPUID that you're able to change on a VIA Nano processor, NOT the rest of it which tells the software which instruction sets it supports or doesn't.

Here's one article on it -

http://www.theinquirer.net/inquirer/...977=obinsource

Here's an article in 2010 (shortly after the FTC accepted a settlement from Intel for among other things an "unfair" compiler) where they did the same thing that I'm doing now -

http://ixbtlabs.com/articles3/cpu/vi...d-fake-p1.html

Here's the text of the FTC, Intel Settlement agreement -

http://download.intel.com/pressroom/..._Agreement.pdf

Quote -

Spoiler:

A. Within ninety (90) days of the date on which this Order becomes effective,
Respondent shall Clearly and Prominently inform its Compiler Customers on its
web site, documentation, and compiler presentations that relate to compiler
performance or optimizations that:
1. Intel’s Compiler may or may not optimize to the same degree for non-Intel
microprocessors for optimizations that are not unique to Intel
microprocessors. These optimizations include SSE2, SSE3, and SSSE3
instruction sets and other optimizations. Intel does not guarantee the
availability, functionality, or effectiveness of any optimization on
microprocessors not manufactured by Intel. Microprocessor-dependent
optimizations in this product are intended for use with Intel
microprocessors.
B. Respondent shall not misrepresent, or assist others in misrepresenting, expressly
or by implication, the level of optimizations available in its Compilers for
Compatible x86 Microprocessor.
C. By the time of the next Compiler release, including update releases, but no later
than six (6) months from the date on which this Order becomes final and on an
ongoing basis, Respondent shall Clearly and Prominently provide the following
disclosures in its product documentation (whether in paper form or on an internet
site), marketing literature, and promotional literature, where optimizations are
discussed, including but not limited to any descriptions of compiler optimization
options such as those in user manual tables or in descriptions of library
dispatching mechanisms.
1. If an Intel Compiler optimizes for any Intel x86 Microprocessor for
instruction sets that are common to Compatible x86 Microprocessors, such
as SSE, SSE2, SSE3, and SSSE3 instruction sets, but does not do so
equally for Compatible x86 Microprocessors, Intel must Clearly and
Prominently disclose that fact, including identifying the specific
instruction sets implicated.
2. If other optimizations which could run on both Intel x86 Microprocessor
and Compatible x86 Microprocessors are reserved for Intel x86
Microprocessors, Respondent must Clearly and Prominently disclose that
optimizations not specific to Intel microarchitecture are reserved for Intel
x86 Microprocessors.
D. Within ninety (90) days of the date on which this Order becomes final,
Respondent shall implement, and shall notify its Compiler Customers that it has
implemented, a program to reimburse Compiler Customers who (i) have
detrimentally relied on Intel representations as to Compiler availability,
functionality or effectiveness when using an Intel Compiler to compile code to be
executed on a Compatible x86 Microprocessor and (ii) decide to recompile using
a Compiler not developed or sold by Respondent (the “Intel Compiler
16
Reimbursement Program”). Such a notification must include a link to or a copy
of this Order and specifically reference this Section VII of the Order in the
notification. The features of the Intel Compiler Reimbursement Program shall
include the following:
1. Reimbursement shall be made based upon documented costs of such
recompilation (including without limitation testing, distribution, or direct
communications with customers) provided by the customer;
2. Respondent’s total obligation to provide reimbursements under this
section shall not exceed ten (10) million dollars;
3. Respondent shall hold all applications to the Compiler Reimbursement
Fund for six (6) months after Respondent’s notification to customers of
the Compiler Reimbursement Program. If requests for reimbursement that
comply with the requirements of Section VII.D of the Order received in
the first six (6) months after Respondent’s notification to customers of the
Compiler Reimbursement Program exceed ten (10) million dollars,
customers shall be reimbursed from the Compiler Reimbursement
Program on a pro rata basis.
4. All requests for reimbursement from the Compiler Reimbursement Fund
that comply with Section VII.D of the Order shall otherwise be reimbursed
on a first-come first-served basis until the fund is exhausted;
5. Respondent may condition reimbursement upon receipt of a declaration
from the customer asserting that it has relied upon Respondent’s
representations, describing the representations upon which the customer
relied, and attesting to the accuracy of and basis for the recompilation
reimbursement amount requested;
6. Respondent may condition reimbursement upon a release of claims by the
customer for any damages or other relief relating to Respondent’s
representations or to the recompilation; and
7. Respondent may terminate the program once ten (10) million dollars has
been reimbursed to customers under the program or two (2) years after
announcement of the program, whichever comes first.
E. Respondent shall not represent, in any manner, expressly or by implication, that
its Compiler provides the same or superior performance than any other competing
Compiler unless the representation is true and non-misleading, and, at the time of
making such representation, Respondent possesses and relies upon competent and
reliable evidence sufficient to substantiate that the representation is true. For
purposes of this Part, competent and reliable evidence means tests, analyses,
research, or studies that have been conducted and evaluated in an objective
manner by qualified persons and are generally accepted in the profession to yield
accurate and reliable results.

[Bubba-Hotepp]

Originally Posted by EarthDog

I know you probably dont want to be answering this right now bubba... but Im just tossing this out there to chew on eventually.

Regarding the AIDA64 (memory) testing. I would also imagine that due to the architectures being so much different within the CPU's and the IMC's, that you cant compare cross platform and confidently say that it was the coding in the application that caused the large discrepancy... can you? Why wouldn't the method that each maker uses to access/process memory, both R and W, be the difference?

I guess my question or what I dont understand, is how does anyone know that the code is the problem when your testing two completely things that handle the same thing (memory for example) in different ways to begin with? CPUID may be the answer, but that wasnt done (right?). Is this what the VIA chip is allowing you to do? You are picking a 'neutral' CPU and changing the ID with it?

That's the thing, the "Vendor ID String" has nothing to do with how the CPU performs. With that completely removed the CPU is supposed to perform the same because the rest of the CPUID is what tells the software about the capabilities of the CPU. Changing the Vendor ID changes none of that and is not supposed to have a performance impact at all. That is precisely why the courts found in favor of AMD in that portion of the lawsuit (among the rest), and that is precisely why the FTC included that in the anti-trust suit against intel back in 2009 and added certain future requirements for intel to comply with.

Let me sum up what's going on and how it works. Take the open source compilers for example. When writing the code path it will look to the CPUID for certain "flags" that tell it which optimizations it's compatible with be it SSE, SSE2, 3DNow (not used anymore) etc. If the CPU tells it that it's compatible it writes a code path to use that optimization. With Intel's compiler and libraries, sometime back around 2005 they inserted code that FIRST looks for the vendor ID BEFORE checking for the optimization flags. So let's say you are coding a program and you want it to take advantage of for example SSE2. You choose that option in the compiler. Now the program is compiled by both the "open source" compiler and Intel's compiler. For the open source compiler it will look for the flag in the CPUID which shows compatibility for SSE2 and if it's there it will utilize that code path. If it's the Intel Compiler (as it was proven prior to 2010) it would FIRST look for the vendor ID. If it came back as "genuineIntel" then it would look for the SSE2 flag which if it finds then chooses the SSE2 code path. IF however it comes back as anything but "genuineIntel" it would stop at that point and choose the non-optimized code path regardless of whether or not the SSE2 compatibility flag is present or not. It wouldn't even check. Now Intel has stated that in it's "compiler" they've disabled this function and there are some who've said they have checked it and it doesn't affect performance anymore. But here's the thing, Intel has NOT stated whether they've disabled that function in their various libraries where it was also found. I find that interesting considering that the "libraries" aren't mentioned in the FTC settlement as well as the AMD v. Intel settlement, just the compiler itself. Now clearly there is something going on with some of the benchmarks like AIDA64. Either they are STILL using an older, "unpatched" version of the compiler (agner fog and others have posted "fixes" online to patch the compiler to disable the vendor check) OR the libraries are STILL using the CPU dispatcher with the vendor ID check coded in.

Edit - Another possibility is that the programmers who wrote "Aida" themselves inserted code that behaves differently depending on the CPU Vendor ID. Although I think that is the most unlikely scenario.

[Archer0915]

Originally Posted by EarthDog

I know you probably dont want to be answering this right now bubba... but Im just tossing this out there to chew on eventually.

Regarding the AIDA64 (memory) testing. I would also imagine that due to the architectures being so much different within the CPU's and the IMC's, that you cant compare cross platform and confidently say that it was the coding in the application that caused the large discrepancy... can you? Why wouldn't the method that each maker uses to access/process memory, both R and W, be the difference?

I guess my question or what I dont understand, is how does anyone know that the code is the problem when your testing two completely things that handle the same thing (memory for example) in different ways to begin with? CPUID may be the answer, but that wasnt done (right?). Is this what the VIA chip is allowing you to do? You are picking a 'neutral' CPU and changing the ID with it?

And let us not forget the AVX instructions in which AMD FX cpus have tested slower on. The fact is Intel made the compiler for Intel chips. So unless your chip is exactly the same as an Intel chip you will not bench the same as in intel chip. As far as I know the clones stopped with i486.

I think the thing to do if you really want to see what is going on is get benchmarks from the days when AMD was ruling the roost. If the Intel's come out with a lead that matches modern testing software percentages then the testing being done here is just an exercise and nothing more.

[Archer0915]

You know I actually went through all of this in a GD thread a few years back. Intel compilers are for Intel processors. It is like saying the PhysX should run on AMD cards. Nobody is forcing software vendors to use the Intel Compiler.

[Frakk]

I would think that if a program achieves better performance by thinking its an Intel CPU than it does while thinking its an AMD CPU, then its not using the CPU to its full potential unless it thinks its an Intel CPU.

It would not achieve that better performance if it did not have the components or horsepower to begin with.

You can not make a shopping cart go like a Porsche unless it has the horsepower.

I would be very interested to see all your results Bubba-Hotepp.

[Archer0915]

By the way I do look forward to seeing the test results.

[Bubba-Hotepp]

Originally Posted by Archer0915

And let us not forget the AVX instructions in which AMD FX cpus have tested slower on. The fact is Intel made the compiler for Intel chips. So unless your chip is exactly the same as an Intel chip you will not bench the same as in intel chip. As far as I know the clones stopped with i486.

I think the thing to do if you really want to see what is going on is get benchmarks from the days when AMD was ruling the roost. If the Intel's come out with a lead that matches modern testing software percentages then the testing being done here is just an exercise and nothing more.

Completely wrong. Intel's compiler is NOT "for Intel CPU's". It was never advertised as such and was never intended to be so. What they did was create a compiler, offer it dirt cheap or even free to software developers who widely considered it one of the better compilers out there. THEN Intel secretly added the vendor ID check to the compilers code. It's evidenced through the fact that it's prior compilers didn't "cripple" functionality in either VIA or AMD CPU's. In fact if what you're saying is true then explain this:

http://www.ftc.gov/opa/2009/12/intel.shtm

In addition, allegedly, Intel secretly redesigned key software, known as a compiler, in a way that deliberately stunted the performance of competitors’ CPU chips. Intel told its customers and the public that software performed better on Intel CPUs than on competitors’ CPUs, but the company deceived them by failing to disclose that these differences were due largely or entirely to Intel’s compiler design.

Now explain to me how programs like Everest Ultimate, PCMark05, Aida64 etc. which are BY DESIGN meant to test the perfromance of CPU's from Intel AND AMD are compromised by as you put it an "intel only" compiler?

[Archer0915]

Let me look because I do not believe it is the case. From what I remember the compiler does look at the ID string and if Intel runs the Intel optimized and if not it runs the stuff that will run on anything.

[Bubba-Hotepp]

Originally Posted by Frakk

I would think that if a program achieves better performance by thinking its an Intel CPU than it does while thinking its an AMD CPU, then its not using the CPU to its full potential unless it thinks its an Intel CPU.

It would not achieve that better performance if it did not have the components or horsepower to begin with.

You can not make a shopping cart go like a Porsche unless it has the horsepower.

I would be very interested to see all your results Bubba-Hotepp.

Exactly. You'll have to give me a few days at least to compile everything together and coherent enough to post. I'll have to run each benchmark or program 3 times (as via, AMD and intel). I posted the Aida results just as an example of what I'm doing.