6386 SE Speed/Throttling Issue

[Jmtyra]

Calling all AMD experts!

I've got two OEM AMD servers, same HW config, same OS but one is running at a much lower multiplier than the other.
I cannot figure out what is causing the one to run slower than the other (and it's driving me crazy).

Any help or ideas would be appreciated. Hardware info:

• 2x Dell C6145; Dual-node chassis
• Two nodes per chassis; Four nodes total
• Each node: 4x 6386 SE CPUs, 128GB DDR3 RAM
• Each chassis: Rackmount in a DC, enterprise cooling/power
• Redundant PSU, dual power

Work log:
[Will be updated as we try different things]

• Verified BIOS settings are identical
• Checked 'power profile' is the same
• Made sure both were cooling properly (nothing overheating)
• Both are running the same SW producing CPU load

The one that's running at 'full speed' looks like this:


While the one running slower looks like this:


Any ideas?

 

[ATMINSIDE]

What operating system are the servers running?

 

[Jmtyra]

Currently they're running W2k8 R2 Standard. Also tried Ubuntu Server, Debian, W2k8 R2 Datacenter and Arch Linux.

 

[ATMINSIDE]

Currently they're running W2k8 R2 Standard. Also tried Ubuntu Server, Debian, W2k8 R2 Datacenter and Arch Linux.

Interesting.
I had a PII X4 965 that wouldn't clock up, but that was in 2012R2.
The issue for that chip didn't exist in 2008R2.

The motherboard, does it have any option to turn off CnQ?

 

[Jmtyra]

Not that I can see. Here are the available options under advanced CPU settings:

 

[EarthDog]

C1E is a power saving feature...

 

[ATMINSIDE]

Turn off C1E Support and see if the multi comes up.

 

[Jmtyra]

Doh! I should have caught that. Thanks guys, will give that a try.

Also found these with some creative googling, might try this next?
http://support.microsoft.com/kb/2645594
http://support.microsoft.com/kb/2646060

 

[ATMINSIDE]

The Bulldozer hotfixes couldn't hurt for this, I would install them whether C1E fixes it or not.

 

[RGone]

IF we have two supposedly "alike" cpus on two alike motherboards with the same bioses and the OP declares the bioses are set the same? And the same O/S's on the boards and I bet he says they are set-up alike...then why?
RGone...

 

[trents]

I'd be interested to know what "CPU DownCore Mode" is all about. Currently it's on Auto.

 

[ATMINSIDE]

I'd be interested to know what "CPU DownCore Mode" is all about. Currently it's on Auto.

Downcore cuts off cores for compatibility purposes.

 

[Jmtyra]

The Bulldozer hotfixes couldn't hurt for this, I would install them whether C1E fixes it or not.

Understood, will do. Just changed the C1E option and rebooting the box to see. Then I'll proceed with the hotfixes.

IF we have two supposedly "alike" cpus on two alike motherboards with the same bioses and the OP declares the bioses are set the same? And the same O/S's on the boards and I bet he says they are set-up alike...then why?
RGone...

I don't understand your question, could you clarify as to what you'd like to know? Perhaps there is a configuration aspect on which I could elaborate for you?



Update1: No luck with the C1E option. Going to apply the hotfixes and try again.

Update2: *sigh* Is it Monday? Neither hotfix was 'applicable' to my computer. Fresh W2k8 R2 install, so perhaps I'm missing some updates. I'll cycle through Windows Update until it's happy and try again.

 

[ATMINSIDE]

Let me know if the updates allow you to install the hotfixes.
Shame that C1E didn't work...

 

[RGone]

Originally Posted by RGone View Post
IF we have two supposedly "alike" cpus on two alike motherboards with the same bioses and the OP declares the bioses are set the same? And the same O/S's on the boards and I bet he says they are set-up alike...then why?
RGone...

I don't understand your question, could you clarify as to what you'd like to know? Perhaps there is a configuration aspect on which I could elaborate for you?

I said what I said because you are pretty certain the boards and systems are alike. IF that be accurate and you do n0t have some power saving feature ON in the O/S or in the BIOS, then the slow cpu is defective in some manner.
RGone...ster.

 

[Jmtyra]

Let me know if the updates allow you to install the hotfixes.
Shame that C1E didn't work...

Sure! I've got quite a few, shows about 1/2 a gig worth.
Yea...I was really hoping that would do the trick

I said what I said because you are pretty certain the boards and systems are alike. IF that be accurate and you do n0t have some power saving feature ON in the O/S or in the BIOS, then the slow cpu is defective in some manner.
RGone...ster.

Thanks for clarifying. The systems in question are OEM servers (Dell C6145) with quad CPUs per system node, and two nodes per chassis. They were ordered identically and at the same time, so they are indeed exactly the same. That said, you may be on to something, since the two nodes having slowness are in the same chassis. Basically, two nodes in chassis 'A' are fine, while the other two nodes in chassis 'B' are slow.

For the BIOS settings, I pulled up a node out of each chassis, and went line-by-line to make sure they match. Maybe I could have missed something, but they appeared to be the same.

I'll update the original post with more information and the "work log" to keep it easier for everyone to follow (and to keep me from going crazy lol).

 

[ATMINSIDE]

Sure! I've got quite a few, shows about 1/2 a gig worth.
Yea...I was really hoping that would do the trick




Thanks for clarifying. The systems in question are OEM servers (Dell C6145) with quad CPUs per system node, and two nodes per chassis. They were ordered identically and at the same time, so they are indeed exactly the same. That said, you may be on to something, since the two nodes having slowness are in the same chassis. Basically, two nodes in chassis 'A' are fine, while the other two nodes in chassis 'B' are slow.

For the BIOS settings, I pulled up a node out of each chassis, and went line-by-line to make sure they match. Maybe I could have missed something, but they appeared to be the same.

I'll update the original post with more information and the "work log" to keep it easier for everyone to follow (and to keep me from going crazy lol).

Just half a gig?
Won't take TOO terribly long

 

[Jmtyra]

Haha! You'd think so, but it's on our lab network which is reaaaaaaaaly slow.

 

[ATMINSIDE]

Oof, slow networks ftl

 

[Jmtyra]

RGone, your comment got me curious so I started poking around in the chassis-level sensors and information. It looks like there is an issue between the chassis and the nodes.
I'll have to get with the vendor and have them take a look, but at least I have something to point at now.

Update1: Spoke with the vendor; downloading a bootable ISO to grab logs from the nodes and chassis to send back to support. Will provide updates.

Good nodes, showing correct power metrics:



Bad nodes, showing incorrect power metrics: