My adventures in multi-GPU folding

Well, since there appears to be very little info around concerning multi-GPU folding I thought I'd share my experience.

The official FAQ is here: http://folding.stanford.edu/FAQ-ATI.html

This will be tailored towards ATi cards, as there is currently no nV GPU client available.

[#1]
First you'll need at least two cards (or one card with two GPUs). I have two Sapphire x1950pros, with 512MB each. I'm running them on a P5WD2-P, one at 16x and one at 4x. The X1950pros have internal Crossfire connectors, much like nV's SLI ribbon cables. You don't really have to connect the cables as FaH doesn't work in Crossfire mode, but if you want to use Crossfire for games you can leave them connected full time and just toggle Crossfire on & off using the ATi control panel.



When installing the cards I'd suggest:
- If replacing an older card remove all previous drivers and reboot (Windows should load in standard VGA mode)
- Shut down computer
- Install one card (don't forget power plug), restart, and let windows detect it (but don't install any drivers)
- Shut down computer and install 2nd card, restart, and let windows detect it
- Install latest ATi drivers (and restart)

You should now have four display adapters listed under Device Manager. If you don't try moving your screen plug (DVI/VGA) to the other card to force Windows to detect it. You can also use the "Detect Displays" button on the "Display Manager" page of ATi's CCC.

[#2]
On your hard drive make two folders - I call mine FaH1 and FaH2.

[#3]
Now you'll need to download the FaH GPU client (console version).
http://folding.stanford.edu/download.html

[#4]
Place a copy of the downloaded client into each folder that you just made.

[#5]
Open a command prompt and change to the first folder you made (ie: cd\FaH1).

Type "FAH5.91beta3-console.exe -local -configonly"
(without the quotes)

Note that the executable name will change as the client is updated over time. Just type "dir" if you can't remember what you saved it as.

Go through the configuration questions (team 32 )

[#6]
Change to the 2nd folder (ie: cd\FaH2) and do the same thing to set up the second client. Remember to use a different machine ID when it asks.

You can close the command prompt when you are done.

[#7]
Browse to the first folder (FaH1) using Windows Explorer.

Create a shortcut to the executable (FAH5.91beta3-console.exe or whatever). Rename the shortcut to "GPU0" for simplicity. Alt-click on the shortcut and go into the properties. On the "Shortcut" tab, in the "Target:" field, you want to add "- gpu 0 -local" (without quotes) after the executable file name. It should look something like: FAH5.91beta3-console.exe -gpu 0 -local

This will force the client to use GPU 0, and the local configuration file.

[#8]
Try launching the shortcut. FaH should run. If it exits and gives you an error about not finding the GPU then change the "- gpu 0" option in the shortcut properties to "-gpu 1". If that doesn't work try 2...

Windows labeled my first GPU #0, and I think it should for everybody.

When you get it working leave it running for now.

[#9]
Now browse to your other FaH folder (FaH2) and create a shortcut just like we did for FaH1. Rename it to "GPU1". Now the "-gpu" option needs to be different then it was for FaH1. If -gpu 0 worked for the first one try -gpu 1, then 2, then 3. Don't foget the -local part.

Since you will only have four GPU outputs (two primary, two secondary) you're just trying to figure out what IDs Windows gave to the two primaries. Unless you have more then two cards (or onboard). Then you'll have more.

Mine were #0 and #1.

If you get a Machine ID error then you probably didn't set a different machine ID in step #6.

[#10]
Now you should have two clients working

Observations

My system:
- Asus P5WD2-P
- i945D @ 17x 250FSB = 4.25ghz
- 2x 1GB @416mhz 4-4-4-12
- 2x Sapphire X1950Pro 512MB

[#1]
Some people are reporting that the second client runs considerably slower then the first. Mine seem to run at virtually the same speed.

Heatsinks temps (measured with non-contact infrared thermometer) and power draw (measured with current clamp on six pin PCIe connectors) indicate similar GPU loads.

So runnning one card in a 4x slot doesn't seem to make any difference (I find the same results in games BTW).

[#2]
Each client seems to use aprox 50%~60% of one CPU core (25%~30% of total). If I try to run the SMP client as well everything gets really slow.

[#3]
ATi Tray Tools seems to work for overclocking. The sliders change both cards simultaneously, so watch out. Memory speed seems to have a much bigger impact on performance then GPU speed.

The cards seem to default to 2D speeds, unless you overclock them. A 9% GPU ("580" -> 631 mhz) increase and a 4% mem ("702" -> 729 mhz) increase gives me a 20~25% increase in folding (from 11 mins per result thingy to 9 mins) which doesn't sound right. So the cards must really be going slower then 580/702 at default speeds.

nice little write up , I have been wondering about this , and also thinking of getting a few more crossfire boards. Then I dont have to buy any DDR2 for a while

In my experience, memory speed makes very little difference and GPU speed scales linearly on a single GPU fed by a dedicated core. A single X1950 Pro should be turning 8 min/frame if running at stock GPU speed fed by a dedicated core. The 11 minute frames indicate something unrelated to clock speed was slowing the clients. 9 minutes/frame might be as good as it gets with two of them running.
The 1950 Pro does not have a 2D mode, it runs full speed all the time.

Nice "How to" may be of use to me in the future when SMP goes away.

EDIT Let me rephrase that, when and if the the bonus's for SMP go away

To change the core/mem speeds all I'm doing is closing down the two clients, adjusting and applying the speeds, and then restarting the clients.

After a bit of research/playing I believe my cards run 500/600 normally, and then ramp up to 580/700 when a 3D app is detected.

500 -> 630 = 26% increase in GPU speed, which is virtually the same as the ~25% increase in folding speed that I see.

I'll keep playing with it.

dfonda said:

... when SMP goes away.

Click to expand...

!?!?!?!

My saphire X1950 Pro AGP runs 580/702 all the time. I'm fairly certain the PCIe model is the same.

Card in my sig is running 708/754 (I'm pretty sure thats it) 24/7 folding. Right now, with fan on 80% I'm getting temps just below 80 deg C. I need to give the HS a good clean again, I think.

Just got my multi-GPU running this evening. It seems to be taking around 10 minutes per frame. It has been a bear to get it set up, but it seems to be pretty stable so far. Hopefully my first WU's that get submitted will show that it was worth the trouble