• Welcome to Overclockers Forums! Join us to reply in threads, receive reduced ads, and to customize your site experience!

6 GPU Folding Rig Build

Overclockers is supported by our readers. When you click a link to make a purchase, we may earn a commission. Learn More.

superducky

Team 32 Folding Member
Joined
Jul 24, 2006
That x8 to x16 jumper is just following a pretty standard form factor for risers and there are no traces in it, you're good to modify it as necessary.
 
OP
torin3

torin3

Member
Joined
Dec 25, 2004
Well, I'm going to hold off for now. Looks like I can't boot the board with 2 GPUs plugged into the MB at one time. Slot position doesn't make a difference, and the BIOS settings don't help either.

So, I'll put the old MB in this case, and I can probably get 3 cards running on it while I try and get the parts for a 6/7/8 PCI-E board that will work with multiple video cards.

Key thing seems to be if they have latches to lock the card in, you can probably use multiple GPUs on it.
 

TerranBrackiatt

Member
Joined
Nov 1, 2009
Location
Austin, TX
yeah it's crazy, even though Nvidia no longer supports SLI, the motherboard manufacturers are bent on mentioning whether or not THEY support SLI on their boards. which i'm glad they do. their marketing departments make sure the label is on the box. part of my motherboard selection had to do with SLI support with 4 slot spacing for 2 cards. unfortunately, my choices were very limited. I'm bummed that you can't use the new board for your rig. Best of luck / skill with the build!
 
OP
torin3

torin3

Member
Joined
Dec 25, 2004
Ok, I was mistaken. Apparently the splitter cable for the PCI-E power to the 2nd card is somehow bad. I found this out when I was trying to set it back up for the old motherboard. I knew the GPU was working in the old system, so I didn't think to check that it was functional on its own.

So, 2 cards are working, 1 on each CPU. I'll give it a few hours to pull new WUs and verify the PPD is at least in the same range as the old system to make sure the x8 speed is good enough for full PPD. Then I'll start adding other cards. I should have enough other cards on hand to have 6 in this and 2 or 3 in my main gaming machine.

Whoo-hoo!!
 
OP
torin3

torin3

Member
Joined
Dec 25, 2004
Well, after both cards have completed WUs and are running fresh new ones, the PPD is less than half of what it should be.

There are 3 things that might be the cause of this.

1. I've got a couple of passkeys. I might have used the wrong one and I'm not getting the bonus. But the FAH shouldn't show this, as the QRB is applied or not applied after the WU is submitted, so the chance of it being this is about 0.01%

2. The x8 slots just can't pass enough data. I can't really verify this without switching motherboards. I'm rating this chance as about 30%

3. The riser cables aren't shielded well enough, and I've got crosstalk killing the PPD. If this is the case, I can test it by swapping out the cables on my main desktop, as they are shielded and the cards on them give the expected PPD. I'm rating this at about a 70% chance.

So, tomorrow I'll swap out a cable from my desktop with the Ubuntu system and see if the PPD on the Ubuntu server goes up and my Win10 desktop goes down.

If they dont' change, then I'll swap back to my other motherboard and look for a sever board that has x16 data width slots and go from there.
 

HayesK

Member
Joined
Oct 11, 2008
in linux, nvidia settings shows what speed the slot is running at and the utilization, which will allow confirm the slots are running at x8 gen3.
 
OP
torin3

torin3

Member
Joined
Dec 25, 2004
in linux, nvidia settings shows what speed the slot is running at and the utilization, which will allow confirm the slots are running at x8 gen3.
Here is what I have.

ppdrate.jpg

My overall PPD for this machine. With the previous motherboard and no riser cables, I was getting about 6.5M PPD

3070.jpg

3070Ti


1080ti.jpg

1080Ti

Looks like the 1080Ti is running at Gen2. This motherboard is supposed to be able to change the slots between Gen 1, Gen 2, and Gen 3, but it isn't in the BIOS that I've seen.

But if I'm remembering right, I think you said that x8 was fast enough for GPU folding without taking a PPD hit.
 

WhitehawkEQ

Premium Member
Joined
Dec 6, 2010
1. I've got a couple of passkeys. I might have used the wrong one and I'm not getting the bonus. But the FAH shouldn't show this, as the QRB is applied or not applied after the WU is submitted, so the chance of it being this is about 0.01%
Question: Why are you using a different passkey from your Windows folders? Don't you just use the same passkey for all folders Windows/Linux?
Back when I was folding with my 4P's, I was using Linux and same passkey as my windows folders.
 
OP
torin3

torin3

Member
Joined
Dec 25, 2004
Question: Why are you using a different passkey from your Windows folders? Don't you just use the same passkey for all folders Windows/Linux?
Back when I was folding with my 4P's, I was using Linux and same passkey as my windows folders.

Because I had one for the original torin3, and one for the chimp challenege so we wouldn't have to wait for it to get validated with 10 WU for the QRB, and one for my current user name. And I can't remember if I kept the original one, or switched over to the newest one when I changed usernames, and they are all still in my Gmail when I search for passkey.

I'd like to check my current setup to see which one it is...but YOU CAN'T DO THAT STANFORD!
 

ihrsetrdr

Señor Senior Member
Joined
May 17, 2005
Location
High Desert, Calif.
Because I had one for the original torin3, and one for the chimp challenege so we wouldn't have to wait for it to get validated with 10 WU for the QRB, and one for my current user name. And I can't remember if I kept the original one, or switched over to the newest one when I changed usernames, and they are all still in my Gmail when I search for passkey.

I'd like to check my current setup to see which one it is...but YOU CAN'T DO THAT STANFORD!
You can see your passkey that's in use:
Code:
 cat /etc/fahclient/config.xml
 
OP
torin3

torin3

Member
Joined
Dec 25, 2004
Ok, the passkey is the right one, and I've swapped cables between computers. x8 PCI-E computer is still about 1/2 the expected PPD. Main desktop is at the expected PPD. Cables seem to be fine.

I guess the 6 PCI-E slot motherboard just can't push the data fast enough.

@HayesK Anything else you think I could try, or do we chalk this up to x8 slots can no longer handle full PPD with 10xx and above Nvidia cards?

In the next day or two I'll swap back to the old MB and see if I can get at least 3 cards working at speed on it until I can get a new 4-8 x16 PCI-E functioning MB.
 

HayesK

Member
Joined
Oct 11, 2008
Nvidia settings will show the pcie gen the slot is running at when select the line with the gpu numer/name. Have a pair 970 still in x8 gen 2 linux, but all the newer cards are in x8 gen3 linux. Think x8 gen 2 linux should be fast enough for 1080Ti.
 
OP
torin3

torin3

Member
Joined
Dec 25, 2004
Ok, here are the pics of the lines you asked, and the PPD on each card.

1080a.jpg

1080 Ti. Says Gen 2. I'm getting beteen 750 and 900K PPD for a card that should be able to do 2 to 2.2M PPD

3070a.jpg

3070 TI card that should be getting me 4.5 to 5.5M ppd is getting me about 1.8 to 2.1M PPD


bios.jpg


And this is the BIOS for setting the gen (1/2/3) for each slot.

I'm clueless as to why I'm only getting a little less than half of the PPD I should be getting.

Seriously, anybody have any idea of what my problem is here? Fresh install of Ubuntu. Running driver 510.
 

HayesK

Member
Joined
Oct 11, 2008
From the nvidia settings screen shots, GPU0 (3070) is running at x8 Gen3, GPU1 (1080Ti) is running at x8 Gen2. As long as there is at least one real CPU core per gpu, everything should be fine. Odd that the GPU1 is reporting address pcie 130.

From the bios screen shot, thinking the first and third pcie slots would be the first slots on each cpu.

Perhaps try plugging the adapters into different slots and see what speed nvidia settings reports for each slot.

My 970s on x8 Gen2 make about 800K ppd +/- 150K.
My linux 1080Ti on x8 Gen3 make about 2.3M +/-300K.
My linux 3070Ti on x8 Gen3 make about 5M +/- 1M.

Something odd going on.