- Joined
- Jul 24, 2006
That x8 to x16 jumper is just following a pretty standard form factor for risers and there are no traces in it, you're good to modify it as necessary.
-
Welcome to Overclockers Forums! Join us to reply in threads, receive reduced ads, and to customize your site experience!
6 GPU Folding Rig Build
- Thread starter torin3
- Start date
OP
- Joined
- Dec 25, 2004
- Thread Starter
- #22
Well, I'm going to hold off for now. Looks like I can't boot the board with 2 GPUs plugged into the MB at one time. Slot position doesn't make a difference, and the BIOS settings don't help either.
So, I'll put the old MB in this case, and I can probably get 3 cards running on it while I try and get the parts for a 6/7/8 PCI-E board that will work with multiple video cards.
Key thing seems to be if they have latches to lock the card in, you can probably use multiple GPUs on it.
So, I'll put the old MB in this case, and I can probably get 3 cards running on it while I try and get the parts for a 6/7/8 PCI-E board that will work with multiple video cards.
Key thing seems to be if they have latches to lock the card in, you can probably use multiple GPUs on it.
- Joined
- Nov 1, 2009
- Location
- Austin, TX
yeah it's crazy, even though Nvidia no longer supports SLI, the motherboard manufacturers are bent on mentioning whether or not THEY support SLI on their boards. which i'm glad they do. their marketing departments make sure the label is on the box. part of my motherboard selection had to do with SLI support with 4 slot spacing for 2 cards. unfortunately, my choices were very limited. I'm bummed that you can't use the new board for your rig. Best of luck / skill with the build!
OP
- Joined
- Dec 25, 2004
- Thread Starter
- #24
Ok, I was mistaken. Apparently the splitter cable for the PCI-E power to the 2nd card is somehow bad. I found this out when I was trying to set it back up for the old motherboard. I knew the GPU was working in the old system, so I didn't think to check that it was functional on its own.
So, 2 cards are working, 1 on each CPU. I'll give it a few hours to pull new WUs and verify the PPD is at least in the same range as the old system to make sure the x8 speed is good enough for full PPD. Then I'll start adding other cards. I should have enough other cards on hand to have 6 in this and 2 or 3 in my main gaming machine.
Whoo-hoo!!
So, 2 cards are working, 1 on each CPU. I'll give it a few hours to pull new WUs and verify the PPD is at least in the same range as the old system to make sure the x8 speed is good enough for full PPD. Then I'll start adding other cards. I should have enough other cards on hand to have 6 in this and 2 or 3 in my main gaming machine.
Whoo-hoo!!
OP
- Joined
- Dec 25, 2004
- Thread Starter
- #25
Well, after both cards have completed WUs and are running fresh new ones, the PPD is less than half of what it should be.
There are 3 things that might be the cause of this.
1. I've got a couple of passkeys. I might have used the wrong one and I'm not getting the bonus. But the FAH shouldn't show this, as the QRB is applied or not applied after the WU is submitted, so the chance of it being this is about 0.01%
2. The x8 slots just can't pass enough data. I can't really verify this without switching motherboards. I'm rating this chance as about 30%
3. The riser cables aren't shielded well enough, and I've got crosstalk killing the PPD. If this is the case, I can test it by swapping out the cables on my main desktop, as they are shielded and the cards on them give the expected PPD. I'm rating this at about a 70% chance.
So, tomorrow I'll swap out a cable from my desktop with the Ubuntu system and see if the PPD on the Ubuntu server goes up and my Win10 desktop goes down.
If they dont' change, then I'll swap back to my other motherboard and look for a sever board that has x16 data width slots and go from there.
There are 3 things that might be the cause of this.
1. I've got a couple of passkeys. I might have used the wrong one and I'm not getting the bonus. But the FAH shouldn't show this, as the QRB is applied or not applied after the WU is submitted, so the chance of it being this is about 0.01%
2. The x8 slots just can't pass enough data. I can't really verify this without switching motherboards. I'm rating this chance as about 30%
3. The riser cables aren't shielded well enough, and I've got crosstalk killing the PPD. If this is the case, I can test it by swapping out the cables on my main desktop, as they are shielded and the cards on them give the expected PPD. I'm rating this at about a 70% chance.
So, tomorrow I'll swap out a cable from my desktop with the Ubuntu system and see if the PPD on the Ubuntu server goes up and my Win10 desktop goes down.
If they dont' change, then I'll swap back to my other motherboard and look for a sever board that has x16 data width slots and go from there.
OP
- Joined
- Dec 25, 2004
- Thread Starter
- #27
Here is what I have.
My overall PPD for this machine. With the previous motherboard and no riser cables, I was getting about 6.5M PPD
3070Ti
1080Ti
Looks like the 1080Ti is running at Gen2. This motherboard is supposed to be able to change the slots between Gen 1, Gen 2, and Gen 3, but it isn't in the BIOS that I've seen.
But if I'm remembering right, I think you said that x8 was fast enough for GPU folding without taking a PPD hit.
- Joined
- Dec 6, 2010
Question: Why are you using a different passkey from your Windows folders? Don't you just use the same passkey for all folders Windows/Linux?
Back when I was folding with my 4P's, I was using Linux and same passkey as my windows folders.
OP
- Joined
- Dec 25, 2004
- Thread Starter
- #29
Because I had one for the original torin3, and one for the chimp challenege so we wouldn't have to wait for it to get validated with 10 WU for the QRB, and one for my current user name. And I can't remember if I kept the original one, or switched over to the newest one when I changed usernames, and they are all still in my Gmail when I search for passkey.
I'd like to check my current setup to see which one it is...but YOU CAN'T DO THAT STANFORD!
- Joined
- Dec 6, 2010
- Joined
- May 17, 2005
- Location
- High Desert, Calif.
You can see your passkey that's in use:
Code:
cat /etc/fahclient/config.xml
OP
- Joined
- Dec 25, 2004
- Thread Starter
- #32
Thank you. Learn something new every day. Though the one in question was on a windows machine, but this works:
Code:
C:\Users\torin\AppData\Roaming\FAHClient\config.xml
OP
- Joined
- Dec 25, 2004
- Thread Starter
- #33
Ok, the passkey is the right one, and I've swapped cables between computers. x8 PCI-E computer is still about 1/2 the expected PPD. Main desktop is at the expected PPD. Cables seem to be fine.
I guess the 6 PCI-E slot motherboard just can't push the data fast enough.
@HayesK Anything else you think I could try, or do we chalk this up to x8 slots can no longer handle full PPD with 10xx and above Nvidia cards?
In the next day or two I'll swap back to the old MB and see if I can get at least 3 cards working at speed on it until I can get a new 4-8 x16 PCI-E functioning MB.
I guess the 6 PCI-E slot motherboard just can't push the data fast enough.
@HayesK Anything else you think I could try, or do we chalk this up to x8 slots can no longer handle full PPD with 10xx and above Nvidia cards?
In the next day or two I'll swap back to the old MB and see if I can get at least 3 cards working at speed on it until I can get a new 4-8 x16 PCI-E functioning MB.
OP
- Joined
- Dec 25, 2004
- Thread Starter
- #35
Ok, here are the pics of the lines you asked, and the PPD on each card.
1080 Ti. Says Gen 2. I'm getting beteen 750 and 900K PPD for a card that should be able to do 2 to 2.2M PPD
3070 TI card that should be getting me 4.5 to 5.5M ppd is getting me about 1.8 to 2.1M PPD
And this is the BIOS for setting the gen (1/2/3) for each slot.
I'm clueless as to why I'm only getting a little less than half of the PPD I should be getting.
Seriously, anybody have any idea of what my problem is here? Fresh install of Ubuntu. Running driver 510.
1080 Ti. Says Gen 2. I'm getting beteen 750 and 900K PPD for a card that should be able to do 2 to 2.2M PPD
3070 TI card that should be getting me 4.5 to 5.5M ppd is getting me about 1.8 to 2.1M PPD
And this is the BIOS for setting the gen (1/2/3) for each slot.
I'm clueless as to why I'm only getting a little less than half of the PPD I should be getting.
Seriously, anybody have any idea of what my problem is here? Fresh install of Ubuntu. Running driver 510.
From the nvidia settings screen shots, GPU0 (3070) is running at x8 Gen3, GPU1 (1080Ti) is running at x8 Gen2. As long as there is at least one real CPU core per gpu, everything should be fine. Odd that the GPU1 is reporting address pcie 130.
From the bios screen shot, thinking the first and third pcie slots would be the first slots on each cpu.
Perhaps try plugging the adapters into different slots and see what speed nvidia settings reports for each slot.
My 970s on x8 Gen2 make about 800K ppd +/- 150K.
My linux 1080Ti on x8 Gen3 make about 2.3M +/-300K.
My linux 3070Ti on x8 Gen3 make about 5M +/- 1M.
Something odd going on.
From the bios screen shot, thinking the first and third pcie slots would be the first slots on each cpu.
Perhaps try plugging the adapters into different slots and see what speed nvidia settings reports for each slot.
My 970s on x8 Gen2 make about 800K ppd +/- 150K.
My linux 1080Ti on x8 Gen3 make about 2.3M +/-300K.
My linux 3070Ti on x8 Gen3 make about 5M +/- 1M.
Something odd going on.
- Joined
- Feb 25, 2004
- Location
- N of splat W of Torin
What is left? Are they throttling?
OP
- Joined
- Dec 25, 2004
- Thread Starter
- #38
I don't think so. I've got a bank of 8 fans blowing on 2 cards that are well separated. And I've got the fans set to 88%.
- Joined
- Dec 6, 2010
- Joined
- May 17, 2005
- Location
- High Desert, Calif.
Just for comparison- a [somewhat] similar MB & proc(skt 2011, Xeon E5 26xx v1), and GPU(GTX 1080ti), same Bus type/Gen/Pcie Link speed, and by coincidence, the same WU:
PCIe bandwidth utilization is similar.
PCIe bandwidth utilization is similar.
