6 GPU Folding Rig Build

[torin3]

And my garage machine keeps crashing. I'm thinking it is the MB that is failing like it was before switching OS on it, which kind of prompted this upgrade quest.

 

[unsafesteagle]

Hey Torin,
I dont know how the crypto miners setup, yet ddr3 setups i recall attempting more than 2 gpu rig wasnt worth the effort.
Not sure this has changed, you must know.
good luck

 

[ihrsetrdr]

Some time after GPU folding started I had an MSI P6N Diamond LGA 775 with 4 PCIE X16 slots, filled them with (3) 8800GTs & (1) 9800gtx.

 

[torin3]

Well, I bought an open box ASRock EP2C612 WS motherboard on Ebay. They accepted my make an offer of $225 within 10 minutes.

It is a socket 2011-R3 MB, that uses DDR4 RDIMMS (also LRDIMMS), dual CPU. I've already made an offer on 2 10-core E5-2634 v4 CPUs. After that, I only need 2 DIMMS and 2 HSFs and I should be back up and running.

This MB has 7 x16 physical PCI-E slots, 1 of which run at x16, and the remaining 6 run at x8 if they are all populated. (3 can run at x16 at the same time with no more than 4 cards).

I'm hoping that this resolves my problems.

Fingers crossed!

 

[dfonda]

Well, I bought an open box ASRock EP2C612 WS motherboard on Ebay. They accepted my make an offer of $225 within 10 minutes.

It is a socket 2011-R3 MB, that uses DDR4 RDIMMS (also LRDIMMS), dual CPU. I've already made an offer on 2 10-core E5-2634 v4 CPUs. After that, I only need 2 DIMMS and 2 HSFs and I should be back up and running.

This MB has 7 x16 physical PCI-E slots, 1 of which run at x16, and the remaining 6 run at x8 if they are all populated. (3 can run at x16 at the same time with no more than 4 cards).

I'm hoping that this resolves my problems.

Fingers crossed!

Pulling for you, hoping you can make up for the 5 to 6 mill I had to drop.

 

[torin3]

Foutd a better deal on the CPUs. Getting 2 12 core CPUs for about $90 shipped. 32GB DDR4 RDIMM (4x8), and HSFs that should all arrive sometime in the next week and a half.

Also may have gotten the current machine running again with 3 cards. We'll see if it is stable. Looks like it wasn't liking that I had added the 3080Ti after the 3070Ti.

So I added them from top end to low end. Seems to be working for the moment. Stability might not be there.

I also found a 2nd 4pin molex plug in for the MB that I'd missed before that may also help (it was side mounted on the edge of the MB, so I'd thought it was a USB cable plug-in until I looked at it from the edge.)

 

[torin3]

Ok. It appears that when I install the folding client, I can fold one set of WUs and then they fail to fold the next ones saying they can't detect CUDA or OpenCL support for the GPUs.

If I uninstall and reinstall the client, it will do another set and then fail again.

This is the error message I get. I installed OpenCL and the CUDA toolkit beforehand and still get this.


Is this possibly the effect of running 3 different cards, or something else? I'd think if was the mixed cards on Ubuntu, I'd find something about it, but I haven't yet.

Any suggestions?

 

[dfonda]

T, Have you tried a few different Nvidia drivers?

 

[torin3]

I tried the 515 version drivers, but I lost video and couldn't even terminal login and had to reinstall the OS. But that was with all 3 cards installed. I might try again one at a time.

 

[dfonda]

I tried the 515 version drivers, but I lost video and couldn't even terminal login and had to reinstall the OS. But that was with all 3 cards installed. I might try again one at a time.

Is there a server version of the nvidia driver?

 

[torin3]

Is there a server version of the nvidia driver?

Yes. I did try the 510 server version, but had the same result. But again, that was with all 3 cards in the machine. I may try it again with just the one card, make sure it is stable, then add the additional cards one at a time.

 

[superducky]

I think you'll have much more luck with the Broadwell/Haswell based mobo. Fingers crossed for you!

 

[torin3]

Thanks!

Well, everything should arrive by Saturday, so I'm hoping I'll have it up and running by Sunday. I've got Constable firearms qualification on Saturday, but that should be done by 11am.

MB and HSFs should arrive today.

 

[WhitehawkEQ]

I think you'll have much more luck with the Broadwell/Haswell based mobo. Fingers crossed for you!

Speaking of Haswell/Broadwell, my 3 soon to be 4 Intel systems are X99 LGA2011-v3 MB's, the MB is the GA-X99-Designare EX by Gigabyte with i7-6800K CPU

 

[torin3]

Well, the CPUs arrived a day early. If I get everything setup for firearms qualification tomorrow, I think I'll try and put this together.

I'll start with 3 video cards, then if they work, Saturday or Sunday I'll add the 2nd PSU and start going for broke.

 

[torin3]

Well, I'm going to have to do a little modification on the case. The 20mmx20mm extrustion that hold up the back end of the video cards is interfering with one of the CPU HSFs. I can move it over an inch and it won't be a problem, but I need to get 2 drills to do the modifications.

I'll post pics in the morning. I've had a killer headache for most of the day, so after getting my constable qual done, I was mostly laying down trying to ignore it.

 

[torin3]

Case modded and MB installed.

Power on, the fans all power up. No video, no beeps, no troubleshooting LED codes (no light from the LED number block).

Onboard video only, 1 memory stick, no change.

Looking at the manual, it looks like I may have plugged one of the CPU fans into the wrong header. Will head back out and check.

 

[torin3]

Well, I can get into the IMPI control, looks like it is working. Sensors and status are all good, but launching the remote control applet gives me a black screen for it. So, I have no idea why there is no video out from it at all.

Any suggestions?

 

[ihrsetrdr]

How about that 4 pin molex PCIE power connector?

 

[torin3]

It is plugged in, even though the manual says that it is only needed with 3 or more cards and I'm starting off with only one.

I'll try with it unplugged just in case.

Edit: no joy.