I don't need a pump. I need a PUMP. Looking for outside the box suggestions.

[WaterMiner]

Background:
Long time lurker and first time poster (man I hate when people say that). I have 24x 5970s (48 GPUs) which need to be cooled. I intend to watercool them not for extreme overclocking (they actually will be modestly clocked for stability) but for economics. I have built numerous watercooled systems in the past but nothing on this scale.

The 5970s are dumping 250W of heat ea, combined that's 6 KW (20,000 BTU). This presents two challenges. The first is the GPU farm has been expanded since last summer when a mini split system installed. To handle 6KW I am looking at $3K for upgraded cooling system and then $1500 to $2000 a year in energy costs. So the problem is "simple". I need to move 6 KW of heat from inside to outside using liquid and if I can do it under $5K it will pay for itself in a year. Later I also plan to tap some of those BTUs to preheat hot water and possibly heat the house in winter.

The details:

The 24x HD 5970s GPUs are currently air cooled 3 graphics cards (6 GPUs) per system. I will be moving them to 4x5970 configuration in a 4U chassis. Yes 48 GPUs in 24U of rack space. To anyone wondering the servers are controlled by a non-GPU "watchdog" server which configures, monitors, and controls them (including power cycle and power off). The watchdog will be expanded to provide monitoring on the entire cooling loop (flow, pump current, ambient air temp, reservoir level, 2 water temp sensors, and 2 outside air temp sensors.

Here is a test system. Excuse the ugly routing. This was more to test that 4x5970s can run under load 24/7 in a 4U rackmount chassis. It has been running like this under 100% load for a week now.

The test rig has an internal MCP655-B and 5.25" bay reservoir. Initially I had considered having a "closed" internal loop (4 GPU -> pump -> flat plate heat exchangers -> 4 GPUs) and link the cold side of flat plat heat exchanger to an outer loop. This would provide fluid isolation (sealed inner loop for each chassis & shared outer loop linking all the chassis). While I still think that is optimal it is simply too expensive.

The revised plan is to have a mainline and then a manifold to run branch lines to each rig. Each rig connects to the hot-side and cold-side manifolds via flexible tubing and quick disconnect fittings. The mainline will connect to a large water to air heat exchanger and 16" 1000cfm fan mounted outside.


ASCII diagram might help:

                        3/8" PEX branch lines
from heat-exchanger
                         |--------{server #1}--------| 
1" PEX cold line         |                           | 1" PEX hot line 
-------------------------|--------{server #2}--------|-----------------    to pump/reservoir/heat-exchanger   ---> 
                         |                           |
                         |--------{server #3}--------| 

          cold side manifold                hot side manifold

                -----(flow direction)--------->

Only 3 servers show for simplicity.  Total of 6 will be used.

Here is rig connected to a test radiator. Yeah 4x120mm is undersized for 1000W thermal load but it being winter helped. Once again it was more a proof of concept to test it before building the system out.

Hey noob didn't you say something about a pump?
Right I just thought some info on the setup might help.

So I could use the help of experts on a couple points.

Pump:
So I (maybe incorrectly see below) am looking for a pump capable of 12 gpm (16 preferably) at a head of 8 feet. Totally wrong? You tell me. Heat dump is less of an issue than in most setups as it is dwarfed by the 6KW being transfered from the GPUs. The problem is that excludes all "normal" water cooling pumps. They are simply not even in the ballpark. Even dual high end pump solutions fall very short. I first looked a radiant heat circulation pumps but most of them are cast iron. Sump pumps might work but I am not sure if they are designed for 24/7 operation. Most aquarium pumps lack the flow I am looking for.

I then looked at Iwaki RD series but the RD-30 only has max GPM of ~6 which is insufficient. The RD-40x might be useful (18 gpm @ 4ft, 16 gpm @ 8 ft) but I can't find it anywhere. Iwaki does make another series, the MD (and WMD). They seem to be a good fit. They have 1" NPT fittings, high head, and high flow.

http://www.iwakiamerica.com/products/wmd.htm

looks like MD is made in Japan and WMD is made in the US.

I was thinking of a WMD-30RXLT or maybe WMD-40RXLT but I thought I would check here.

Flow necessary:
Since there will be 6 servers (w/ space for 8) and they will be connected to manifold in parallel to acheive 2gpm across each server would require a pump capable of 16gpm (at whatever head the system creates). Right?

Parallel vs Serial:
My test rig has the 4 GPUs in parallel. Likely I shouldn't right? Flow is reduced across each GPU. i.e. if flow into SLI block is 4 GPM then when it branches parallel 4 ways it is 1 GPM across each GPU. I understand normally this won't matter but remember each server will also
be in parallel.

Head:
I have no idea how much head would be necessary for a system like this. I also have no idea how to calculate it for one server and if I knew what it is for 1 server is it the same for 6/8 servers (because they are in parallel)? Any insight, thoughts, guesstimates, or tips would be helpful.

Thoughts?

I am open to any other pump recommendations. Also some confirmation/correction on the likely flow and head I am looking at would be nice. Yeah I know it was long but I wanted to avoid a "get a Liang" or "aquarium pumps" suck type response.

 

[johan851]

And a great first post! Very interesting problem you have there.

In a nutshell, the problems with serial routing are higher restriction and "passing" heat down the chain. We usually don't worry about the second issue, but if you have, say, 8 GPUs in a serial chain and the temperature goes up 2C per GPU (2C is extreme, but let's say you have a low flow rate) then the last GPU will have water that's 16C hotter than the first. That may actually not be an issue for you depending on how they're clocked and how the system performs, but it's worth mentioning. If you're doing mild-to-no overclocking then the GPUs won't mind whether they're at 40C or 60C.

The problems with parallel are uneven restriction. If one branch is significantly less restrictive than others, then your water won't go evenly down the branches. If your blocks are identical, though (4 identical GPU blocks per branch) then parallel will work great. We usually advise against that because most watercoolers don't have a substantial amount of identical components, but you do.

Iwaki is a good idea - that was going to be my first suggestion. Diggrr has some experience with loops on this scale, so hopefully he'll chime in as well. I'm pretty sure he uses an Iwaki.

The GPM figures are probably pretty deceptive because they're measured at 0 head pressure (I think). You want a *lot* of head pressure. The MD-40R or MD-40R(T) (non-X!) would be a good candidate, so it sucks that you can't find one. The next most appropriate is probably the MD-55R(T). Alternatively, you can start stacking the pumps in series; in that configuration, head pressure is additive, so two in series will double head pressure.

For your per-server tubing, I'd use 1/2" ID or 7/16" tubing with 1/2" barbs if you can. That'll help to lower overall pressure a bit, since barbs for 3/8" tubing tend to have a really small inner diameter. Unless you've got some really tight routing to do then there's no reason to use 3/8" ID.

Measuring head pressure is hard. But one ghetto idea is to hook one server up to a bucket of water (a 5-gallon bucket) and suspend the bucket 5 feet up in the air before you start the flow. Figure out how long it takes to drain the five gallons and now you have a flowrate at 5ft of head pressure.

Since there will be 6 servers (w/ space for 8) and they will be connected to manifold in parallel to acheive 2gpm across each server would require a pump capable of 16gpm (at whatever head the system creates). Right?

2GPM per server is going to be tough to achieve. I would shoot for about 1GPM per waterblock - that should be plenty.

 

[Diggrr]

The way i see it, you have two options.
A) Use one mcp350/5/x per case using 3/8" line back to 1" feed lines that lead out to your big heat exchanger.
B) Use one large pump to push manifolded 3/8" lines (also using 1" trunks).

I'd suggest using 1" trunk lines, that not only allows a lot of flow, but keeps that flow strong in a large heat exchanger--I used the Rainbow 3000 that pushed 13.65 GPM with 10.33 Ft of head using only 55 Watts through a Toyota copper truck radiator.
Love the pump. Quiet, versatile, big outlets, low(ish) wattage, found at Marine Depot. I actually used it as a sump pump once when my basement flooded, it pushed 10" of water out a garden hose up the stairs and out into the driveway...it's submersible to, but I normally used it dry. It is a mag-drive pump, but the impeller is shaped like a turbo charger...very efficient.

Sounds like a killer fun project!

 

[Diggrr]

I made a quick pic of a good manifold design for ya, plumbers use this all the time for distributed lines. It gives equal flow and pressure to each T, which is what's required for multiple systems, and it's easy to make of PVC or Copper from the Home Depot.
All 1" tubing/fittings save for the 3/8" feed fittings.
You need two, one for goesinta, and on for goesoutta.

A small hint: The 1" PVC is thick enough to drill and tap for your 3/8" fittings without requiring a T fitting...

 

[robble]

I would second the idea of a large pump supplying a large line which then has parallel smaller lines feeding smaller pumps for each server. The water exiting the servers would then enter back into another large line with feeds your radiator system and back to the reservoir/pump. In the military that is how the chill water system worked that fed multiple systems (minus the small pumps - head pressure was very high and all lines were hard lines). Without excessive head pressure from the main pump the little ones would guarantee flow through each server even in parallel.





What do you use the gpu farm for? That would be an incredible folding rig.

 

[Diggrr]

Update on the pump..
I went to the basement and grabbed the pump. It has sat there for 2-3 years unused, still caked with mud from the sump emergency.
I brought it to the kitchen sink and disassembled/scrubbed it clean. Took 3 sinks of water to clean it all.

Ran a 4th sink of water and plugged it in....so glad I'm a bachelor....with a mop!

Runs great still. It used to do 24/7 duty for 3 years until retirement and subsequent abuse.
Impeller is not quite a turbo design, but has 3 radial curved veins.
Not super quiet, but quieter than a Sanyo-Denki server fan, with a low pitched rumble.

Still impressed with it!

*of note: Marine Depot says 55 watts, pump data plate says .66 amps...a little math says 79 watts.

 

[unijabnx2000]

What if you could mount the reservior and radiator up high, have the servers in the middle, and the collection tank and pump down low. That way depending on how much water you keep in the tank at the top, it will mostly flow downhill at a good rate on its own.

BTW... Im wondering how you said they will pay for themselves. How will they make income for you?

 

[Diggrr]

Gravity plays no part in a sealed system. Water flows up at the same rate it flows down. Pump flow will spank gravity flow in tubing too.
It's always best to have the pump at the bottom though, so it's always primed and moving water.

 

[Seebs]

Oh wow... this is a gargantuan undertaking to say the least.

One thing that comes to mind, setting aside the whole pump/flow/head/gpm issue, is the heat dump side.

Have you considered geothermal? Is that even a possibility where you live? I think it'd be quite cheaper to run pipe underground than it would be to run the number of radiators that you'd need to dissipate 6KW worth of heat from them GPUs.

 

[unijabnx2000]

What about using an air-to-water intercooler for a car in reverse?

 

[Diggrr]

That would work nice with a blower for the air supply, but alas would not have enough surface area to remove enough heat.
Temperature differential is the key here. With a car the exhaust is much much hotter than the water system, so it works. With a computer, the difference between air/water temp isn't nearly as great, so it would leave you immediately wanting for more.
I loved my truck radiator, it had the surface area of 4 or 5 120mm triple rads in one high flow package, as well as the same preferred materials. He'd probably need two (or more) of those even to cool 24 GPU's very well.

 

[unijabnx2000]

If it has enough surface area to remove the heat from a turbo charger, id say it does.... But then again that air thats its removing the heat from is compressed

so what if you used expansion to cool it?
seal both ends, drill and tap them for nozzles. Use an air compressor to fill the space, and a bleed valve on the other... I know this it too elaborate to try to use, but its a cool idea. lol

 

[Automata]

If it has enough surface area to remove the heat from a turbo charger, id say it does....

Heat exchange does not work like that. The temperature difference in a car is massive, so a lot of energy gets transferred quickly. The temperatures in a computer water cooling system are a few degrees Celsius apart, so it won't as readily transfer. A turbo runs a lot hotter than a computer.

 

[Diggrr]

Here's a better explanation, if not longer (**writing while Thideras posted )..
Say your exhaust gas is ~1500° and your water temp is ~ 200°. Thermal energy would easily push from air to water because they are 1300° apart, despite the relatively small surface area between the two.
It's a difference greater than your hand and dry ice!
A car also has the advantage of a very large water pump and a 350 CI++ 5,000+ rpm pump pushing the air.

With a computer, the temp difference can be as little 10°C with relatively small fans and pumps, so a much larger surface area is needed to coax the heat from one side to the other.

The two just don't compare to each other because the temps are so much closer together on computers.
A plain heater core or radiator do a much better job.

 

[unijabnx2000]

The water in a A/W system is separate from the engines cooling system, so it doesnt get to the same temp the coolant does. The inlet air in even turbocharged vehicles should never get above 140F. But I understand what you are saying.

 

[Diggrr]

Cool, I drag raced air cooled motorcycles...now I've learned something!

I'd thought of an easier explanation too.
Dip a hand in ice water for a second. No problem.
Dip a hand in liquid nitrogen for a second. Frozen stump, missing hand.
Same time, same surface area, much greater heat exchanged only because of the difference in temps.

Anyway, on to teh cooling of teh ubersystem!

 

[Theocnoob]

I would second the idea of a large pump supplying a large line which then has parallel smaller lines feeding smaller pumps for each server. The water exiting the servers would then enter back into another large line with feeds your radiator system and back to the reservoir/pump. In the military that is how the chill water system worked that fed multiple systems (minus the small pumps - head pressure was very high and all lines were hard lines). Without excessive head pressure from the main pump the little ones would guarantee flow through each server even in parallel.





What do you use the gpu farm for? That would be an incredible folding rig.

If this is a vertical arrangement the top units will receive significantly less water flow. I do aquariums. It all applies. It'd need to all be level. Even then you're going to see reduced flow on the lines further away from the pump just because the pressure will be less due to water flow being siphoned off at the previous small pumps.

 

[robble]

If this is a vertical arrangement the top units will receive significantly less water flow. I do aquariums. It all applies. It'd need to all be level. Even then you're going to see reduced flow on the lines further away from the pump just because the pressure will be less due to water flow being siphoned off at the previous small pumps.

as long as the smaller pumps have adequate water supply there will be enough flow through the server it supplies. Even if the lower servers get 2.5 gpm and the top ones get 1.5 gpm it's still good.

If he was using hard tubing (to handle the pressure) a manifold with 1 very strong pump would suffice. I know this from the chill water systems I used to work with on a submarine. The pump was two stories down and 200' away from some of the equipment (and it was a LOT of equipment being fed). While this example is way overkill for him, it's not underkill like the aquarium example.

 

[WaterMiner]

Thanks for all the responses guys lots of good ideas. Will try to cover them all.

On the heat exchanger. I have a very large heat exchanger and a very high rpm fan. If necessary I have the space to install a second one. I based the size of heat exchanger on one that cools a 5KW industrial laser. Geothermal is something I may consider in the future.

Robble, nice diagram. That is the concept. I hadn't considered a "booster" pump for each server but if necessary that may be a route to go. I would like to avoid that if possible (and it may not be) simply because adding a pump to each rig means pump cost *6 (or possibly 8) and that adds up fast.

On waterblocks. All waterblocks are identical. Danger Den HD 5970 waterblocks. EK blocks with SLI bridge would be nice but when you are dealing with 24 cards even small incremental cost explodes the budget.

On what the GPU farm is used for ... I don't want to derail the tread. The farm has been very profitable but I am facing rising competitive pressures. Hopefully a low operating cost water cooled setup will let me run the 5970s for some time.

On HD 5970s. Come on games ebay them. Supply & Demand. They are still going for $400 used with no warranty. Sell those HD 5970s and buy some shiney 7000 series cards.

On mainifolds and mainlines. I was thinking the same concept for the manifolds diggr. I considered a prebuilt manifold but man they are expensive. I found some nice solid block manifolds that would be perfect but they are .... aluminum. Sucks. The mainlines will be 1". I was thinking of 3/4" PEX but max safe slow is 8 gpm. I don't want water wearing down the pipes because velocity is too high. Safe limit for 1" is 12 gpm.

On barbs and tubing. 1/2" barbs and tubing sound fine. No problems there. One thing that sucks is koolance doesn't make 1/2" QDC. I need some way to disconnect the rig from the manifold and without dumping coolant or disrupting the other servers. I could use a pair of ball valves and a union but it isn't any cheaper than a QDC. Still no 1/2" QDC makes me wonder if 1/2" barbs and tubing do any good w/ 3/8" QDC. Anyone care to weigh in?

On serial vs parallel. johan851 based on what you are saying (>2 gpm per rig is unlikely) I think I have no choice but to put the 4 cards in series right? Otherwise if rig is closer to 1 gpm then each branch of the SLI will be 1/4 gpm or less. Way too low. Right?

The parallel manifold and limits on flow rate make putting the cards is series a requirement wouldn't you say? Temps aren't really an issue. If I can keep all the cards <60C year round it is fine for what I am doing. I already have software which dynamically adjusts the clock based on core temp for each core.


So that brings us full circle back to the pump.

Based on the input and insights I am thinking WMD-R40 series is a good choice.

http://www.iwakiamerica.com/products/wmd.htm

I don't think the WMD-30 family is strong enough (but that is just a guess based on likely head for the system)
WMD-30 curves: http://www.iwakiamerica.com/Literature/MD_WMD/40famcrv.pdf

WMD-40 curves: http://www.iwakiamerica.com/Literature/MD_WMD/40famcrv.pdf

Now I know generally the "X" version is shunned for its reduced head but take a look at the curves.

The two curves intersect at 11 gpm and 13ft of head. The standard model (WMD-40RLT) is better with higher head but how likely is >13ft of head? Even if it is at > 15ft of head the gpm is <8 which is not good. So really the WMD-40RLT only "shines" if the system head is 13 to 15ft. Any less and the high flow (WMD-40RLXT) is better. Any higher and a bigger pump (MD-55RLT) is likely needed.

BTW:
W = American Built Motor (no W = Japan). "W" models are cheaper but use more power
MD = the pump series
## = motor size
R = Rotating Spindle (pump type)
T = Threaded (has NPT threads instead of barbs)
L = UL rated
X = High flow impeller (you can simply switch impellers to change the flow vs head)
Z = Ultra high head impeller (ultra low flow, no X or Z is the avg head, avg flow impeller)

Now I have one worry about the "bigger pump"

MD-50RLT
curve: http://www.iwakiamerica.com/Literature/MD_WMD/md55Rcrv.pdf

At a heat of <15 the pump will be pushing >12gpm. That likely is too much for the sytem. Now if I knew the system is >15 ft of head and thus <= 12gpm then this likely is the perfect pump (except for the price ... don't ask).

Since this is an AC pump I can't vary the power. Is there any way to increase head of the system if necessary (to prevent excessive flow in mainline)? A partially closed valve? Not a common problem but this is a monster pump. It is 200W, 1/8 HP (98W) water power, 11 pounds, and almost a foot long.


So which pump?

TL/DR version:
Help me pick a pump.

WMD-30 series? Convince me it is powerful enough.
WMD-40RLT? Only shines >13ft of head.
WMD-40RLXT? High flow model. My current pick. Tell me why it is wrong.
WMD-55RLT? Is there such a thing as too much pump.

 

[m0r7if3r]

*of note: Marine Depot says 55 watts, pump data plate says .66 amps...a little math says 79 watts.

Not if it's an AC pump (I didn't look it up), if the pump is an inductive/capacitative load then voltage and current won't peak at the same time and your max will be less than if it was a purely resistive load.

/ee