Water leak destroyed two 290x's

Hey guys...So this was my first water cooled system. Here are the components:

EK H30 360 Kit
3 EK Radeon R9-290X blocks
EK FC Bridge Triple Parallel Z77
Cost me roughly $800



Now, the leak didn't happen at any of fittings. It was from their screws on their block. I had no idea it was happening. I did notice the system randomly shutting off from time to time but I was mining cryptos so I though it might have something to do with that but I was wrong. Here's the top block:



Here's the damage to the cards underneath it:



And here are the water blocks with this black gunk in them:



And this one is actually warped:



Since you guys probably have more experience with this, I have a few questions.
What the hell happened here?
What is black gunk? Is it corrosion or something else?
Is the manufacturer responsible or is it some mistake I made?
Can the water blocks be cleaned out?
I know its highly unlikely but are the gpus salvageable?

Oh yea, and the reservoir had this EK logo in the center of it. The sticker or whatever it is apparently heated up and came off the plastic and ended up inside the pump. I don't know if that has something to do with it. Any info is much appreciated.

to OCFs.

Oh wow, Sorry to hear about this.

First question is, how long have you had the system running from since you put it together? Was it running 24/7? Did you do a 24 hr leak test prior to firing everything up? Did you do maintenance when it comes to colored fluid since it could gunk up everything?

Thanks for the welcome. Yea, it was running pretty much 24/7 for about 5 months. I did do a 24 hr leak test first and it was fine. Other than adding a little bit of distilled water every couple weeks I didn't really do any maintenance to it.

Did the rig ever shutdown on its own?

Not that it will help you this time, but if you have to add water every couple of weeks in the future, that is not normal and it means you have a leak. When i bleed out my system i top off the res after 5-7 days and then i dont need to add water at all for a year+.

It looks like gunk from the dye (or growth, its hard to tell exactly what that black stuff is through a picture) clogged a passage and back pressure blew out the o-ring, at a guess. Its also possible there was some kind of adverse reaction between the dye or whatever you have in your water and the o-ring. The blocks are cleanable and salvagable, you will need to order new o-rings from ek, and they are fairly straight forward to open up and clean.

The cards could go either way. What you want to do is get some 99% rubbing alcohol and a very soft toothbrush or the like and wash off all that stuff. After washing them let them dry very thoroughly, a day or more in a hot dry area, and there is a chance they will work again. I have had water leak onto a running card shut the computer down and live, it can happen. But they could also easily be dead.

GTXJackBauer said:

Did the rig ever shutdown on its own?

Click to expand...

He mentions that it did.

What were the temps like? It seems that the water got pretty darn hot to me, probably from the dye gunking up in the blocks and I would guess inhibiting flow.

GPUs could potentially be salvaged, I would get some of the highest purity isopropyl alcohol you can (some places like Walgreens will have 99% behind the pharmacy counter) and spray the cards down with everything removed that can be removed. Hang to dry overnight and reassemble with stock cooling and check each individually.

Strange that it leaked out of the screws which suggests it leaked between the o-ring and block cover first and just found its way out the screw caps.

My guess is that a combination of the sticker in the pump, excessive heat, and the gunk buildup caused the loop to be pressurized on the side before whatever the primary blockage was and push water out of the block. Is it only the top card that leaked?


Edit: Supertrucker basically said the exact same things as me, my bad for leaving tabs open for an hour before replying!

I have had a block leak onto a card, when you clean everything off the cards, pull the cooler and check the other side of the card also and clean the pcie slots on the motherboard.
I used everclear to clean mine and set it and the motherboard outside in the sun for two days and got lucky with mine.
I run distilled water and 25% zerex G05 coolant and add 2 drops of cooling system UV dye from the auto parts store so all i have to do is turn off the lights and shine a UV light through the side of the rigs once a week to inspect them.

best of luck with the salvage operation.

Supertrucker said:

It looks like gunk from the dye (or growth, its hard to tell exactly what that black stuff is through a picture) clogged a passage and back pressure blew out the o-ring, at a guess. Its also possible there was some kind of adverse reaction between the dye or whatever you have in your water and the o-ring.

Click to expand...

My thoughts exactly. The pressure had no where to go and add the fact that the fluid got to its hottest point sitting still. Sticker peeling off is a good sign of very high heat. I would change all the O-rings as well since they probably got damaged or lost their purpose from the high heat possibly. That looks like clear tubing as you can see a color change from the tubing color from the fluid and the reservoir/CPU block. The fluid was cooking and might have gone past the threshold permitted for the tubing. Could have been a build up of plasticize as well.

When I asked if the system had shut down, I meant prior to see if he ever got a thermal system shutdown to see any warning signs prior to getting a leak.

We always recommend folks to run distilled water and biocide, that's it. Its a expensive lesson to learn but you might be able to salvage those GPU's again if the appropriate steps are taken as already said by the other members.

Edit: And I almost forgot to add, the heat must have been very high with only a 120.3 rad cooling a AMD CPU and 3 290xs running 24/7. I can only imagine. Running it for months without maintenance, etc.

There was a sticker inside the pump? I'll bet that sticker was supposed to be removed before final hookup and start of the loop, it probably had some glue on it and that gat washed away with all the flow and deposited itself in that one gpu block, and stopped flow through there, are the gpus in series or parallel? But I have no experience, just trying to learn from as many people as I can, this is just a calculated guess.

@Supertrucker
I had a feeling that I shouldn’t have to add water but from what I could tell the water was evaporating in the reservoir that I figured was from heat build up inside the case.

@tool_462
It looked like the second card was about to start leaking. There was some slight residue around the screws on the second card.

@GTXJackBauer
The system did start shutting down but I assumed it was from mining. It never crossed my mind that it could be from something else. All I ran in the system was distilled water and the fluid that EK has in their kit.

@t1nm4n
It’s possible that the sticker was supposed to be removed. The gpus are in parallel.

Thanks for all the info guys. So what I’ve gathered from your info. The sticker seems to have been the main problem. After it got stuck in the pump the pressure built up on one side eventually causing it to leak. I’m thinking now that by the end, the water was barely moving through the loop causing the dye near the gpu to basically burn which is why its black. It’s also localized to right around the gpu and the outlet which you can see in the 6th and 7th picture.

All because of damn sticker.

looking at picture #2, post #1, that looks like a copper based coolant, if so could this have added to the issue?

I don't know about it being copper based. This is the coolant... http://www.frozencpu.com/products/2...centrate_100mL_-_UV_Blue.html?tl=g57c589s1547

From what I've been told and read online parallel splits the flow between the number of blocks you have it run to. so you had only 1/3 the flow over your gpus, which could also have created a problem. This article about CPU blocks and how flow rate affects temp helped me a lot in understanding how flow rates affect temp and how blocks all have different characteristics.

It truly does suck that your cards are in jeopardy of not working, I do hope they cut out before damage was done. The blocks you have are the same ones I'm getting and if all the testers reviews are correct then they should be run in series so that they all have the same flow. I initially thought this was bad cause it would mean that water is getting hotter as it passes over each card, but the water is flowing so fast that it doesn't have time to absorb much heat, I would guess (no practical experience here, just from what I've read) you'd only see a 2-4c temps from the first gpu in the series to the exit from the cpu.

That is my biggest fear for cooling. I would cry for data if this happened. Good luck and I hope everything gets fixed