Is the system with a top rad in the second pic of your first post a different system? Or was that the old alu system. The rad you are using appears to be very slim (in order to fit both rad and fans in the rear chamber of the case). It is probably thinner than the original rad that was on top of the case and not adequate to cool the components used.
I have the same case and a similar (high wattage) GPU, and I use a 60mm top rad (exhaust) and 15mm bottom rad (intake) with EK vardars. This still causes them to spin in excess of 1500rpm, partially because the air is warmed from the first rad. Unfortunately I chose to place a reservoir in the side fan / rad mount area, (the front mount ones were not available at the time). In our current configuration I would use the side as intake, drawing fresh cool air into the rad, and the top as exhaust (air warmed by the rad will still be adequate for the passively cooled components on the motherboard and memory). If you are unsatisfied with EK fans, there are some similar high pressure fans but they are expensive and not RGB from a couple brands: gentle typhoon (not sure which brand is making them now) and Noctua NF-A12. Corsair is known to make nice RGB fans as well, however they do have several models and I don't recall which ones are best for use with a Rad.
I believe the ideal setup for a high wattage GPU in a dual component loop for this case is a two rad setup with rads on the top and bottom (one can probably be 240mm) with the side as an intake. I'm not sure if the bottom would be better as an intake or exhaust in this setup. Top for sure exhaust. I'm still working it out myself and I don't have all the answers, but hopefully this will help a bit.