Hey!
I'm still using one of those as well and have built several xeon rigs for people needing a good rig for a -very- low budget.
As for your question and from my experience, the only way to be sure about the ram is to try... the platform is very finicky about that. I've used mismatched brands/size sticks with success and specially made kits with complete failure. Nobody has tested every type of ram and nobody ever will (especially exotic use of ecc ram on a non workstation mb) and even if you get a kit that worked on another system it might not on yours^^
You're almost certain to encounter troubles so:
- The problem is not really populating all the slots but mostly the quantity of ram. Above 12 it gets a bit wild for some reason. Most likely a mix of traces, power and controller not working well on hardware that was never meant to meet.
- I have noticed that if you populate all slots, it might be needed to slam the voltage to 1.6-1.64 (or a bit under, just test) or the system won't even boot. It's quite safe, I've done this on machines used to do heavy stuff without any problem for years.
- You're also likely to have undetected ram issues and I've found that it's usually solved by lowering the ram frequency and perfs, the voltage seems to have zero effect on that. So to avoid most problems when I need to put 24gb I get a 5680 because of the higher CPU multiplier that allows lower frequencies (the 5690 is way too expensive). Then maybe you can cut the timings a bit to compensate. Note that the max multiplier is the turbo so it won't really work even if it shows the proper frequency, you have to set it one step below max at least to really be able to use the overclock.
- But it can also just not work and then for some reason after fiddling, it will. It's a real pain. I have in front of me a perfectly tested and functionnal set of 6*4gb non ecc that worked fine on a machine for 3 years and after a cleanup the board don't want anything to do with it anymore. Same ram, same slot. Sad.
- 2Rx4 ram is an absolute no go, be careful when buying.
- ECC is fine only if you end up buying a xeon. Technically the memory controller can handle 48gb, the manual states otherwise because they didn't intend people to put a xeon on it. Dual xeon 56xx server can handle 96gb no problem. I've never tried because of the pain it already is to get 24gb to work.
The 1070 is plenty enough for such a rig but get a Ti or get a 970/980 if you can't find one. They're much bigger and more power hungry but you'll get a close gaming experience. I use mine mostly for some graphic work but without maxing everything like a madman, I can also enjoy any game on a x5650 @4.3 air with a gtx770 4gb. It's a very good system still for that kind of money and age.
Don't bother upgrading this dinosaur unless you overclock it (it's the last xeon generation that allows this so you should use it
). On basic air cooling, you can at least squeeze 3.8-4.1ghz with real ease on all cores with HT enabled even with entry cpu like the i7 920 / x5650. Probably 4.3-4.5 on a x5680. It's a world of difference from stock!
If you don't and if you're mostly gaming, then the 2 cores won't be of much difference. Resale value of X58 mb is pretty high, so consider this in your upgrade budget. Without oc it's very much wasted money and hassle regarding the ram issues.
If money is the main argument, I'd go for 12-16gb of non-ecc ram and overclock of the current cpu, if you haven't. Those 2 things will feel like a really good upgrade. Stock 920 is very meh nowadays and I have no idea how you lasted so long with 6gb on such a rig^^ Maybe throw in a used sata ssd for your games and stuff, it's cheap and will make your life nicer. If you don't mind spending more, the xeon run much cooler on overclock so that's always good and a 3650 will probably do. They will work with non-ecc but don't mix ecc and non-ecc.
Good luck!