Oh man, I heard lots of horror stories about OC system thru softwares. Its always thru BIOS my friend, just to be on the safe side.
People OC hardware thru nTune and stuff sometimes ran into deep poo-poo holes.
Careful.
Yes baby steps is the way to go instead of one giant step, this case you will know your last stable step.
That 8800 is probably overloading your 450W PSU, your instability is probably coming from the PSU, being on the lowest level of what those 8800s require.
Good Luck.