The sad part is that your 9770 CPU is good and probably adequate for what you want to do. It's the motherboard that's letting you down and probably not upgradable because they aren't making AM2+/DDR2 boards anymore. Your only option would be to look for a good used 790FX AM2+ motherboard. Otherwise you will have to upgrade CPU, ram and motherboard to AM3/AM3+ and DDR3 level. But you are wise in starting with the PSU as the first upgrade. Go with a good name brand like Seasonic, Antec, or Corsair if they are available to you and get one with a rating of about 550-600 watts.
And what about your case? A good roomy case with lots of fans for good ventilation is another foundational piece for successful overclocking.