to the forums!
Well.. to be blunt, what you're doing wrong is trying to OC via software. Don't let the BIOS scare you.. it is in fact much easier than most people realize to OC in teh BIOS.. and it's what everyone on this site recommends doing if you have a BIOS that allows you to.
First off, would you kindly create a full system spec sig? It helps us to know all the hardware you're running and trying to OC.
Second, download a few programs to help you monitor temps and stabilize your system. These programs would be CPUz, HWmonitor or RealTemp, and either Prime95 or OCCT. I use HWmonitor and Prime95 myself. HWmonitor will show you lots of information, like voltages, CPU and Mobo temps, and most importantly it will display your 'Core' temps. This is the most important thing to keep an eye on and when running an OC on any CPU. And with the Phenom II try not to let it go past 55-56*C as this has been determined to be the temperature ceiling for these chips.
Third, I would give Dolk's Guide to the Phenom II a good read through.. it has tons of information in it and is THE guide to follow to OC a Phenom II CPU. You can find it in the AMD CPUs Ultimate Sticky Thread at the top of this forum, but I'm going to provide a link
HERE.
After you have given that a good read through, post back here with any further questions.