Agreed. In the few instances (what a game or two?) That is hurts, it's just not worth it for the other benefits...and you bought a 4c/8t cpu, not 4c/4t...get the most out of your money.
My suggestion is to stop babying your system. Give it all threads, I wouldn't worry about offset (let window manage your power) voltages, none of that. Just set your voltage you need for the clock and go!