Well, I did remove the pads. I tried the HSF without any thermal grease the first time, and my temps had dropped about 8 degrees C just from removing the pads. Then I applied the thermal grease (to the core only) and it made my temps drop only about 2 degrees C. So, removing the pads was a big help in cooling my CPU down.
I also bent back the metal clips on the HSF a bit to take some of the pressure off the CPU to make sure it won't crack the core.