Do you have a pre-applied TIM pad on your HSF? If you didn't scrape one off, you may very well have one on the bottom of your HSF right now vs. the bare metal-to-silicon scenario your are suggesting. If you are running bare metal-to-silicon, then I guess you need to ask yourself how much better your PC would run if you had some sort of TIM between the mating surfaces to fill in the surface imperfections and gaps, eliminate the air in the thermal junction (which always impedes heat transfer) - something with thermally conductive properties (unlike air) that actually promotes heat transfer from the "hot" surface to the cold "surface."
Of course, if you're happy with the current performance of your PC and you don't need or want to implement a cheap tweak to make it run cooler, faster and better, then I guess you're all set.