Agreed. Unless you're actually using the metal to store heat, it's thermal conductivity not specific heat that matters.
For a given cross section, pure copper will always outperform aluminum, but for a given mass, aluminum outperforms copper. Hence copper is always better if weight is not a consideration, but aluminum is useful to get enough surface area for cooling without going over weight. This is why some of the best CPU heat sinks and automobile radiators are made of aluminum. For our purposes, though, copper is superior.
Silver offers slightly less thermal resistance than copper, though its relative weakness and high cost are serious drawbacks. Diamond is a bit pricey as well, though that will probably change in our lifetimes as no amount of De Beers thuggery will get the synthetic diamond genie back in the bottle. The manufacturers of a diamond-copper composite called Heathru claim it is cost-competitive with existing materials, but they don't say which materials.