i wud bump on the memory first then get on with the core. oc-ing the memory offers more bandwith, and of course bandwith smoothens up games. also, after uve maxed out on the mem, try bumping the core as high as it kud go, stable. good luck.
on SDRAM video memory the memory clock will usually influences performance better, while on DDR along with a powerful GPU like the geforce3 and 4 both do.
if youre looking for info about one specific card i recommend you to take a look at the anandtech reviews, since they usually do that kind of comparison.
the memory probably....but youll still have one heck of a time getting GTA3 to run smooth on that card at any res at any setting....maybe you could get UT (original) to run smoother by ocing both...but newer games really struggle on the TNT2