Keeping in mind the dangers of doing so:
1. To remove the cooler from the video card, simply turn the card so that the fan is facing down. Two clips hold it in place over the core. They extend through the card and out the other side (which you're now looking at). Pinch the little prongs together at the end and the pins should come out. They may or may not be spring-loaded, so they might actually pop out (be careful, you need them). After you get the pins out, I believe it's held on using thermal tape, so it won't just *come off*. I don't know how people get past this step, so look around the forum to find out how to do this. Twisting is probably not a good idea.
2. You'll need to clean off any old thermal compound. Use the highest percentage of rubbing alcohol you can find, and try cleaning it off with that. Industrial cleaners are not okay, and nail polish remover leaves a film. If there's still some left, you need to find a way to scrape it off without damaging the core, which might be difficult.
3. Use ASIII paste, not epoxy, on the core. Place only a very small amount (probably the size of the head of a pin with the little ball on the end, maybe smaller. I've never seen the core, so I don't know how much surface area you have to cover). Make a thin, smooth layer using a ZipLoc bag turned inside out over your finger. When you've got the whole core (not the chip, just the core) covered with paste, turn the bag right side out and seal it up and throw it away. Inspect to make sure paste isn't on any part of the chip other than the core. When I put paste on my processor, I always use a little rubbing alcohol on a Q-Tip to clean off every surface around the core after putting on the paste, just to make sure that no paste is where it shouldn't be. Then put the heatsink back on (or the replacement if you get one. There are a few aftermarket coolers, but the stock cooler is already very good) and line up the holes. Insert the pins and you're good to go.
4. If you take it a step further and remove the shim, when you get the heatsink off, you'll see that the core is a little recessed and doesn't make *complete* contact with the heatsink because of the shim that sits on top of it. This shim is to keep the core from getting damaged, so removing it means you need to be careful when putting on and removing the heatsink. I think most people use a hobby knife and just sort of carefully pry it up, but I might be wrong, so you'll want to look around the forum for directions on this too. PM somebody for specific instructions if you find a thread that isn't specific enough. Most people are perfectly willing to help.
Before you go to this extreme, add RAMsinks. It's much easier and damage is much less likely. As your RAM is what's holding you back, this is obviously the first place to start.
Z