OEM and no OEM are both LEGAL.
Whether the version is OEM or not is usually writen in the description of the product.
OEM were first developed for the PC manufacturers (they don't change their build and don't necessarly want to pay twice the price for a no OEM version which in any case is only available for one and just one computer).
Was is not LEGAL is to use the same no OEM licence for more than just one PC.
Legally, using your copy on your 3 personnal computers at home or delivering a mirror on a torrent website is basically the same: you do not own the right to use the copy on more than one machine.
The no OEM versions can be bought like every kind of software in store or in an online shop. You can at first glance see if the version is OEM or not: the no OEM price is twice the OEM price!
But I personally prefer the no OEM copy: more flexibility (and I need it regarding the frequency I change my rig...).
You can upgrade to 8GB, it isn't stupid at all. But don't except huge change in the global performance!
It is more a matter of being comfortable in recent games like BF3 and so on...
DDR2 begins to be quite old, I would say that you are at the limit were upgrading instead of buying a complete new rig is still viable (Q6600 is still a good CPU).
Don't use too much money in upgrading though. If you go for 8GB RAM and this GPU, it would be the best you can do to upgrade and still see a gain.
Maybe in two years, you will have to change to be able to play to the biggest games and keep the GPU (which will still be good) so don't waste too much right now
RAM stick have to be paired: two different sticks of RAM (different frequency/ timings/ manufacturers/ type of RAM, etc...) won't make it.
If you buy for example two times only one RAM stick.
Both stick do have the same specs, same manufacturer, capacity or whatever.... you MIGHT have no problem but it is not guaranteed.
So in your case: the success ratio of mixing both RAM is really low! (not the same manufacturer, so not exactly the same design, specs, ...)
If you go for 8GB RAM, you will have to buy two 4GB sticks for example which are paired (really often, RAM sticks are sold by two or four; it is made so because those sticks has been manufactured at the same time in the same place in the same conditions so are as symetrical as possible).
Not this thread but ask in a new post explanations about how OEM works for reinstallation/ modifications maybe.
Regardless of what we have said, don't expect any of this forum users to give you a trick to make something illegal (forcing an OEM version to be installed after modification is not really legal...).