GTX 260
Pros: Performance gains from driver support, PhysX support, partner company warranties
PCIe V: 2.0
DX V: 10
Power: 2x 6-pin PCIe Power Connector
HD 5770
Pros: Shorter card, lower power consumption, 3x monitor support, price
PCIe V: 2.0 or 2.1
DX V: 11
Power: 1x 6-pin PCIe Power Connector
--------------
In raw performance, HD 5770 512MB < GTX 260 C196 < HD 5770 1GB < GTX 260 C216. All of them are relatively comparable to each other.
--------------
Go with the HD5770. The GTX 260 cards used to cost around 140 USD 15 months ago. They're nearing 200 now because of Fermi's flop and dwindling supply as partner companies move on to the GTX 300 and 400 GPUs. Don't be a victim of supply and demand, AMD's evergreen GPUs are more efficient electrically, smaller, newer, versatile, and most importantly cheaper.