AMD OR INTEL?

Which of these would perform better?

i5-760 @ 3.8ghz or AMD Phenom II X6 @ 3.9ghz?

Depends on what you're doing.
Something that uses more then four cores, the AMD.
Something that uses less, the Intel.

Overclocked i5 vs X6 @ 4GHz.

http://www.xbitlabs.com/articles/cpu/display/amd-phenom-ii-x6-1055t-overclocking.html

As Bobnova said, it depends on the application or benchmark. In most cases, highly threaded software will be faster on the X6, but lightly to moderately threaded software will run faster on the i5.