what slot do you have the card in??????? If it's the lower one I think It will default to 8x no matter if you have 1 or 2 cards installed.
Also I have Gtx 275 and have had zero problems with it or the gtx 260 that I used for about a year on both this and the 780g chipset. So no amd + nvidia do not equal problems.
Have you got the latest bios also dont think that would be a problem but who knows?
Also if you hit CTRL+F1 in the bios main screen it will open up a menu that says like advanced Chipset Options, try getting into that screen and look for a pci express screen and playing around with the settings to see if it changes anything. I know mine has something like primary dual card mode (should be uppermost option) and something like auto and disabled, I put mine on disabled because I only have 1 card, also there was a setting for PCIE Gen 2 x16 speed or something to that effect, and beside it were the optioins hardware switch or software switch, I put mine of hardware switch, besides that I disabled all other settings listed below it (which I think referred to the other pcie slot)
Also can remember the exact screen but one that allows you to set the boot order of hard drives or dvd drive etc, and gives you options to disable or enable onboard sound etc, Is a setting that says something to the effect of primary graphics adapter, and has like 4 settings, they are internal, pci, peg1 and peg. For the longest time I thought that should be at PEG 1 (thought PEG 1 was the uppermost slot) but when I read the book PEG was the 1st slot, and PEG1 was the lower, make sure that is one PEG.
Sorry I could'nt be more clear on names of menus in bios but If you find the settings I have told you about changing you have to be in the correct menu.