The nvidia drivers involve two separate components, one source, and one binary. The binary contains the bulk of the driver and is precompiled. The source part is the wrapper that makes the modules work with your kernel. It's basically a way of releasing closed source software that works with different kernel configurations.
The binary modules are different for 32 bit and 64 bit systems. They are not compatible. You cannot interface a 32 bit binary module with a 64 bit kernel, and vice versa. On the nVidia site, when you download the driver installers, you get either the 32 bit or the 64 bit version. They are quite different.
Userspace programs work somewhat differently from kernel modules, which must very closely match the kernel.
X.org or other userspace components shouldn't matter to kernel modules. The kernel is self-sufficient and does not dynamically link to any libraries. I have no idea why your system thinks it's 32 bit and won't build the 64 bit nvidia modules. I've built and successfully inserted nvidia modules on systems that don't even have X.org. I suspect you did not correctly set up multilib.