Here's my limited understanding from various readings on the Internet over time. You or anyone else, feel free to correct if I'm wrong:
Theoretically, faster RAM draws more current, because I=C dV/dt, so dV/dt = I/C. So in order for voltage to change fast, you have to either increase current, or decrease capacitance. Here's a good article on how much current DDR SDRAM can draw, which can be quite significant:
Overclocks article 696
So this is a possible hypothesis of why your system was rock solid when you ran it at DDR266 but unstable when run at DDR333 (because DDR333 draws more current than DDR266.)
As for VIO, my understanding is: CPUs these days run their internal circuits at a Vcore that's lower than 3.3V to minimize heat generated (eg. the Athlon TBred B runs at 1.6V Vcore.) However, other components eg. chipset, memory, are manufactured independently so there needs to be a common voltage for communication between the CPU and these components. VIO = 3.3V is the common voltage that the CPU uses to send signals to other components. So, basically, within a CPU, the internal circuits operate at Vcore, while the signals that travel in and out of the CPU (thus the name VIO) operate at VIO.
There's a component on the motherboard called the Voltage Regulator Module (VRM) that converts either the 5V rail (for most Athlon motherboards), or the 12V rail (for all P4 motherboards) to the two voltages Vcore and VIO required by the CPU. Thus there are actually two stages of voltage regulation between the power supply and the CPU, one at the power supply unit itself, the other at the VRM. (This is the same VRM that you need to buy when you add a 2nd CPU to dual processor boards.)
You said that lowering your CPU multiplier from 12.5 to 12 seemed to improve stability. One possible hypothesis is maybe this lowers the current drawn by the CPU, and thus allowing the DDR memory to draw the amount of current it needs to achieve stability. This is because on Athlon motherboards, the CPU and the DDR memory share the same 5V rail. If it was a P4 system, the CPU would use the 12V rail, and won't have to share power with other components that use the 5V rail.
In terms of overclocking, increasing Vcore or VIO both might increase stability, although Vcore more so than VIO I think.
So I would suggest trying another power supply and see if it helps. By the way, you don't want to increase the voltage of the HyperX more than 2.8V. Someone over at Anandtech forums reported burning their RAM at 2.8V:
Anandtech forum thread on PC3000 HyperX memory
Cheers,
Dave