Basically, memory is in sync with the processor. For each CPU cycle, there will be a cylce of memory. Say your memroy is rated DDR200, that means it's default timings are 100 fsb. If your 1.7ghz mhz runs default at 100 fsb, then the memory and CPU line up nicely.
However, your memory is rated at DDR266, which means it lines up nicely when the CPU is at 133fsb (you've actually been underclocking your memory all along). When you upped the fsb of your p4, you also upped the frequency of your memory (both would be 133 fsb). So now your memory is running at DDR266 speeds.
Where memory selection comes in is in the rated speed (of the memory). You'd have a hard time getting up to 166 fsb with DDR200 memory, since that would be DDR333 speeds. However, if you put DDR333 in the system, that memory would be perfectly in sync with your CPU's overclocked frequency at 166fsb. Hope I haven't lost you.
Furthermore, you relax the memory timings to push the memory further. That's basically allowing the memory to not work so fast moving information around, which will allow the memory ultimately to work at faster frequency. This is what you'd have to alter if you didn't want to buy new memory (and it wold be iffy since 133 to 166 is a considerable jump).
This can get more complicated when you throw in CPU/Memory ratios, which is where you can underclock memory in order to get more fsb.
