Seriously, 512 MB seems like it would be enough for even the most involved document. He hasn't done anything weird like disabling the swap file or something, has he?
I haven't done any forumula editing in WordXP, but I have had no problems with using the editor in Word2000 (or previous versions). I'm wondering if there are some compatibility problems with the formula editor between versions. Was the original document produced in a different version?