It's not so much that a cross side is no good for milling, but it's the combination of a cross-slide vise and a drill press aren't suitable for milling. Why? 3 reasons: (1) Drill press bearings aren't designed to absorb radial loads, only axial and (2) the Jacobs taper on drill press chucks will not withstand axial loads at all. They come loose. Trust me, when this happens, you don't want to be around. (3) Cross slide vises are very inaccurate and have a lot of backlash, which can be very dangerous when milling.
The best type of vise for milling is a sturdy, precision ground vise (like a Kurt.) And they work best when used with a machined designed for milling, AKA a milling machine.
The best type of vise for milling is a sturdy, precision ground vise (like a Kurt.) And they work best when used with a machined designed for milling, AKA a milling machine.