What's hit quote, gone fishin? Isn't the attatchment you're referring to an arbor? That is basically what the drill chuck fits on to. It's what I've got a problem with at the moment because the chuck I bought to replace the bust one has a steeper taper, and so wobbles on the drill press. Even though the old one didn't wobble at all, it still worked loose, because moving the work sideways cause the chuck to gradually come off the arbor.
To counteract this, you'd normally use a screw thread fitting, or on mills, as far as I'm aware, you use a collet chuck, which grips the mill bit along the length of the shank - the same way as the dremmel.
What's happening in both loner's and my own setups, is that the play inherent in the drill and the vice we're using is causing the juddering, and the mill bit is grabbing the work and biting into it causing the vice / drill chuck to move. You aren't going to completely eliminate this unless you lock down the vice to stop it moving AND tighten the bearings on the drill. Or get a milling machine. A square column milling machine is made to cut out these inconsistencies, rather than the drill press, which in essence is made only to apply pressure vertically. Sorry if that's overstating the matter, but I've looked into this and tried to find a way around it now for the last 6 - 8 months or so.