if you use AS as a lubricant during the final lapping you still get a little alu and grit in the small imperfections. I suggest (haven't tried this) cleaning the HS with water (or alcohol) after the final lapping and then doing it again with AS, but not on gritpaper but on a piece of glas. You won't be making the surface any more smoot, but you can be sure that there is only AS in the small imperfections.