I've just re-built my front axle, using far to many new bits.
What I have found is, yes you can take play out the swivel housing by removing shims, but I still had a little bit of play, did some measuring and found that the lower king pins where worn, thus the straight pin was wobbling about inside the lower bearing race, the lower pin is not tapered and is just a slide fit onto the tapered part of the lower bearing. By removing the upper shims you are just squashing the bearing (admitadly that will create more resistance to the sloppy lower pin, but isn;t solving the problem) I've just bought 2 new lower pins to sort this.
Mav