Vacuum has nothing to do with the running of a tdi - vacuum is generated by the pump that runs off the cam shaft, not by the induction system.
If you had a leak between the turbo and the inlet manifold you'd struggle to attain boost, as it would be leaking away. if anything you might get black smoke due to not getting enough air to match the fuel injected - but i'd expect the leaking boost to also affect the pump, so you'd end up with less fuel, less air, and under performance as a result.
What are the glow plugs like? tdis have a good reputation for easy starting but if it's starting on 2 or 3 pots it could be those glow plugs have failed.
FWIW, I have had very similar symptoms to you, I in fact had 3 out of glow plugs failed. Ultimately, the belt shed a dozen or so teeth and I've been doing a belt replacement, so haven't had it running again since. I wonder if the belt had stretched at all, retarding the timing a bit and causing the cold start problem, like you, once it was running it was pretty ok.
I'd say, check the glow plugs, and who knows, maybe it's worth checking when the belt was last done, so you don't have to buy new push rods et al like i've had to. don't mean to scare you - could be completely unrelated.