I'd get fasttrack unless you've got Vietnam Airlines platinum status. I found HAN FAR easier than SGN both times I've gone in (out was a slightly different vibe) but equally, it's Vietnam not Singapore so you're subject to the usual stuff. Last time I was leaving HAN, they did a changing of the passport officers where they all stopped work for a few mins, straightened their uniforms, waited for replacements, saluted etc. It was all very ceremonial but very annoying as I was getting nervous about being on-time (in the end I was bumped to the priority line thanks to status, but it depends on luck I think).

HAN is quite a long way from the centre of the city. I enjoyed walking around Hoàn Kiếm Lake, looking around the temple and having a bite to eat around there on my last layover.