I can remember when I was routinely the only woman and by far the youngest person in the room for a business meeting, did lots of solo travelling outside major metropolitan areas (like tiny southern towns where the best hotel in town gave you a flyswatter when you checked in!), and frequently encountered groups of men from decidedly different socio-economic backgrounds from my own. (The mention of it being a union conference suggests OP was made uncomfortable partly by a blue-collar association, though I could be reading that wrong.)
Back then, I often felt decidedly the outsider. I can imagine OP's discomfort with the group, but while I think it was misplaced, only broader experience of people is likely to make her more comfortable in such situations.
The concern about being the obvious "killjoy" IS more realistic, especially if alcohol is present, but having acted on her initial discomfort, OP's stuck with that, and should simply double-latch the door.
When concerned about groups congregating in hallways, I use 11 pm as a litmus test...and the two times I have recently set my mental alarm clock to call the front desk at that time -- the groups dispersed well before 11. Generally, my 11 pm trigger means that it isn't obvious that I'm the one who has made the call to the front desk, though if I were to arrive around that time, I'd give it at least another half-hour - partly to weaken the assumption that I'm the caller.