My empirical evidence is that I never get asked when travelling with the family, but very often get asked when travelling alone. I just assumed I look more shifty than the rest of my lot.

Having said that, I guess I am first off the flight a lot more when travelling alone so Lolbert's suggestion is consistent with my experience....