As a frequent flyer, I have learnt which of my shoes set off the detector and which do not. As a frequent observer of other passengers, I have learnt that very few seem to share this advanced knowledge of footwear. So I can completely understand why HAL insist on everyone taking them off, as it does very clearly cut down on the number of accidental detection triggers - which by necessity then require a security agent to come closer to you for a manual search, which increases the personal risk to them.
So the pertinent question really is why MAN has decided not to... and for that, we can only refer to the adage that MAN security is really a law unto itself.