I think the embassies are just being overly cautious. They probably figure that even if you are eligible for VWP, there is no harm in getting a visa, so if there is anything suspicious or questionable in your past it's safer to just get one. To them, it's probably too much hassle to figure out which arrests and convictions disqualify you and which don't, so they just tell everyone to get one. They don't want to be held responsible if you get denied entry. The fact that CBP has a reputation in some circles of treating foreigners in an arbitrary and heavy-handed way probably fuels this sentiment further.