You joke, but speaking more slowly actually helps many non-native speakers understand you, depending on their level of fluency. I've facilitated meetings with global audiences and received feedback that they appreciated the fact that I deliberately spoke more slowly and enunciated more clearly than my colleague. Not so slow that it's insulting, mind you, but reducing the speed does help.

On topic, what is the rationale for whispering? I've never whispered when holding normal conversations in foreign countries and don't see the need in regular circumstances.