I have a (female) Japanese friend who insists that it is in fact not necessary to wear any sort of underwear under a yukata, and that traditionally, this was not done. A quick Google indicates that opinions on this seem to vary.

In any case, I certainly wouldn't worry about wearing one with your usual underwear, if that is what you were concerned about. Sweats/gym shorts would definitely be a bit excessive, though.