Take it off when eating or drinking of course
You're not supposed to do that either. The idea of "mask kaishoku" is that you are only supposed to expose your mouth literally at the moments where you are putting food or beverage into it, to the point where the mask should always be attached to at least one ear.
Yes, I do do this. And yes, I am almost always the only other person that I see doing this.