Well, if you're not going back to the restaurant ever again, eh, just stiff 'em ...just kidding.
Just ask to break the twenty. Yes, it's kind of stupid to bring change like that but if the service was decent it would be pretty mean spirited to just tip three dollars for which there could easily have been a number of reasonable explanations for the denominations. On the otherhand, if this just topped off a really poor dining experience overall, I'd say sure, consider tipping the three.