A t-shirt outright is pushing it. I would consider one that has logos/pictures on it (e.g. like a "I Love NYC" shirt) to be tacky, but a plain solid-color t-shirt would probably pass muster at a place like Gramercy Tavern, if you were sitting in the tavern room.

Pants - as long as you're not wearing distressed jeans or cargo pants, you should be fine.