Will an middle age guy like me look THAT out of place??

In most cases, no. But obviously maybe not for business meetings, or visits to some royal temples/sites (although you can 'rent' pants to gain entrance).

I'd suggest a shirt as well, preferably one with at least short sleeves.