Unless it's a buffet, there is nothing wrong with taking leftovers home at any restaurant in the US, regardless of how fancy it is. Other countries may vary.

I personally always take my leftovers. If I am staying in a hotel with no place to store them, I may give them to a homeless person, or eventually throw them away myself, which is what the restaurant would have done anyway.