If you're planning to spend all your nights at the Toyoko Inn at/near the airport, I suggest changing your plans to spend at least a couple of nights at one of the Toyoko Inns near Nagoya Station. The Sakura Dori side of the station has two of them about a 5 minute walk away, on opposite corners from each other. The neighborhood has a bunch of restaurants, a few of which I've tried and found to be good, one of which is now my favorite place in Nagoya (as much for the atmosphere as for the food). I also recommend some Nagoya chicken wings and a beer to go along with them, both of which can be found at countless places.
From Nagoya Station you can hop on the subway and trains to explore the area. Nagoya Castle is interesting enough, though it's a replica since the original was destroyed in WWII. Inuyama Castle, mentioned above, looks interesting, perhaps more so than Nagoya Castle because it's older. The
Toyota Museum is worth the trip if you like cars and have a few hours. The aquarium is OK - not super but far from something to avoid. A train trip into the mountains (Takayama, etc.) will get you some nice scenery.
Nagoya isn't the most exciting/interesting place in Japan, but I've spent enough time there that it's grown on me a bit. If this is your first time to Japan you'll find plenty of interesting stuff just by wandering around and soaking it all in.