Agree with the others about DUS -- get out of town and see the Dom in Köln, grab a quick lunch and head to your next destination.

And nothing against Amsterdam, just not the first place that pops into my mind for castles. If you decide to go north instead of south, I think three nights in Amsterdam and two in Gent would be appropriate. That gives you the option of spending part of a day in Bruges if you so desire.