Looking at the OP's plan, I don't think (s)he will have any time for the "odd castle" on the way. And with this timing (Brussel late afternoon) (s)he'll spend 2 hours just trying to get in and out of Brussel (unless (s)he drives during the weekend). Even passing around Brussel will be difficult.

So take the train, or alternatively take the slightly longer route driving by Metz, Reims and Lille. Plan to have lunch in Reims. (Lille will be busy to get by, but not as much Brussel.)

If the OP really wants to visit Brussel, (s)he can easily do that from Brugge by train as a day trip. (And forget about Luxembourg... )