I think Ghent is best from a centrality standpoint for the major places you want to get to in Belgium. It's also a nice balance between the much larger (Brussels) and the cozy and small (Bruges). You can also take the train direct to BRU without having to transfer in Brussels, which will probably be convenient / handy with a baby and luggage.