Rooms at the Courtyard are pretty small, but it is easy to get to from Tokyo Station or Takaracho Station (if you get a direct train on the Keikyū line from HND). It is easy to get on the Narita Express train or take a bus to Narita from Tokyo Station on your way to NRT. Not sure what offerings are like now, but pre-COVID you would get evening apps and drinks in the cafe downstairs, and then breakfast in the restaurant. I also got a little bag with drinks and snacks at check in. No upgrade the two times I've stayed there, so make sure the room size you book will be adequate for the three of you.

I'll echo CPH's comments about the Marriott. It's a fine hotel, but the transit options are just far enough away to be annoying. There was a bus that went from Shinagawa station to the Marriott that would make it easier for you. It's a bit of a hike if you have to walk. Probably too much with a baby and luggage. Depending on where you are exploring you are likely going to be making the trek to Shinagawa or Osaki station to catch a train.

I haven't stayed at the other two options.