There is a process for agents to follow in these circumstances which is broadly similar to that used for gold card holder tcp seating and it very disappointing that this has not been followed.
The agent needs to cross reference your booking to that of your wife and daughter and send to back office for allocation of adjacent seat. Obviously this is subject to that seat being available but you have indicated that this is so. Please be aware however that if the adjacent seat is also indicated as a bsct position, which it may be dependant on aircraft type, then this cannot be allocated as it would potentially deny a bsct seat to another adult with infant.
Please re try and ask agent to check on procedure for tcp infant pnr.