The gate agent's behavior was unacceptable.
The FAA rule says "A condition or responsibilities, such as
caring for small children, that
might prevent the person from performing one or more of the applicable functions listed in paragraph (d) of this section; or "
He's not caring for an infant in a different class of service, and the mere fact that said infant is pretty far away from Dad (i.e., Dad won't be holding the baby, I'm sure Mom will volunteer to do that) means that some witch gate agent using the "might prevent" clause to boot you out of the exit row is just as asinine as "Anne can't sit in the exit row because she's preoccupied with her 12 cats at home and who is going to feed them if she gets killed on the flight"
Why is common sense so rare?