I suppose your plan works but you may find a problem crossing cabins in mid-flight. In recent travels I have seen people try and trade seats, etc. to no avail.
If your infant is well behaved no worries. If he/she is going to fuss a lot PLEASE don't sit in biz or first.