provided you know the person you are taking the infant to then I can't see the problem

A couple of my friends who are families have taken award flights where one has been in F and the other in J (or else one has been in J and the other in Y when they couldn't get three seats in the same cabin) - they have swapped the infant (actually a 2 year old) for a couple of hours so the other could enjoy their meal/get some sleep/watch a movie etc with some peace and quiet. It was a little bit more difficult when one parent was in Y - less space - but they did have a spare seat beside them so it worked out ok. Don't think it would have been appreciated by fellow pax if there was no spare seat.

This would generally only work where the infant is ticketed in the higher cabin, as is your case. I don't think it would be possible for an infant ticketed in a lower class to spend time in a higher class.

A word with the crew in advance should be able to sort it out - can't hurt to try - but I guess it would depend on the amount of space and inconvenience to passenger in the lower class.