I, sadly, have to do this all the time.
You see, i upgrade my wife because she HATES flying with our two kids ... but sometimes this doesn't work out well and we end up swapping seats and back in coach she goes with the kids. oh darn.
I always ask, but FA's have never cared once. Actually, an FA once asked me if we were on the same itinerary (making sure i wasn't some random coach guy trying to steal a first seat) .. which i replied yes