The only thing I can caution you of is that there are generally limits to lap babies based on the availability of the oxygen masks. Assuming that grandma and grandpa are sitting next to each other in F, there probably won't be another baby in that row, so they should be OK.

There have been posts here before about the swapping, specifically on flights to Hawaii, but yours isn't much different. The general concensus then was to do it, but be polite, and warn the FA in advance so they don't freak out.