As a former flight attendant I would say its polite to mention you will be swapping with spouse just to avoid some confusion (greeting as mentioned above or meals) but not necessary. I never would have said no it made no difference to me who is sitting in the seat. Then again I didn't work for ANA