I have no knowledge of how much people moving would affect trim, but on exceptionally quiet flights I've sometimes heard the crew have ask people to stay in their allotted seats for trim reasons until the seatbelt signs go off. I guess its one of those things where even though one person isn't a big deal, the procedure is there to ensure a mass migration doesn't go unnoticed! We all know pax can be referred to as self loading freight. I guess we could also be considered portable ballast!