Does anyone have a link to the relevant legislation that requires passengers to obey crew? As I'm in the UK, Google is mostly giving me UK-centric results. For example the relevant UK legislation states "Every person in an aircraft must obey all lawful commands which the pilot in command of that aircraft may give for the purpose of securing the safety of the aircraft and of persons or property carried in the aircraft, or the safety, efficiency or regularity of air navigation". Obviously UK legislation is irrelevant, but I would be very surprised if the equivalent US legislation required passengers to obey absolutely any command by crew. For example, passengers would not have to obey a command to undress and run down the aisle naked. There will be limits as to purpose of commands that must be obeyed, and I'm guessing that leaving the aircraft for the purpose in this scenario might not be one of them.
Your assumption is correct. FAA rules require that passengers comply with any crew member instruction. The pilot can ask you to leave the plane for any reason or no reason at all, so long as it is not based on unlawful discrimination based on protected classes of race, gender, age, etc.
you can google around for FAA regulations passengers must obey crew instructions or something similar, it comes up a lot in everything from how much alcohol can be served to storing overhead luggage.
https://www.faa.gov/documentLibrary/.../AC121-24C.pdf