Yeah, I think removing them without a by-your-leave's a bit gauche. One would think that the same process to remove them could generate an email to the account saying "uh, we had to clean up your AS account- you might want to check and see that everything's there that's supposed to be".