I've seen delays of a few minutes after First and Business until they call elites. The reason seems to be to let the front aisles clear before they let in a horde of folks who just want to get past them to the cheap seats. (Seems reasonable to me.) So, if they've boarded the front cabin(s) and haven't called anyone else, I tend to wait. Most of the time they do call elites first, even if it's not right away.