I just check in at Business Class and it takes about a minute.

If by immigration you mean US pre-clearance , other than the 6 a.m. rush it is no issue at all - I last flew out on the 9:30 MIA flight and waited zero minutes for INS, 1/100th of a minute for Customs, and 2 minutes for our version of the TSA.

On return at 5 pm Canada Immigration was empty and fully staffed. First in line, 3 minutes, Customs zero minutes.

I arrive a lot at 10:30 - 11:30 p.m. as well with a wait time of maybe 5 minutes.

So your generalization besides being vague is not always right.