Here's my solution, which I'm sure people will find a million problems with:

Have 3 special lines with dedicated agents on each:
- "less than 45 minutes before before departure"
- "45 minutes to 1.5 hours before departure"
- "1.5+ hours before departure"

Someone at the entry to each line checks tickets/flight times to ensure no cheaters.

People are only pulled to the front of the line in the 1.5 hours+ queue (so only people who got there at least 1.5 hours before the flight and waited in line are guaranteed to get checked in). Maybe also in the 45min to 1.5 hour queue... but then would people bother showing up more than 1.5 hours in advance? My guess is that people would, but it should be tested.

If you arrive less than 45 minutes before a flight and are not able to get through checkin- tough luck, you miss your flight. The 45 minute line should have a big sign that says "You are not guaranteed to be checked-in before your departure if you arrive with less than 1.5 hours before your flight."

ok, let the criticizing begin...