No it is not your imagination. There are likely two reasons:
1) There is no equivalent to Southwest flying transborder to keep fares low
2) Taxes and fees add $100 to the cost of even an inexpensive ticket. As an example, with a $288 base fare YYZ-ORD-YYZ, there are the following additions:
Canada:
- Nav Canada fee $15
- Airport fee (YYZ) $15
- Security charge $8
- GST (sales tax) $22
Total $60

US:
- US transport tax $33
- Facility charge $6
- Security fee $3
- Immigration fee $8
Total $50

Ouch.