The issue you're going to face is distance. Toronto's airport is actually not even remotely close to the downtown core, and the 30-45 minute estimates are sadly accurate. It's about a $40 cab fare one way, by the way.

The Airport Express, as mentioned above, will still be costly - around $40 roundtrip. Your better option may be to get to Yorkdale Mall and hop on the subway. That line will take you almost to the front door of the HHOF and it's going to cost you about $6 roundtrip... plus transport to Yorkdale mall, of course. Total transit time is the same - about a half hour.