Yes, I think you are hosed.
This is what I'd do: Hie thee to the nearest airport soonest.
Plead insanity if it is noticed. "It's all a misunderstanding" You were given to understand that as long as you confirmed or whatever it's OK. Don't say that it was you who gave yourself that understanding. After all it's the truth.
Pray the agent is not reading this thread.