Basically, identity theft on steroids is probably the worst that could happen to you.. which is nothing to sneeze at. Here's what I mean : someone skims the data off your e-passport, goes home, decrypts it, and records all the data on a new, blank passport. The attacker places his own picture on the passport, but all the name data and all that stuff that is printed on the first page is you. Obviously, the information stored digitally in the passport's chip is also pertaining to you.
Attacker travels overseas, commits crimes, etc. in your name. Or, attacker could be a terrorist who enters a country legally when his real identity would have been red-flagged.

It's not science fiction either - passports are already being cloned. The most difficult part would be obtaining a genuine blank passport, but then in some countries these have been bought or stolen.

The only way really to foil such an attack is when fingerprint scanning or IRIS scanning becomes mandatory for everyone, at all ports of entry, in all countries... which is probably only a matter of time. Until then though, the scenario I depicted is very doable.