Japan has no GSM service. Your phone will not work (nor will any cell phone currently sold in America).

A 3G WCDMA phone (such as one from Europe) will allow you to roam in Japan. You can rent one of these at the airport and then put your AT&T SIM card into it, provided that you have international roaming enabled. Expect to pay insane fees of $2 a minute or more in addition to the handset rental fees.

Prepaid SIM cards are not sold anywhere in Japan. You either have a 3G phone with a SIM card that is on a contract, or a PDC (Japanese 2G) prepaid or postpaid phone. The PDC phones do not use SIM cards, so you will need to buy a new phone along with your prepaid service.

In addition, officially, prepaid phones are not sold to tourists, only to those with an Alien Registration Card (ie, someone not on a tourist/transit visa). However, many places don't bother to check, and you can get a phone using only your passport and a credit card as ID.

In any case, your AT&T phone will NOT work in Japan.