Petra and Wadi Rum are well worth a visit. Both are easily accessible from Amman.
Your assumption about Amman is correct. It is not a great place to spend a few days lounging about.
If you have more than a few days in Jordan, I would suggest getting a Syrian visa and taking a service taxi from Amman to Damascus. There is plenty to do there.