I haven't looked at schedules recently, but about a year ago my Mom was trying to carve out a Morocco trip while renting a house in Portugal. She had a heck of a time scheduling it because of flight logistics, and realized she would effectively lose 3-4 days to travel back and forth to Morocco from Portugal. So you may encounter similar problems.

Morocco is a fantastic country, but I think it easily deserves ~2 weeks if you want to see most of the major cities. If you're thinking about just a 3-5 day trip, I'd suggest you do one city or two, max, and skip the Sahara this time around.