In September, with the wife and kids, we flew to Malaga and rented a car. We stayed in Marbella for a couple of days. We got up Sunday morning and went to Gibraltar where we spent a few hours. We than drove to Seville where we arrived late in the afternoon.
We felt that 4 hours in GIB was plenty. If you wanted to squeeze in an extra day maybe you could to a side trip to Morrocco. We chose not to because of my young kids.