I think your expectations are way high. UBER classify XL certain high end cars that meet their criteria. That same car can also function in X, poo, and express poo (driver terms for the service) as well if the driver choose too. The driver qualifies for UBER based on its background check. The requirement for English speaking and "high end" conversation is not mandatory to become an UBER driver. Drivers can certainly do their best to try and please the rider in order to perhaps get a nice tip. In the end, you are booking a ride to take you from A to B, the UBER app shows you the car and driver that will be picking you up, you have the option to cancel and get another one at no cost if done timely, just like the driver can have the option to cancel once he sees you stumbling out of a bar and he does not want to risk dealing with a rider might puke on his car. XL is geared towards higher end market that might want an exclusive high end service. The bus/taxi/shuttle driver does not to be conversationalist in order to get and do the job, however it does help to impress the passenger and perhaps get a tip.

If you felt disappointed on the car, you could have canceled, if you felt the driver should have been perhaps a bit more attentive, you should send a complaint to UBER detailing what happened, however, even if you do so, UBER will not advise the driver on what happened, all he will get is a "professionalism" notification saying that a rider was not too happy with him, but no further detail regarding the incident. Feedback is key, and UBER lacks a bit of it. I do agree with you that an XL driver should be a bit more attentive as passengers are paying a higher fare for the "experience". I myself have seen several cars that I'm surprised are being used by drivers, in particular a Tesla in ORD, and a late model corvette as well. As a driver, the XL ride will pay more, but at the same time there are not that many XL requests as opposed to X. You do not make money while the car is parked waiting for pick up request, hence while I might qualify for XL, I'll take X and offer the same service and attention to the rider.