The "no additional operators permitted unless authorized" or similar wording on the rental contract could be construed to invalidate the contract if anyone not listed is caught driving. Just like any other contract violation - racing, towing a trailer, driving in to Mexico without proper coverage, etc. - this could expose you to assorted liabilities over and above insurance issues.

IMHO, it's not worth it to save a few dollars (or Euros, yen, whatever). In the context of the total cost of any trip where one is likely to need a rental car with more than one driver, this fee is surely a trivial amount.