If a passenger is denied entry for lack of proper documents, the carrier transporting him inbound is responsible for the outbound. This is considered a "must fly" and, on orders of the local authorities, a carrier will deny boarding to some passenger on a full flight to accomodate the "must fly". The carrier is subject to a fine, may be required to pay for the costs associated with security, e.g. detention if that extreme, and during more hostile times, the inbound aircraft held until the passenger is processed.
If the passenger is denied entry for other reasons not apparent on the documents, it is generally the responsibility of the country denying entry to arrange and pay for outbound transportation.
Most carriers' COC require that the passenger accept the financial responsibility for any adverse consequences to the carrier as it is always the passenger's responsibility to be properly documented.