Many countries would have deported them (back to country of origin, UK in their case, and not just to Australia) ... it is a very common requirement to either have proof of onward travel (not just the intention to do so!) or a visa that allows entry without such proof; they had neither. There are dozens of threads on FT over the years of families that have been deported prior to entry for their vacation when such open-jaw trips were to the wrong country -- it is not rare, particularly in Latin/South America and Africa (though often a bribe settles the matter). Technically BA was supposed to refuse to board them at EDI (and NZ could have required BA to pay for their transport back to UK); luckily for you the EDI check-in is quite sloppy all the time (something I know well from my few trips there).