Enter and leave a country on the same passport. That's all.
So this time you enter with Canadian and leave the US on that same Canadian.
You can never anticipate what US Immigration may do, though, and while there may be folks here who have traveled with two passports, I suspect few of them got a new one from a completely different country during a trip, so your specific question may not have an experienced reply.

The airline may want to see both passports (if leaving Malaysia on the British passport you entered with) to make sure you can enter the U.S. meeting proper visa requirements. (They don't like sending you back to your origin point at their expense if you can't get admitted.)