There are two general rules

1) you leave a country on the same passport you entered it.
2) if you are a citizen of a country then you must enter & leave that country on a passport for that country.

So you leave/enter the UK on your British passport and enter & leave Malaysia on your Malaysian (though you might not even have to do that if you stay airside as you won't formally enter Malaysia).

Otherwise it's up to you to decide which passport to use - generally the one that gets you easiest entry / cheapest visa / longest stay...

And if you have any problems you just show your other passport