Malaysian are Oneworld but BA will not, 100%, through check bags unless it is all booked together on one ticket. This means you'll have to pass immigration, collect your bags, recheck them and go back airside. Family that live in KL and travel in and out reguarly allow an hour for inbound immigration and bags, so you don't have a huge amount of buffer in the event of a delay to your inbound. Personally I'd stay overnight at KLIA and fly the following morning. Why make things more stressful than they need to be?