I think that if you left and your colleague stayed s/he would be fine unless someone noticed you leaving and somehow remembered that your colleague shouldn't be there on his own. Tbh, I wouldn't worry about it.