On the plus side the Heathrow-Oxford bus you have your eye on runs like clockwork and is very pleasant to use; it drops you at Gloucester Green coach depot bang in the middle of Oxford central. Several hotels within a few minutes' walk. On the anti side there won't be much for you to do by the time you arrive but walk around and drink. The Ashmolean is worth a day in itself but won't be open late; as noted above most colleges, Radcliffe Camera, etc. will be closed (although you can still sneak into some of the quadrangles); etc.
Logistically perfectly doable but maybe not too fulfilling.