Agreed, I keep mine fastened so that I can move around in the seat, just enough to put my hand under it but not much else. I don't want to be strapped in so tight that I can't move at all, but I realise the importance of being restrained in the seat if something untowards happens.
After all, why have it done up at all if you can fall out of the seat with a bit of CAT?
Dave