I totally understand, I am from the UK and live in Canada. My parents are elderly, and father has alzheimers, mother is struggling to cope with him. The temptation to go visit is strong, but quarantine is two weeks on either end, so I am letting my UK based brother handle things, even though I have more free time than he.
I do not think anyone can decide for you, but just wanted to say I empathise.
I have gone back to the UK at various points when my kids were tiny, they always handled the travel well. They are usually happy as long as you are with them. I hope your decision becomes clearer.