If the agent says you can upgrade and you go ahead in good faith, then if there are any problems it's up to BA to make it right.
How long do you have before travel? A longer time would certainly give me more confidence to chance my arm.
However the terms of the voucher clearly state it can't be used for F.
You can't upgrade an Avios booking using Avios except at the airport. Instead you can change it and attract the change fee which means re-issuing it. That's a new reward booking and cancelling the old one, which means it doesn't meet the voucher criteria. If the agent cancels your J and then fails to book in the F, then they should know about it straight away and be able to reinstate your J reward.
But to be honest I think if you ring up again, they'll say you can't do it anyway.

You said you had an Amex voucher. Why not book the F with that and cancel the Barclaycard J?
If you don't have enough Avios to do this yourself without cancelling the J, then they can do it for you on the phone.