The question is going forward, do I still want to pay $95 a year for this? I know that free breakfast comes with the Gold status which is nice.
It depends on how much you value breakfast on your stays + extra earning at US supermarkets (1 extra HH point per dollar) and gas stations (3 HH points per dollar) plus spending at Hilton hotels (5 HH points per dollar).
Downgrading to the no annual fee card is a fine option if you only stay at Hiltons occasionally and have other cards to use for gas station spending. You have to spend a fair amount of money on the card (thousands, exact number depends on what you spend it on) to make it break even on the annual fee if you don't assign any value to the Gold status.