Assuming you chose the room upgrade as your VIP benefit selection it seems that this Hilton is following the HH rules:

** Executive Lounge Access Policy

If you select a space-available room upgrade as your on-property VIP benefit, the following policy will apply effective August 15, 2008, at all Hilton®, Conrad® Hotels & Resorts and Doubletree® hotels:

If you are a Gold VIP, you'll be given access to the Executive Floor lounge when you receive a room upgrade to the Executive Floor. If an Executive Floor room upgrade is unavailable, you'll still receive daily complimentary Continental breakfast in lieu of lounge access.

Found here:

http://hhonors1.hilton.com/en_US/hh/about/gold.do

So your best bet would be to ask them politely to not give you the upgrade (switch rooms again) so long as you get the breakfast you want. They may refuse you, however, because your VIP benefit was pre-selected by you and they don't have to let you change it at will. From the HH website:

"You can update your preferences up to 24 hours prior to check-in."