There will be strong opinions on both sides, but here is my $.02 based on my experience with both.
I use both of these depending on location. Sometimes IHG is more convenient and sometimes HH. For US domestic travel I notice little difference and the elite benefits are minimal beyond additional points earnings. If traveling internationally, I like the HH Diamond treatment in full service Hilton properties. Unless you are Ambassador and use IC hotels, IHG seems to have lesser appeal outside the US.
So if you use and value international redemption I would stick with HH. But if you are primarily a US traveler keep the IHG and cancel the more expensive Hilton Amex.