The flush lever on the toliet doesn't bother me in regards to germs. Why? The first thing I do after touching it is wash my hands. The doorknob and remote are much more likely to transfer germs in my opinion.

When I go on cruises (where norovirus is rampant) I take disinfectant wipes to clean the doorknobs, remote, and telephone before I use them. I don't even think twice about using the bathroom "as is."