I drive in London the whole time and find it to be fine, although it helps that I know where I'm going and how to avoid some of the slower spots. Friends from out of town almost never drive because they can't park and don't know the roads well enough to avoid the slowest roads.

Note that you only pay the Congestion Charge if you don't move the car during the charging hours.

If you really don't need the car in London then it's likely to be faster and less stressful to abandon it outside London. That was my big learning experience when I nonchalantly drove int NYC once- how difficult can it be in a grid system? Very, it seems.