Not sure why you say you don't want a KVM - I suspect that you may have been put off by the kind of thing Belkin, IOGear, etc make. A good one will do what you want with not problems, but will cost you a fair bit of meney - here's why:

Switching USB2.0 with a mechanical switch is a no-no. You can just about do it and make spec using very high quality analog multiplexors if the switch is very close to either the upstream or downstream ports with carefully matched signal traces to the upstream or downstream chip. Some blade server backplanes do this. The only right way to do this if you want to plug in cables both upstream and downstream is to build a hub with 2 upstream ports - which would require the use of an FPGA with external USB transceivers, as there is no "standard silicon" with this feature, and technically, such a product is not compliant with the USB spec (which is silly, but that's the way it is).

A similar problem applies to the video - for a quality signal you are going to need a video buffer/switch chip inside the KVM box - and if they have engineered it right you won't be able to tell the difference between the KVM and what you have now. The higher quality KVM boxes have this, the cheap ones don't. Better still would be a DVI switch, but I haven't seen a KVM with DVI switching capability.

You probably won't want to spring for the kind of prof. quality KVM that Avocent make at "new" price, but you may be able to find one on ebay at an attractive enough price to make it worthwhile.