The vBulletin search engine -- which doesn't scale very well -- is unable to cope with the current size of the board. As you note, it is possible to create search engines that scale better. We've done actually done that. Unfortunately the new search is radically different on the backend, so we can't just "plug in" features from the old search and make them work. Everything has to be redone from scratch. Moreover, some of the things that were easy in the old engine are tricky in the new (and vice versa, but that doesn't cause any problems).

Short version is we're working on it.
Kevin