We continue to analyze the speed thing. Came across this on a Board that serves admins of the bigger boards. Given that there is a "search tool" war going on now - all these new search engines - not just Google anymore, it seems this may also be contributing to our problems. Remember, we do have nearly 4 million posts here.

Here's comments from a discussion by owners of other large bulletin boards.:
Some robots are well behaved and gradually cache sites. Others are killers. They jump in and start hammering away as fast as they can, eating up my entire T1 line and basically making my system unavailable to legitimate users. While there are some advantages to letting the robots in, my feeling is that they do more harm than good. So my robots.txt file basically tells 'em to get lost.

I'd keep google... I've seen some new spiders, gigabot, that hammer my site. I'm thinking about blocking some of the less-known spiders such as gigabot and only allowing Yahoo, MSN, Google and AltaVista.

MSN bot is the killer on my forum. Every hour never fails to come.


We'll tinker with this slightly to see what happens perhaps as one Board operator has done and lock out gigabot. With A3 and MSN now really in the search area it does present more problems for us.