ClueByFour: Very interesting post; I wonder if a number of us working together might be able to figure out something here.
To add a couple of points to what you wrote:
Is this really a scalability problem? Is it slower because of more activity or because of a problem with file corruption? Smells to me more like a software bug than somthing adding more RAM would help with, although in general of course that is a good suggestion.
I've observed that nFT seems to be doing a lot more in terms of page updates than old FT. For example, when I bring up the front page of FlyerTalk miles, it always refreshes, and thus I lose the light bulbs right from the start of a session. In the old days, it seemed to do this much less often. That "feels" true for other pages as well. Refreshes always seem to take a lot longer on the new FT. I wonder if some timeout parameters have been changed, which might be putting much more of a load on the server and making the user experience feel slower.
Posting is always slower on nFT than it was with old FT.
I've heard rumors that due to the number of mods FT has made to UBB 5 that it would be nearly impossible to upgrade to 6. I don't know if this is true or not.
BTW I also was surprised when I took a look at the innards of UBB. Excellent piece of software from an end user point of view; very powerful tool but not very well thought out as a large scale piece of software. (I have worked with large scale distributed applications for a number of years.)