United & AA technical problems...for all concerned
 

Good evening everyone.

I thought I'd let you all know that I've been in talks with InfoPop for a fair amount of the day and we are discussing our options for fixing the problems on FlyerTalk and improving the performance.

There are definitely a number of issues to deal with and I'm uncertain at this point in time about how we are going to proceed.

After my talk with InfoPop we determined that, there are a number of hardware issues that need to be addressed...this has become a necessity because UBB was never intended for the kind of scalability that we are needing at the moment. There are a number of "temporary" things that we can do to solve some of the problems but the long term solution is going to need to be more drastic. This is due in part to the hardware and in part due to the ineffeciency of the language that UBB is written in a ( a language that is fantastic for some things but not others )...as well as the data structure of the BB system on the whole.

The United forum will be down for one more day ( hopefully no longer ) and the American forum is also under duress. I intend to shut it down this evening to see if I can implement a "temporary" fix...please forgive me.

As usual, thank you all very much for your time and patience and for your continual feedback.


------------------
Michael at WebFlyer
mailto:[email protected][email protected]</A>
2428 Hall Ave.
Grand Junction, Co. 81501
970.263.4201

Thanks for the update !

Thank You Michael for the explanation. I imagine that this has been a frustrating time for you (since the "enhancement") and I for one appreciate your taking the time to let us know what is going on.
Good Luck!

-Bob Mc

Michael,

Thank you for posting an update.

As a programmer, I understand that software will have problems and I wish you and all of the FT staff the best during this.

I would like to make a suggestion though. Please don’t forget your FT users http://www.flyertalk.com/forum/smile.gif You have many people here from all types of occupations that would probably be glad to give freely of their time to FT when in need (speaking for myself I would gladly do this).

Just a thought, thanks again for the great work!!!

George...

Wow, you guys are great. I'll keep these offers and suggestions in mind and pass them on to Randy and my collegues...thank you.

I have updated AA and it appears to be running fine at the moment. How long this will last is anybody's guess but I'll keep at it until we have an adequate solution to all of our problems.

Thank you all again for your support.

------------------
Michael at WebFlyer
mailto:[email protected][email protected]</A>
2428 Hall Ave.
Grand Junction, Co. 81501
970.263.4201

Are you running FT on a Sun box?

I work for Sun Microsystems. Feel free to email me for assistance of any kind.

-David


[This message has been edited by LIH Prem (edited 08-28-2001).]

ClueByFour: Very interesting post; I wonder if a number of us working together might be able to figure out something here.

To add a couple of points to what you wrote:

Is this really a scalability problem? Is it slower because of more activity or because of a problem with file corruption? Smells to me more like a software bug than somthing adding more RAM would help with, although in general of course that is a good suggestion.

I've observed that nFT seems to be doing a lot more in terms of page updates than old FT. For example, when I bring up the front page of FlyerTalk miles, it always refreshes, and thus I lose the light bulbs right from the start of a session. In the old days, it seemed to do this much less often. That "feels" true for other pages as well. Refreshes always seem to take a lot longer on the new FT. I wonder if some timeout parameters have been changed, which might be putting much more of a load on the server and making the user experience feel slower.

Posting is always slower on nFT than it was with old FT.

I've heard rumors that due to the number of mods FT has made to UBB 5 that it would be nearly impossible to upgrade to 6. I don't know if this is true or not.

BTW I also was surprised when I took a look at the innards of UBB. Excellent piece of software from an end user point of view; very powerful tool but not very well thought out as a large scale piece of software. (I have worked with large scale distributed applications for a number of years.)

Michael: Thanks for keeping us posted...much better than being in the dark about what's going on, and it's good to know you're working on the problems.

Good morning everybody.

Some really wonderful feedback here.

ClueByFour, you are dead on the money. Very astute observations considering you have never worked on our particular system. Your suggestions are also things that we are considering.

Unfortunately, in addition to the hardware issues, there are a number of other problems as well...as dgolds points out above. There is an issue of scalability, at least that is what the folks at UBB told me yesterday and I'd have to say that there are a fair share of bugs...at least I see them as bugs...I consider any contingency that can be accounted for in a program ( but that is not accounted for ) as a bug...that's pretty big territory. http://www.flyertalk.com/forum/smile.gif

Any further suggestions?

Thank you all for the truely excellent feedback. Maybe together we really can come up with a good working solution.

------------------
Michael at WebFlyer
mailto:[email protected][email protected]</A>
2428 Hall Ave.
Grand Junction, Co. 81501
970.263.4201

...nice to have a 50 yard line seat at the software Super Bowl. http://www.flyertalk.com/forum/biggrin.gif

ClueByFour, very nice work. Thank you for the workup. I'm very, very impressed. Give me a bit of time to look this over thoroughly and when I have a minute I'll email you with my comments and we'll see where it takes us...sound good?

Thank you very much for this feedback, you've been very, very helpful.

------------------
Michael at WebFlyer
mailto:[email protected][email protected]</A>
2428 Hall Ave.
Grand Junction, Co. 81501
970.263.4201

One more observation:

The three fora that are down regularly (MilesBuzz, American, United) all have &gt; 4K threads and &gt; 32K posts. Those can often be magic numbers that cause bugs to reveal themselves. Quite possible UBB was never QA'ed for that amount of posts. If you haven't followed up with InfoPop on this being a possibile source of our misery, I strongly suggest you do so.

If either a 4K or 32K limit is causing a problem, you might start to see problems in Coupon Connection (which just reached over 4K threads) or Delta (which is about to go over 32K posts.

Michael, I wonder if you might be willing to post your configuration here (or e-mail it to myself and others who are interested in helping out).

ClueByFour: I love you, man...

Personally, and with a tip 'o the hat to my technoguru observer friend RichG, I think it's a miscompiled terraflop resource terminator that is the main problem.

That, or take the 110 volt plug and shove it into a 220 volt outlet and see if that gooses things a bit.

But then again, I could be mistaken. http://www.flyertalk.com/forum/wink.gif

Those are all excellent suggestions.

While, at this point, I do think that blaming the hardware is a bit of a red herring (the software still has to work, albeit it might work very slowly) on a slow system, you should not have data corruption. There are other reasons to blame the hardware, but I doubt if it's the cause of the data corruption as long as you have working disks and enough swap space configured .. of course, on the other hand, you really don't want to use swap that much, so you do need to have enough memory.

I'd be interested in knowing what the configuration is (prtconf -vp output) and what sorts of numbers you're seeing with 'vmstat 5' and 'iostat 5'.

I can ask around about the NCA thing if it becomes interesting.

-David

Okay, this might be a bit crazy...but I have an idea (don't jump all over me!!).

What about making:

United II
American Airlines II
etc.

I know...there would be a lot to figure out here...which I clearly haven't thought through...just figured I'd plant a seed.

Dorian

Thanks - love this thread and convo (while UA is down... hmmm could make a few comments on this http://www.flyertalk.com/forum/wink.gif ). I'm a Sun person myself, but I don't work for them (LIH Prem you rock http://www.flyertalk.com/forum/smile.gif ), so I'm very interested too in the specs of what si going on.

The 4k/32k one is definately magic type numbers to watch, even more so if it's an older release of Solaris (eg: &lt; 7).

Should we open up a "Geek Forum" for this? http://www.flyertalk.com/forum/biggrin.gif

Do you have any idea when the UA board may be back up. Also - how about posting a notice on the front page about the UA board so that new users don't think that flyertalk doesn't have such a board?

I noticed that pprune.org closes their threads at 100 posts for 'board management' issues.

Dorian

While I understand that the problem may not be the size of the forums (after all the largets forums are also the ones getting the most traffic on any given day), but is there a particular reason why the largest boards do not archive posts to an archive forum after so many posts or months?

There's very little functionality lost by searching the last 6 months separately from rest of time. I guarantee you that the commonly queried topics come up again and again now anyway, and an archive won't make people miss and repost questions anymore than they do now.

Like I said, just curious.

OOH I love when you guys talk data corruption...

As well...Infopop would certainly love to have nFT running on their 'bigger' product OpenTopic. They would suck another $80k +++ out of Randy that way...

Dorian

I agree with you, Clue, about the flat files. It must be a nightmare to fix them when they do get corrupt.

I wish I knew what they were running this thing on. Can you figure it out?

-David

Michael at FT has soaked up the suggestions, in a well intentioned way, but I think those who have insight could help even more if he would disclose the hardware and OS they are running. Other posters have made assumptions, that I would have hoped that Michael would have taken as nudges to disclose what they're running, but all the helpers are doing now is flying blind. Michael, how about prying loose a little info about your config...?

Perhaps they don't want to reveal that Sabre is running everything under contract, and that's why AA's board works again and UA's has gone down the toilet!!!

http://www.flyertalk.com/forum/biggrin.gif http://www.flyertalk.com/forum/biggrin.gif http://www.flyertalk.com/forum/biggrin.gif http://www.flyertalk.com/forum/biggrin.gif http://www.flyertalk.com/forum/biggrin.gif http://www.flyertalk.com/forum/biggrin.gif http://www.flyertalk.com/forum/biggrin.gif

Or maybe it is just time to upgrade that old Vic-20 to a C-64 http://www.flyertalk.com/forum/wink.gif


------------------
RichardMEL, UA 1K
A Star Alliance Member.

My suggstion is for Randy to take some miles from an account and redeem a First Class round trip from Pittsburgh to Colorado Springs, and lay on a darn fine lunch and dinner for Clue By Four and Michael , and I suspect the problem might be readily solved. :-)

I'm with RichardMel,

I think either Sabre is running the show,... or UAL gave randy and crew all the 1K swu's left over for the entire year,.... in order for him to keep the board down and all of us without knowning if the reconfiguration of the fleet is done and other burning issues...

UAL IS DOWN, but help is under way... I am happy to know so many of us are trying to get this issue solved.

RedRobin

geordie6 has made an excellent suggestion http://www.flyertalk.com/forum/smile.gif

Any official word on the status of the United forum? Is there an ETA for a fix or creation of a replacement forum?

Thanks,


------------------
Sagy

RE: 4K and 32K limits and other magic numbers.
Don't assume that a "power of 2" limit is always due to a bug. I used to manage a product that displayed info in tables, and I specifically limited the max size to 1024 rows. I could have picked any number, but using a power of 2 meant that everyone assumed it was a technical limit and didn't bother asking me to increase it :-)

Bob

Michael...United seems to be up and humming along nicely now. Whatever you did...Thanks (and hope it "sticks")!

I notice however that the total FlyerTalk Miles post count is still indicating a low 449,250 posts and the United forum only has just over 8,000 posts. I assume you archived some stuff to get things going again?

Any idea when you'll be able to bring 'em back?

[This message has been edited by PremEx (edited 08-30-2001).]

Good job to Michael and all who helped out!!
Thanks http://www.flyertalk.com/forum/smile.gif

I play with internet community software for a living, although we don't use InfoPop...

Bottom line here is that there is likely a scaleability issue on the message board, and it doesn't handle too large forums, such as United etc... hardware tweaks might get you a few inches here and there, but they won't get you over the finish line if the software doesn't scale.

So I don't wish to nor will I make this an ad for something we sell.. but there are highly scaleably message board solutions available and people who manage them. I know FT doesn't want to spend a fortune here either.

One of the most interesting non commercial solutions right now is a java based board called Jive.. it's open source, almost entirely free and very very scaleable. Sun are working on switching most of their boards to Jive based technology, as are several others.

There's always the more expensive commercial solutions, if you really want to spend $$ I can give you a list, but looking at this thread there's enough expertise out there to get things moving.

The second and more immeidately addressable area is how the community is structured. The issue of course is scaling... 2 general United boards isn't going to work, where would you look for information? What will work is breaking United and AA into smaller chunks that are topic specific... United Policies and Issues, United Flights and Aircraft, United MP Information and so on... I'm sure with a couple of hours of analysis every topic in United can be broken into a list of three or four general topics where people could clearly see where to post... suddenly you have a bunch of smaller forums that aren't overloading the limits of InfoPop, yet a structure the users can easily follow.

There are systems that can deal with this amount of traffic, eBay uses an external vendor with over 60,000 new posts a day (on their message boards, that's not the bidding system), CNN, HBO... I could go on, but there are plenty of sites handling bigger loads... at some point someone needs to "byte" the bullet (sorry couldn't resist) and decide to go forward with a scaleable replacement.

The choice of spending big $$ for a commercial grade solution and vendor or spending IT resources implementing something like Jive is one only Randy can dictate... but step one is to admit (at least internally if not to the users) that the current system is at the end of it's scaleability capability. The time to do that is now so you can put your resources into a solution that works, rather than spinning cycles trying to hold together something that's bursting at the seams.

Someone said buy InfoPops bigger setup for $80K... for the record $80K is over the top for a scaleable solution for the volume I suspect this board has (cheap for eBay, but WAY expensive for FT).

Anyway... the short term fix probably lies in topical sub forums for UAL and AA, longer term something that scales. So there's my 2 cents worth... at least here on FT it's 2 cents.

Let me know if you want any specific information, I've tried to stay general here and avoid advertising our products or anyone elses, because that's not what this board is about.

Good morning everybody...well, almost noon now. http://www.flyertalk.com/forum/smile.gif

I thought I'd drop you all a quick line or three and let you know how much I have appreciated all of the help I've received from you. There have been a large number of very, very good suggtestions here...and I am looking into all of them.

I apologize that it has taken me so long to get back with you...this has been a very big week. In addition to the FlyerTalk bulletin boards I have a number of other fairly substantial responsibilities...not the least of which is assisting our in house editors with making portions or our magazine "InsideFlyer" available electronically.

I won't go into all of the details at this point in time, but as you can see we have the United forum up and running again...man, has this ever been a fiasco. Randy has been an extremely busy man and it's sometimes difficult to move forward on "controversial" solutions without his approval and this is compounded when you are unable to get ahold of him long enough to explain the situation and it's various remedies. I'll post more about these solutions as time goes by.

Suffice it to say that a couple of the gents from InfoPop had to remove and backup a sumstantial portion of the United forum in order to get it up and running again. It seems to be working fine at the moment which seems to bear out our observations that the hardware and software are not up to the task of handling such a large load. We will probably be doing the same for the other 3 or 4 largest forums on our boards. Randy is adament about having that data available to our community and although I agree it's been very, very trying accomadating such a demand within the limits imposed by the conditions.

I apologize if I have offended anyone by not giving out the hardware specs but I haven't had the chance to clear it with my systems admin...I don't see a problem with releasing such info nor do I believe that he will, but since I'm not all that systems savvy I thought it best to wait until I can discuss it with him. You all of made some pretty close gestimations and I think that, that is something that you should all applaud eachother for...pretty amazing really.

As for alternative BB systems...I have been looking into them for sometime now. I've even setup and tested many on our local server. The list includes "WWWThreads", "vBulletin", "WebCrossing", and a number of other less reputable boards. "Jive" came to my attention through a personal email from an individual who I found very warm and knowledgeable. I've taken a cursory look at the software and have liked what I have seen so far but it'll need to be scoped out a little better. I will be doing a thourough investigation of it's relative strengths and weaknesses here very, very shortly.

I think that about covers it for the time being. Any questions? Comments?

Thank you all again for your time and patience.

------------------
Michael at WebFlyer
mailto:[email protected][email protected]</A>
2428 Hall Ave.
Grand Junction, Co. 81501
970.263.4201

Sounds like a plan. WebCrossing will scale better, but with over 50 WebX installations in place and working we're likely to move away from them, also a flat file system and subject to some strange things happening from time to time, but definately a nice step up.

Jive certainly seems to be the flavour of the moment because it is Open Source and thus very customizeable. The side issue is that it takes a little more engineering resources to implement it... or you can pay the folks who wrote Jive a few bucks and they'll make it work, they're good guys and relatively inepxensive to hire.

Whatever the alternative be aware that moving the data cna be a challenge in some systems, look further ahead and consider both moving the data into a new system, and later out of it... some systems don't do exports gracefully if at all. Jive is nice, it uses Oracle for it's DB... neither flat file nor a problem moving data in or out.

Jive is certainly a very good recommendation.

I have seen single threads in excess of 75 pages that have no speed issues at all.

Dorian

Thank you. I hope people keep the feedback coming in this particular thread/forum...I find it helpful and reassuring.

You guys hit one of our major problems right on the head...the legacy data. We have consistantly found problems getting the old data into either a database or into another flat file data format.

WWWThreads did a very nice job but we found that it appeared to be a bit too slow...as slow, or slower than UBB...at least that's what I was told by our testers. Looking back on it I'm not sure that it was really any slower than UBB has been for the last many weeks. http://www.flyertalk.com/forum/smile.gif

WWWThreads also had a lot of very nice additional functionality. Although I'd settle for a stable, fast and scalable solution at the moment...we could worry about additional goodies later. http://www.flyertalk.com/forum/smile.gif

Thanks again.

------------------
Michael at WebFlyer
mailto:[email protected][email protected]</A>
2428 Hall Ave.
Grand Junction, Co. 81501
970.263.4201

I was chatting about Jive with a few our hacker/technician types...they said that it is the only BB software out there for &lt;$50k that is highly scalable without onerous hardward requirements.

Dorian

Michael: In the spirit of the open source movement, if you are thinking of going with something like Jive, why not see if there are people here who might be willing to help out for the good of the community? I bet a few among us would go for it...

"Will code for miles or points"

There is an interesting thread on massively large deployments over at Jive.

They state that serving &gt;1,000,000 pages a day is no big deal.

Dorian

Have any of you looked at phpBB?