kaka

it does read choice. (thats why i searched HKGSIN over 5 days the very first time, so i could see what the limits of what Flightplan can do heehe!

jd20

Use a command line tool to query the database, like "sqlite3" (which you seemed to figure out).

Yeah, maybe a parser issue, I still need to take a look.

Yep, CX choice is supported. If you're not seeing it, but it shows on AM website, then it's a parser bug.

percysmith

Hi jd20

Sorry for further query.

A search is running slow on my older computer, so once my faster computer's finished one set of routes on the faster compauter I had that computer run the search for part of the remainder dates of the slower computer.

I then moved the html.gz and png files from the faster to the slower and tried to consolidate them using flightplan parse

This resulted, and the results procured from the faster computer cannot be displayed on the slower computer
Is there anything I can do to consolidate results? Thanks in advance!

jd20

You have to use "flightplan import" to add search requests from another computer. You can run "flightplan parse" before or after importing, though running after importing tends to be safer (you can even run both before and after, won't hurt anything).

Do not just manually copy the *.html.gz files, this won't work (because there's additional data in the database that is needed). If you already manually copied them, no biggie. Copy the flightplan directory (the one that has "data" and "db" in it) from your slow computer to somewhere on your fast computer. Let's say you have: "c:\flightplan" and "c:\slow_computer". You'd run:

cd c:\flightplan
flightplan import -d "c:\slow_computer"
flightplan parse

percysmith

Thanks!

(Damn I don't have slow_computer's db subfolder with me...)

jd20

You could try to recreate the needed database rows from the HTML files, but not sure how good your command line fu is It'd probably be faster just to re-run again (and this time remember to copy over both "data" and "db" directories from the other computer.

abielp

I would like to help in tinkering the codes but im not so sure where to start. :/

percysmith

Thanks for reply. Yes I am getting Choice hits for YVR.

It's just only available for YVR https://www.flyertalk.com/forum/cath...l#post29977452

And not SYD https://www.flyertalk.com/forum/cath...l#post29977447 nor LHR https://www.flyertalk.com/forum/cath...l#post29977453 in J for a whole year...

And damn, I was pining my hopes on porking out on Choice awards, because I can drive my miles purchase costs down but need availability on peak or shoulder.

jd20

Easiest way is to start by modifying an existing engine or parser (to fix a bug for example). You can use Chrome devtools to step through either, line by line, to see how it works (by using --inspect-brk on command line). Creating a new engine / parser is usually just copying an existing one, and changing the code as necessary. If you want to submit a PR on Github, I'll happily take a look

Thanks, will try and look at this tomorrow, sounds like their a couple CX parser bugs that I didn't see on the routes I was checking.

percysmith

Not a bug. I mean, CX really released no Choice availability for a year on those routes. Only Standard and Tailored, very extreme.

I have not found exceptions when I was checking by hand either.

kaka

How do i.... delete old data? (say i would like to re-run my KIX searches cuz 2 months have passed. if i do nothing it will see the old files and skip the dates (or all dates, hehe)

jd20

Run the cleanup command with a --maxage value, that tells flightplan how recent data should be to keep. Here's a few examples:

flightplan cleanup --maxage P1M # Deletes everything older than 1 month
flightplan cleanup --maxage PT30M # Deletes everything older than 30 minutes
flightplan cleanup --maxage PT0S # Deletes everything

Those funky looking time strings are a special format called ISO 8601 durations, you can read more about them here: https://en.wikipedia.org/wiki/ISO_8601#Durations

Note: Current behavior is cleanup command only removes the HTML files and corresponding entries in database. It doesn't touch the awards table, which is what the web UI is showing. When you run "flightplan parse" it'll blow away all those awards generated from old search requests, and re-compute them based on your new searches. So, a typical workflow would be like:

flightplan cleanup --maxage P1D # Get rid of old data
flightplan search <...> # Refresh with new search data
flightplan parse # Rebuild the awards table, from the new search data

And then what you see in the web UI would be the latest.

kaka

so when you run parse does it (only) rebuild with what you have in the DB entries?

so if you want the HKG<>FRA flights that u built over the last 4 days, HKG<>MXP last week and delete the old junk from last month,
you cleanup the junk before you start MXP search, but only run parse after building MXP and FRA (or running in between is ok too?)
then at the end of it you run server and client to tabulate them all?

sorry - key questions bolded :P

PS: Does the time required for parse depend on how much stuff there is? it seems like its taking longer and longer... does having more RAM help? (I'm now 2x2 on a 10 year old machine, but i could upgrade to 4x4GB (its wont be expensive now anyways))

jonlai9

So are you just scraping the reward websites? Can you configure how and what sites are scraped?

For example, what if I only want to see CX availability for Avios? Thanks!

jd20

When you run parse, it rebuilds from ALL searches stored in the database (which map to all the HTML files stored on disk). Later, I might change the behavior, so it will run incrementally, because it's starting to get kind of slow for large databases You can limit it to parsing only results for a certain airline (i.e. CX or SQ) to make it a bit faster.

Yep, sounds right. You can run "parse" in between, or at the very end, it won't affect the end result.

Yes, exactly The time it takes is proportional to how many searches (i.e. HTML files) are in the database. I've looked at the performance of parse, the main bottleneck is the time it takes to parse the HTML files (I use a library called CheerioJS, which is already pretty well optimized). Some airlines are much slower than others: CX has multiple large complex HTML files (one for each "tab" or award tier) and so it's one of the slowest. The CheerioJS parser doesn't use much RAM, so unless you had < 2GB total RAM I wouldn't bother upgrading just for Flightplan. It's really determined by your CPU speed. I do my testing on a very recent 2.9 Ghz Core i7, so if you have a 10 year old machine, I can only imagine it's super slow There's really two things I can do to make it faster:

1) Add some form of caching: if a search request hasn't changed since last time we ran parse, no need to parse it again.
2) Parallelize parsing: Most CPU's have lots of cores, so we can parse multiple HTML files in parallel.

I'll get to both of these eventually (especially #1 ), it's on the TODO list.

This is coming soon! Currently, I ignore partner awards, working on adding support for it though. I only use credit card points, so my miles aren't really tied to specific airline, so for example, I'll check award availability with CX first (which is already kinda limited inventory), and then if I find dates I like, I'll manually check each OW partner to see which I can redeem it through (so I'll manually check BA, JAL, etc...). A little more work I realize, especially if you already know you have to use Avios, so I'm working to expand this area.