jd20

TLDR: Flightplan is a set of command-line tools I built to scrape airline award inventory, and then display it in a fancy web UI. It's free, cross-platform, and open-source. You download and run it on your own machine. If you know how to program JavaScript, you can also use it as a library to write your own scraping scripts. Below is a condensed write-up of why I created it, and how it works. You can read the full version here: Flightplan: A new way to research flight awards

New Website: https://www.flightplantool.com

Github Repo: https://github.com/flightplan-tool/flightplan

Video Tutorial:
Flightplan: A new way to research flight awards

I was getting fed up with my manual process for planning award trips. I'd go to various airline websites, search the segments I was interested in, and click day-by-day, recording the results manually into Evernote. It looked something like this:



Manually searching for awards is slow and error-prone.

For our next trip, I knew the manual process wasn't going to cut it. So I decided to automate it, with the help of Headless Chrome, Puppeteer, Node, React, and D3.js. After a weekend of hacking, I had the first working prototype, and christened it: Flightplan. While it certainly took me longer to finish building Flightplan than I normally spent on an award booking, it did allow me to look at a lot of routes I would've otherwise ignored, because finding all the award availability would've been too time-consuming.

The numbers were sobering: with the help of Flightplan, I was able to analyze over 40K award fares (41,839 to be exact), across four different airlines: Cathay Pacific, All Nippon Airways, Korean Air, and Singapore Air. We redeemed over $90K-worth of flights for 6 passengers visiting 6 different cities in three different cabins (First, Business, and Premium Economy) for just under 1 million miles.

Choose your weapon...

Before building Flightplan, I spent a bunch of time trying other tools: KVS Tool, ExpertFlyer, Award Nexus, award.flights, and more. While these are all great in their own way, and they could speed up my manual process, none of them could replace it entirely. My ideal tool would:

• Run offline for hours, without any need for intervention
• Aggregate all award data together in a database
• Allow real-time visualization of award availability, to aid in trip planning
• Run on Windows, Mac, or Linux (I use a mixture of all three)
• Be nearly indistinguishable from a real human to the website operator
• Handle failed requests gracefully
• And finally be open source

This last point was important to me, because I didn't see myself writing the scrapers for every airline under the sun, and didn't want people to wait for a single person (me!) to have to fix the tool when it broke due to a website change.

The Puppet Master

Airlines often employ a cat-and-mouse game of trying to block automated scrapers. The latest tool to come along is Headless Chrome, which is just like real Chrome, but can be automated and run headless (with no visible window). Since Chrome is cross-platform, so is Headless Chrome. I use a JavaScript library to control Headless Chrome called Puppeteer, which is also built by Google. Using Node.js, I was able to write a script which would run the award searches, and then save the results to a database, alongside the raw HTML.



The beauty of a cross-platform browser like Chrome is it can run on Windows, Mac, or Linux!

Bringing the Data to Life

With all the award data in hand, I needed a way to visualize it. I was inspired by vgutkovsky's FT thread, on using a radar chart to layout the data. A normal calendar display would be too tall, so you'd have to break it in columns, making it hard to see certain patterns in the data. Researching radial calendar layouts, I came across the fantastic work of Bureau Oberhaeuser:



Beautiful radial calendars designed by Bureau Oberhaeuser.

D3.js seemed a natural fit for bringing this concept to life, and someone had even already posted a working implementation to Github. I created a ReactJS web app, for searching the award data and rendering the calendar with D3. The final result looked like this:



The final React and D3 web app

You can checkout the final project on Github, there are more details on how to install and run it in the Readme. Hope this helps you plan your next big trip, it’s definitely been an educational month for me. What would you like to see added to Flightplan next? Let me know!

rickywk

How can I run it in Windows environment?
Seems the readme is not for Windows environment

jd20

It should work on Windows, though I haven't tried it yet myself. For installing Node, click the link that says "Windows", and for installing Yarn, after you click the link there's a dropdown to select your platform.

After that, the commands in the readme should be pretty much same on any platform. If you're getting hung up on a particular part, let me know which step (and any errors you're seeing).

HMPS

Interested...if easy to use .

rickywk

Here you go

I also don't know how to launch the web UI or perform simple search

abielp

so this library is only for direct flights? non-direct or connecting flights are not included?

jd20

Hi Ricky, what do you get if you run "python --version", and what version of Windows are you running? Also, can you try running (in the same directory) just "yarn add syncprompt" and let me know the output? It sounds like what's happening is building that native extension on Windows is failing, I can file a bug against the creator of that module if it is indeed the case.

Also, once everything is installed, from that directory, the following commands are useful:

"flightplan" - shows help for all commands
"flightplan server" and "flightplan client" - you run these two commands, and then open a browser pointed to "http://localhost:3000/" That will let you see results.

jd20

Currently, the parser only returns direct non-partner awards. I'd love to get more feedback on how useful the non-direct and partner flights are. I know for myself, I always plan out my trips by searching segment by segment, and I don't fully trust partner awards (because the airlines don't always release all awards to their partners). So, when I search awards, I always go to the source (i.e. if searching SQ awards, I go to the KrisFlyer website).

So, it wouldn't be hard to modify the parser to return the additional data, I'd just like to get a better sense of how people would use it, before tackling it.

daft009

Partner data would be great but not sure how well the SQ partner data is loved.

United Airlines partner data is the one most people seem to use for star alliance partner data

rickywk

Seems I missing python, which python build you would suggest for windows?

jd20

I personally haven't redeemed through UA, but I agree it's probably one of the more popular options here on FT. I'll maybe look into adding it next. What UA route(s) are most popular for redeeming awards?

jd20

I think your issue is with node-gyp (which is what builds the syncprompt dependency that was failing). This sounds like the commonly accepted solution to getting node-gyp working (installing Python manually may not totally fix your problem):

After that, try running "yarn global add flightplan-tool", and see if you get success or not. If it works, I'll add an extra step to the Readme that you need to install windows-build-tools for Windows. Thanks!

rickywk

Still no luck, here is the error screen

daft009

Its a good tool for finding star alliance saver tickets to be used on other Star alliance airlines, but only if its a saver fare.
EG: I could use my krisflyer miles to book United or Air Canada or AirNZ or any other *A airlines if the united website showed availability in saver business for the chosen route.

jd20

Now try running "python --version" and "where python" and let me know what you get. I'll also try installing on my Windows laptop, so I can follow along and help debug it better. What version of Windows are you running, btw?