Go Back  FlyerTalk Forums > Travel&Dining > Travel Tools
Reload this Page >

Flightplan: How to search a year of award inventory

Flightplan: How to search a year of award inventory

    Hide Wikipost
Old Jan 16, 19, 8:52 am   -   Wikipost
Please read: This is a community-maintained wiki post containing the most important information from this thread. You may edit the Wiki once you have been on FT for 90 days and have made 90 posts.
 
Last edit by: IBobi
Wiki Link
Print Wikipost

Old Jul 6, 18, 11:13 am
  #1  
Original Poster
 
Join Date: Jan 2010
Posts: 188
Cool Flightplan: How to search a year of award inventory

TLDR: Flightplan is a set of command-line tools I built to scrape airline award inventory, and then display it in a fancy web UI. It's free, cross-platform, and open-source. You download and run it on your own machine. If you know how to program JavaScript, you can also use it as a library to write your own scraping scripts. Below is a condensed write-up of why I created it, and how it works. You can read the full version here: Flightplan: A new way to research flight awards

New Website: https://www.flightplantool.com

Github Repo: https://github.com/flightplan-tool/flightplan

Video Tutorial:

Flightplan: A new way to research flight awards

I was getting fed up with my manual process for planning award trips. I'd go to various airline websites, search the segments I was interested in, and click day-by-day, recording the results manually into Evernote. It looked something like this:



Manually searching for awards is slow and error-prone.

For our next trip, I knew the manual process wasn't going to cut it. So I decided to automate it, with the help of Headless Chrome, Puppeteer, Node, React, and D3.js. After a weekend of hacking, I had the first working prototype, and christened it: Flightplan. While it certainly took me longer to finish building Flightplan than I normally spent on an award booking, it did allow me to look at a lot of routes I would've otherwise ignored, because finding all the award availability would've been too time-consuming.

The numbers were sobering: with the help of Flightplan, I was able to analyze over 40K award fares (41,839 to be exact), across four different airlines: Cathay Pacific, All Nippon Airways, Korean Air, and Singapore Air. We redeemed over $90K-worth of flights for 6 passengers visiting 6 different cities in three different cabins (First, Business, and Premium Economy) for just under 1 million miles.

Choose your weapon...

Before building Flightplan, I spent a bunch of time trying other tools: KVS Tool, ExpertFlyer, Award Nexus, award.flights, and more. While these are all great in their own way, and they could speed up my manual process, none of them could replace it entirely. My ideal tool would:
  • Run offline for hours, without any need for intervention
  • Aggregate all award data together in a database
  • Allow real-time visualization of award availability, to aid in trip planning
  • Run on Windows, Mac, or Linux (I use a mixture of all three)
  • Be nearly indistinguishable from a real human to the website operator
  • Handle failed requests gracefully
  • And finally be open source
This last point was important to me, because I didn't see myself writing the scrapers for every airline under the sun, and didn't want people to wait for a single person (me!) to have to fix the tool when it broke due to a website change.

The Puppet Master

Airlines often employ a cat-and-mouse game of trying to block automated scrapers. The latest tool to come along is Headless Chrome, which is just like real Chrome, but can be automated and run headless (with no visible window). Since Chrome is cross-platform, so is Headless Chrome. I use a JavaScript library to control Headless Chrome called Puppeteer, which is also built by Google. Using Node.js, I was able to write a script which would run the award searches, and then save the results to a database, alongside the raw HTML.



The beauty of a cross-platform browser like Chrome is it can run on Windows, Mac, or Linux!

Bringing the Data to Life

With all the award data in hand, I needed a way to visualize it. I was inspired by vgutkovsky's FT thread, on using a radar chart to layout the data. A normal calendar display would be too tall, so you'd have to break it in columns, making it hard to see certain patterns in the data. Researching radial calendar layouts, I came across the fantastic work of Bureau Oberhaeuser:



Beautiful radial calendars designed by Bureau Oberhaeuser.

D3.js seemed a natural fit for bringing this concept to life, and someone had even already posted a working implementation to Github. I created a ReactJS web app, for searching the award data and rendering the calendar with D3. The final result looked like this:



The final React and D3 web app

You can checkout the final project on Github, there are more details on how to install and run it in the Readme. Hope this helps you plan your next big trip, it’s definitely been an educational month for me. What would you like to see added to Flightplan next? Let me know!
kaka, percysmith, Edvard and 13 others like this.

Last edited by jd20; Nov 19, 18 at 2:40 am Reason: Added demo website link
jd20 is offline  
Old Jul 8, 18, 6:36 pm
  #2  
 
Join Date: Oct 2013
Posts: 286
How can I run it in Windows environment?
Seems the readme is not for Windows environment
rickywk is offline  
Old Jul 8, 18, 8:07 pm
  #3  
Original Poster
 
Join Date: Jan 2010
Posts: 188
It should work on Windows, though I haven't tried it yet myself. For installing Node, click the link that says "Windows", and for installing Yarn, after you click the link there's a dropdown to select your platform.

After that, the commands in the readme should be pretty much same on any platform. If you're getting hung up on a particular part, let me know which step (and any errors you're seeing).
jd20 is offline  
Old Jul 8, 18, 8:18 pm
  #4  
 
Join Date: Feb 2010
Location: ORD
Programs: US Air, UA BA LH AI DELTA MARRIOTT CHOICE SGP
Posts: 9,788
Originally Posted by finiteyoda View Post
It should work on Windows, though I haven't tried it yet myself. For installing Node, click the link that says "Windows", and for installing Yarn, after you click the link there's a dropdown to select your platform.

After that, the commands in the readme should be pretty much same on any platform. If you're getting hung up on a particular part, let me know which step (and any errors you're seeing).
Interested...if easy to use .
HMPS is offline  
Old Jul 8, 18, 9:10 pm
  #5  
 
Join Date: Oct 2013
Posts: 286
Originally Posted by finiteyoda View Post
It should work on Windows, though I haven't tried it yet myself. For installing Node, click the link that says "Windows", and for installing Yarn, after you click the link there's a dropdown to select your platform.

After that, the commands in the readme should be pretty much same on any platform. If you're getting hung up on a particular part, let me know which step (and any errors you're seeing).
Here you go

I also don't know how to launch the web UI or perform simple search
rickywk is offline  
Old Jul 8, 18, 9:36 pm
  #6  
 
Join Date: Jan 2018
Posts: 46
so this library is only for direct flights? non-direct or connecting flights are not included?
abielp is offline  
Old Jul 8, 18, 10:16 pm
  #7  
Original Poster
 
Join Date: Jan 2010
Posts: 188
Originally Posted by rickywk View Post
Here you go

I also don't know how to launch the web UI or perform simple search
Hi Ricky, what do you get if you run "python --version", and what version of Windows are you running? Also, can you try running (in the same directory) just "yarn add syncprompt" and let me know the output? It sounds like what's happening is building that native extension on Windows is failing, I can file a bug against the creator of that module if it is indeed the case.

Also, once everything is installed, from that directory, the following commands are useful:

"flightplan" - shows help for all commands
"flightplan server" and "flightplan client" - you run these two commands, and then open a browser pointed to "http://localhost:3000/" That will let you see results.
jd20 is offline  
Old Jul 8, 18, 10:25 pm
  #8  
Original Poster
 
Join Date: Jan 2010
Posts: 188
Originally Posted by abielp View Post
so this library is only for direct flights? non-direct or connecting flights are not included?
Currently, the parser only returns direct non-partner awards. I'd love to get more feedback on how useful the non-direct and partner flights are. I know for myself, I always plan out my trips by searching segment by segment, and I don't fully trust partner awards (because the airlines don't always release all awards to their partners). So, when I search awards, I always go to the source (i.e. if searching SQ awards, I go to the KrisFlyer website).

So, it wouldn't be hard to modify the parser to return the additional data, I'd just like to get a better sense of how people would use it, before tackling it.
jd20 is offline  
Old Jul 8, 18, 10:44 pm
  #9  
 
Join Date: Jan 2017
Location: Australia
Posts: 23
Partner data would be great but not sure how well the SQ partner data is loved.

United Airlines partner data is the one most people seem to use for star alliance partner data
daft009 is offline  
Old Jul 8, 18, 11:12 pm
  #10  
 
Join Date: Oct 2013
Posts: 286
Originally Posted by finiteyoda View Post
Hi Ricky, what do you get if you run "python --version", and what version of Windows are you running? Also, can you try running (in the same directory) just "yarn add syncprompt" and let me know the output? It sounds like what's happening is building that native extension on Windows is failing, I can file a bug against the creator of that module if it is indeed the case.

Also, once everything is installed, from that directory, the following commands are useful:

"flightplan" - shows help for all commands
"flightplan server" and "flightplan client" - you run these two commands, and then open a browser pointed to "http://localhost:3000/" That will let you see results.
Seems I missing python, which python build you would suggest for windows?
rickywk is offline  
Old Jul 8, 18, 11:31 pm
  #11  
Original Poster
 
Join Date: Jan 2010
Posts: 188
Originally Posted by daft009 View Post
Partner data would be great but not sure how well the SQ partner data is loved.

United Airlines partner data is the one most people seem to use for star alliance partner data
I personally haven't redeemed through UA, but I agree it's probably one of the more popular options here on FT. I'll maybe look into adding it next. What UA route(s) are most popular for redeeming awards?
jd20 is offline  
Old Jul 8, 18, 11:41 pm
  #12  
Original Poster
 
Join Date: Jan 2010
Posts: 188
Originally Posted by rickywk View Post
Seems I missing python, which python build you would suggest for windows?
I think your issue is with node-gyp (which is what builds the syncprompt dependency that was failing). This sounds like the commonly accepted solution to getting node-gyp working (installing Python manually may not totally fix your problem):

Install all the required tools and configurations using Microsoft's windows-build-tools using
Code:
npm install --global --production windows-build-tools
from an elevated PowerShell or CMD.exe (run as Administrator).
After that, try running "yarn global add flightplan-tool", and see if you get success or not. If it works, I'll add an extra step to the Readme that you need to install windows-build-tools for Windows. Thanks!
jd20 is offline  
Old Jul 9, 18, 12:06 am
  #13  
 
Join Date: Oct 2013
Posts: 286
Originally Posted by finiteyoda View Post
I think your issue is with node-gyp (which is what builds the syncprompt dependency that was failing). This sounds like the commonly accepted solution to getting node-gyp working (installing Python manually may not totally fix your problem):



After that, try running "yarn global add flightplan-tool", and see if you get success or not. If it works, I'll add an extra step to the Readme that you need to install windows-build-tools for Windows. Thanks!
Still no luck, here is the error screen
rickywk is offline  
Old Jul 9, 18, 12:25 am
  #14  
 
Join Date: Jan 2017
Location: Australia
Posts: 23
Originally Posted by finiteyoda View Post
I personally haven't redeemed through UA, but I agree it's probably one of the more popular options here on FT. I'll maybe look into adding it next. What UA route(s) are most popular for redeeming awards?
Its a good tool for finding star alliance saver tickets to be used on other Star alliance airlines, but only if its a saver fare.
EG: I could use my krisflyer miles to book United or Air Canada or AirNZ or any other *A airlines if the united website showed availability in saver business for the chosen route.
daft009 is offline  
Old Jul 9, 18, 12:36 am
  #15  
Original Poster
 
Join Date: Jan 2010
Posts: 188
Originally Posted by rickywk View Post
Still no luck, here is the error screen
Now try running "python --version" and "where python" and let me know what you get. I'll also try installing on my Windows laptop, so I can follow along and help debug it better. What version of Windows are you running, btw?
jd20 is offline  

Thread Tools
Search this Thread
Search Engine: