Visualizing a year of Singapore Suites award availability
Dec 9, 2017, 8:42 pm
#1
Original Poster
Join Date: Dec 2017
Posts: 50
Visualizing a year of Singapore Suites award availability
I put together a little project, just for fun. The part about the visualization is always going to useful (I hope), but the data I used for the specific example is best while fresh. It's about 48 hour old at this point. Hope this helps some folks, look forward to hearing from you in the blog comments!
https://www.infinityinsight.com/blog/?p=328
https://www.infinityinsight.com/blog/?p=328
Dec 9, 2017, 9:49 pm
#2
Join Date: Feb 2017
Programs: MM, Krisflyer, QFF, VFF
Posts: 441
How hard (or easy) is it to pull this dataset? I'm curious if you run a query or need to manually generate the data through the webpage? If a query it is possible to run it daily to give us a fresh indication of availability? (Airlines will hate you)
Great graph btw.
Great graph btw.
Dec 9, 2017, 11:01 pm
#3
Join Date: Apr 2011
Location: Singapore / India
Programs: SQ QPP, Hilton Diamond, IHG Platinum, Taj Inner Circle Gold
Posts: 639
This is great. Thanks so much for the effort.
That looks really bad for FRA-JFK with no possibility to waitlist at saver levels.
I thought such route they should open as there must be some pax getting off in FRA.
That looks really bad for FRA-JFK with no possibility to waitlist at saver levels.
I thought such route they should open as there must be some pax getting off in FRA.
Dec 10, 2017, 8:59 am
#4
Original Poster
Join Date: Dec 2017
Posts: 50
Hmm well a couple things. First, this is JFK to FRA, not FRA to JFK. And there are plenty of dates with saver waitlist availability. Even a couple with non-waitlist.
Dec 10, 2017, 9:04 am
#5
Original Poster
Join Date: Dec 2017
Posts: 50
I could share the script that pulls down the 355 offline HTML files, but without a serious ETL tool to parse the files, I don't think this would be particularly useful to anyone. If I'm wrong about that, please lmk.
Dec 10, 2017, 10:54 am
#6
Join Date: Nov 2017
Posts: 2
Thanks! There are a few challenges with your suggestion. First, it takes about 2.5 hours to refresh per route, so even if we're only talking about a single route, that's a decent amount of processing time per refresh period. Second, this is built with an application that isn't available to be shared live for free, so I would also need to manually generate screenshots each refresh period, which can get annoying. Third, I don't particularly want the airlines to hate me :-) This was just meant as a cool data viz.
I could share the script that pulls down the 355 offline HTML files, but without a serious ETL tool to parse the files, I don't think this would be particularly useful to anyone. If I'm wrong about that, please lmk.
I could share the script that pulls down the 355 offline HTML files, but without a serious ETL tool to parse the files, I don't think this would be particularly useful to anyone. If I'm wrong about that, please lmk.
Dec 11, 2017, 12:44 am
#7
Join Date: Jul 2012
Programs: UA 1K, SPG Lifetime Gold
Posts: 32
@vgutkovsky, super cool visualization. and would be really helpful if it weren't for the overhead you mentioned
Dec 11, 2017, 4:24 am
#8
Join Date: Dec 2007
Location: Virginia City Highlands
Programs: Nothing anymore after 20 years
Posts: 6,900
Small data in action. 
First - thanks.
Secondly, I understand that you pulled 355 HTML files, but did you have to write a code to a) pull these html files via iMacros and b) process them with QlikView?
I also understand that it required 2.5h to build a picture for whole 330 days ahead. But would it take less time to find only the days where was saver availability?
First - thanks.
Secondly, I understand that you pulled 355 HTML files, but did you have to write a code to a) pull these html files via iMacros and b) process them with QlikView?
I also understand that it required 2.5h to build a picture for whole 330 days ahead. But would it take less time to find only the days where was saver availability?
Dec 11, 2017, 4:26 am
#9
Join Date: Dec 2007
Location: Virginia City Highlands
Programs: Nothing anymore after 20 years
Posts: 6,900
@vgutkovsky, super cool visualization. and would be really helpful if it weren't for the overhead you mentioned
Well, if one had direct access to DB with ability to put the right query...
Dec 11, 2017, 5:18 am
#10
Original Poster
Join Date: Dec 2017
Posts: 50
Exactly right. But that would be literally hacking. Not sure if it's technically possible, but definitely illegal.
Dec 11, 2017, 5:20 am
#11
Original Poster
Join Date: Dec 2017
Posts: 50
Small data in action.
First - thanks.
Secondly, I understand that you pulled 355 HTML files, but did you have to write a code to a) pull these html files via iMacros and b) process them with QlikView?
I also understand that it required 2.5h to build a picture for whole 330 days ahead. But would it take less time to find only the days where was saver availability?
First - thanks.
Secondly, I understand that you pulled 355 HTML files, but did you have to write a code to a) pull these html files via iMacros and b) process them with QlikView?
I also understand that it required 2.5h to build a picture for whole 330 days ahead. But would it take less time to find only the days where was saver availability?