Go Back  FlyerTalk Forums > Travel&Dining > Travel Tools
Reload this Page >

ITA software scrape

ITA software scrape

Reply

Old Dec 4, 18, 8:57 am
  #1  
Original Poster
 
Join Date: Sep 2004
Programs: LH HON, AF Plat, BA Gold, SPG Plat, HHo Gold
Posts: 139
ITA software scrape

Just curious, whether anybody has succeeded in creating a scraping script for ITA software. I have seen a post over on headforpoints, which suggested it was possible. Any insight?

Thanks,

Jan
Jan@BRU is offline  
Reply With Quote
Old Dec 5, 18, 12:28 am
  #2  
 
Join Date: Jun 2015
Location: SFO/SJC, BWI
Programs: :rolleyes:, DL DM, Mlife Plat, TR 7*, SPG/MR Plat, UA 1K
Posts: 8,979
Originally Posted by [email protected] View Post
Just curious, whether anybody has succeeded in creating a scraping script for ITA software. I have seen a post over on headforpoints, which suggested it was possible. Any insight?

Thanks,

Jan
What exactly do you mean by scraping?

Are you looking for functionality provided by either the Userscript or bookwithmatrix?

ITA-Matrix-PowerTools - Userscript for Orbitz/DL/UA/AA/BA/CZ/IB/LA/LH/LX/TK

BookWithMatrix.com: a tool to easily book with ITA Matrix
Zorak is offline  
Reply With Quote
Old Dec 7, 18, 9:22 am
  #3  
Original Poster
 
Join Date: Sep 2004
Programs: LH HON, AF Plat, BA Gold, SPG Plat, HHo Gold
Posts: 139
Zorak,
No -- that's what I mean. From what I understand, the tools you mention assist you in purchasing specific search results, which indeed is a big help.

Scraping refers to automating requests sent to a website and harvesting the results.
Take for example the recently introduced limitation on ITA to limit the jointly queried departure points to cities in a single country. If you want to do a regional query of flights from all departure points in neighboring countries, you need to several requests.
Scraping would run a script, send out a request, wait for the results to display and than store the results before moving on the next request automatically etc.
It's obvious that google discourages that kind of data harvesting and has put in place a few barriers to do it.... I was wondering whether somebody has managed to successfully scrape ITA nonetheless.

Thanks

Jan
Jan@BRU is offline  
Reply With Quote
Old Dec 7, 18, 11:10 am
  #4  
 
Join Date: Jun 2015
Location: SFO/SJC, BWI
Programs: :rolleyes:, DL DM, Mlife Plat, TR 7*, SPG/MR Plat, UA 1K
Posts: 8,979
I know what scraping means in general, but wanted to understand your specific use case and whether it had to do with booking or searching.

IMO if someone wanted to automate the searching process or aggregate the results of multiple searches, a more direct approach might be to use Developer Tools to try and figure out the underlying search API and then issue customized requests directly to that instead of going via the UI.
Zorak is offline  
Reply With Quote
Old Dec 7, 18, 1:40 pm
  #5  
 
Join Date: Feb 2015
Posts: 179
Same problem discussed in a slightly different way. Here in Western Europe several international airports may be within 150 miles of each other but in four different countries so while the old ITA-Matrix allowed a quick search that has now become more tedious. Any suggestions how a searcher might extend to search 'airports within x miles'
Just a thought

Last edited by gbs1112; Dec 7, 18 at 1:41 pm Reason: spelling
gbs1112 is offline  
Reply With Quote
Old Dec 12, 18, 5:48 am
  #6  
Original Poster
 
Join Date: Sep 2004
Programs: LH HON, AF Plat, BA Gold, SPG Plat, HHo Gold
Posts: 139
Originally Posted by Zorak View Post
I know what scraping means in general, but wanted to understand your specific use case and whether it had to do with booking or searching.

IMO if someone wanted to automate the searching process or aggregate the results of multiple searches, a more direct approach might be to use Developer Tools to try and figure out the underlying search API and then issue customized requests directly to that instead of going via the UI.
Sorry, yes it's about the searching, not the booking.

I appreciate thata developer API might be better, it just so happens that itasodtware closed down its already limited QPX API last month.

So any thoughts on a js or vbnet based routine that jumps the itasoftware anti scraping hurdles would be welcome.
Jan@BRU is offline  
Reply With Quote
Old Dec 13, 18, 8:44 pm
  #7  
 
Join Date: Jun 2015
Location: SFO/SJC, BWI
Programs: :rolleyes:, DL DM, Mlife Plat, TR 7*, SPG/MR Plat, UA 1K
Posts: 8,979
Originally Posted by [email protected] View Post
Sorry, yes it's about the searching, not the booking.

I appreciate thata developer API might be better, it just so happens that itasodtware closed down its already limited QPX API last month.

So any thoughts on a js or vbnet based routine that jumps the itasoftware anti scraping hurdles would be welcome.
There may not be a public-facing API anymore but there's still whatever API their UI talks to. As I said above, my first thought absent any better suggestions would be to use Developer Tools in a browser, sniff whatever requests it's making and try to reverse engineer what it's doing.
Zorak is offline  
Reply With Quote
Old Dec 17, 18, 10:56 am
  #8  
Original Poster
 
Join Date: Sep 2004
Programs: LH HON, AF Plat, BA Gold, SPG Plat, HHo Gold
Posts: 139
Originally Posted by Zorak View Post
There may not be a public-facing API anymore but there's still whatever API their UI talks to. As I said above, my first thought absent any better suggestions would be to use Developer Tools in a browser, sniff whatever requests it's making and try to reverse engineer what it's doing.
Zorak,
oh yes thanks. I misread that. I obviously tried the sniff approach already. That works pretty cool with many websites, particularly if they use the URL to submit the request.
Unfortunately, this site is a lot more sophisticated and way beyond my capabilities. I cant even fill even the userform with all kinds of cookies and sessionIDs and what not ;-(
Jan@BRU is offline  
Reply With Quote
Old Dec 17, 18, 7:08 pm
  #9  
 
Join Date: Nov 2018
Location: San Francisco
Programs: DL
Posts: 75
ITA's API is quite obfuscated. It will probably be easier to screen-scrape it, but you'll presumably get blocked at some point, either because of IP-level throttling, JS checks, or something else. I don't know what ITA does in terms of anti-abuse but it's pretty easy to imagine many ways they can silently screw you (start randomizing prices, hiding low-fare classes, etc).
dlflyer00 is offline  
Reply With Quote
Old Dec 19, 18, 2:36 pm
  #10  
 
Join Date: Oct 2013
Posts: 440
yes, very easy to scrape ITA and no IP ban in place
fuyao is offline  
Reply With Quote
Old Jan 18, 19, 9:06 am
  #11  
 
Join Date: Nov 2007
Location: Beijing
Posts: 266
Originally Posted by fuyao View Post
yes, very easy to scrape ITA and no IP ban in place
care to elaborate please?
timesnaps is offline  
Reply With Quote
Old Jan 24, 19, 11:08 pm
  #12  
 
Join Date: Jun 2005
Location: Mostly 🇸🇬 often 🇫🇷 or 🇭🇰
Programs: Many
Posts: 4,580
Originally Posted by timesnaps View Post
care to elaborate please?
+1
bodory is offline  
Reply With Quote
Old Jan 27, 19, 4:29 pm
  #13  
 
Join Date: Sep 2012
Posts: 29
Has anyone tried scraping using selenium driver ? I tried but get different results on ita matrix. It shows only couple of airlines instead of all airlines.
maverick2202 is offline  
Reply With Quote

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are On
Refbacks are Off


Thread Tools
Search this Thread
 
  • Ask a Question
    Get answers from community experts
Question Title:
Description:
Your question will be posted in: