Jan@BRU

Just curious, whether anybody has succeeded in creating a scraping script for ITA software. I have seen a post over on headforpoints, which suggested it was possible. Any insight?

Thanks,

Jan

Zorak

What exactly do you mean by scraping?

Are you looking for functionality provided by either the Userscript or bookwithmatrix?

ITA-Matrix-PowerTools - Userscript for Orbitz/DL/UA/AA/BA/CZ/IB/LA/LH/LX/TK

BookWithMatrix.com: a tool to easily book with ITA Matrix

Jan@BRU

Zorak,
No -- that's what I mean. From what I understand, the tools you mention assist you in purchasing specific search results, which indeed is a big help.

Scraping refers to automating requests sent to a website and harvesting the results.
Take for example the recently introduced limitation on ITA to limit the jointly queried departure points to cities in a single country. If you want to do a regional query of flights from all departure points in neighboring countries, you need to several requests.
Scraping would run a script, send out a request, wait for the results to display and than store the results before moving on the next request automatically etc.
It's obvious that google discourages that kind of data harvesting and has put in place a few barriers to do it.... I was wondering whether somebody has managed to successfully scrape ITA nonetheless.

Thanks

Jan

Zorak

I know what scraping means in general, but wanted to understand your specific use case and whether it had to do with booking or searching.

IMO if someone wanted to automate the searching process or aggregate the results of multiple searches, a more direct approach might be to use Developer Tools to try and figure out the underlying search API and then issue customized requests directly to that instead of going via the UI.

gbs1112

Same problem discussed in a slightly different way. Here in Western Europe several international airports may be within 150 miles of each other but in four different countries so while the old ITA-Matrix allowed a quick search that has now become more tedious. Any suggestions how a searcher might extend to search 'airports within x miles'
Just a thought

Jan@BRU

Sorry, yes it's about the searching, not the booking.

I appreciate thata developer API might be better, it just so happens that itasodtware closed down its already limited QPX API last month.

So any thoughts on a js or vbnet based routine that jumps the itasoftware anti scraping hurdles would be welcome.

Zorak

There may not be a public-facing API anymore but there's still whatever API their UI talks to. As I said above, my first thought absent any better suggestions would be to use Developer Tools in a browser, sniff whatever requests it's making and try to reverse engineer what it's doing.

Jan@BRU

Zorak,
oh yes thanks. I misread that. I obviously tried the sniff approach already. That works pretty cool with many websites, particularly if they use the URL to submit the request.
Unfortunately, this site is a lot more sophisticated and way beyond my capabilities. I cant even fill even the userform with all kinds of cookies and sessionIDs and what not ;-(

dlflyer00

ITA's API is quite obfuscated. It will probably be easier to screen-scrape it, but you'll presumably get blocked at some point, either because of IP-level throttling, JS checks, or something else. I don't know what ITA does in terms of anti-abuse but it's pretty easy to imagine many ways they can silently screw you (start randomizing prices, hiding low-fare classes, etc).

fuyao

yes, very easy to scrape ITA and no IP ban in place

timesnaps

care to elaborate please?

bodory

+1

maverick2202

Has anyone tried scraping using selenium driver ? I tried but get different results on ita matrix. It shows only couple of airlines instead of all airlines.

UNC

I built a scraper that goes through the advanced controls (using selenium), and specifically searches for a given airline. The scraping seems to give higher prices than a manual search, but didn't bother to check further for the reason behind this. Maybe specifying the fare basis would help with this?

drvb

What about the new ITA Matrix, does this statement apply to that too?