Originally Posted by
VegasGambler
Don't send all the requests from the same place. It's fine if it take a few hours to scrape the data; this data doesn't change that quickly.
A true power-user solution that let you search any combo of origins/destinations/dates/cabins would quickly be making millions of requests. It’d need infra to manage said requests across data centers, logic to fuzz the requests so they didn’t all follow the same patterns, sanity checks and retries to get around GF sending back intentional fake results, etc … And it’d have to build all this hoping Google didn’t just decide to throttle the entire IP range of whatever compute solution it was using.
This would all cost way more money to build/operate than people would pay for it.