karthik

So I was replying to a post about Connexion outages in a Skype-over-Connexion thread and ended up writing out this long-winded explanation about the technical details of your data getting from the plane to the ground...

This covers those 2 - 4 minute dead spots you see on a Connexion flight, with details specific to a LAX-ICN flight on KE018 on May 9:

So as far as the short dead spots... Connexion routes data from the plane in a rather hackish fashion. I'll try to explain this in somewhat-simple terms: each plane in flight is assigned a /24 (a block of 256 IP addresses, which is the smallest size you can advertise to the world; this means letting the rest of the Internet know how to get to you.) The plane then advertises the /24 via whichever satellite/ground station it's connected to via BGP (Border Gateway Protocol, the protocol used to let routers on the Internet know where each block of IP addresses goes.) When you change satellites, and therefore change your BGP advertisement, it can take a couple minutes for the change to propagate across the Internet.

I did some traceroutes back to the plane from a server elsewhere after the three short dead spots, and was very intrigued to see the specifics of what I suspected. When we left LAX, the plane was communicating via a ground station near Vancouver. We stayed on that one until somewhere over eastern Asia (even on both sides of the 2 hour gap over northeastern Russia), where we switched over to a ground station in Japan (BGP change #1.) Then we switched to one in Moscow for a little bit as we went a little further west (BGP change #2.) Didn't stay on that one for long though; about 30 minutes later we were back to the Japan routing (BGP change #3) which lasted for the rest of the flight. So that explains what happens when you see a short drop in Connexion service but with stable connectivity other than that brief period.

When you change advertisements in this fashion, you create a phenomenon known as "flapping" as routers across the Internet see the routing topology for that advertisement change. Sometimes a single router will see the single change come in several times (the reasons behind that are probably beyond the scope of this forum.) Anyways, there's also a technique know as "dampening". This happens when a router sees a particular advertisement change too much too quickly, and the router dampens that advertisement for some time (longer the more flapping there is.) Most networks don't have dampening enabled, but for those that do, the BGP change can make those networks unreachable from the plane for a few minutes until the dampening times out.

So anyways... Connexion's handling of plane traffic is really just a big hack. The latency via the geosynchronous satellites they use is somewhere around 600 - 700ms roundtrip anyways. Internet traffic on the ground from, say, NYC to LAX is around 55ms roundtrip best case, and NYC to LON 70ms or so. So for most of these cases it wouldn't add appreciable latency to switch satellites but keep the IP traffic routed via a more central location. (Thus not using a /24 for each plane, and also not flapping routes just to track a moving plane.)

Oh and after I wrote all of that I googled and found an explanation from one of the folks over at Renesys with graphics and all. Differring opinions on the use of BGP for this type of task; I do agree minimizing latency is important though, and there really isn't a GOOD way to do it past what they're doing with the rest of the setup as-is. Maybe we'll eventually see Internet on planes via non-geosynchronous satellites or via radio to ground stations which will make it actually fast (and not just moderately high bandwidth.) Amusingly enough I was around for the NANOG that the further linked Boeing presentation occured at; though I wasn't actually attending the sessions.

Pegasus23

Enjoyed your post! Very helpful information. Enjoyed the links too!

ClueByFour

I boldly predict that when/if enough carriers and planes offer this, Boeing is gonna have to switch to a routing methodology that does not suck, since half the world (or the intelligent world, anyway) is going to dampen these prefixes into the stone age if they are flapping that much.

I'm also curious as to the "most networks don't have dampening enabled" comment--in my past work for an NSP, I found that most people did (and we certainly encouraged it on the part of our multihomed downstream customers). Or, to put it another way--I did not attend the NANOG where this was presentedu but I'd be shocked if the presenter was not raked over the coals, as Boeing basically admitted that they are going to have /24s coming from inconsistant ASNs constantly and that they had to pin a /19 annoucement up to keep the planes visible to the "dark corners of the internet." Not to mention blowing /24s not in the swamp out all over the place.

The correct way to do this is to pin some much shorter prefix to a global IP provider (or single ASN, if you will) who can provide decent connectivity to all of the ground stations. They are basically admitting to doing something which a great deal of the BGP speaking world would otherwise frown upon.

And the latency to a geostationary orbit bird is only about 240-280ms up and down. Everything else is either internal to the plane, or a function of the terrestrial connections.

A solid to Boeing--they are intentionally doing something poorly, probably to reduce their own costs (in terms of finding or building a single ASN that has decent connectivity at all the ground stations).

karthik

Right. This works fine for limited deployment, but if you have a few thousand planes in the air moving prefixes around people aren't going to be very happy with that. In limited deployment all but the real BGP purists will look at this as a very cool application ('cause it really is) and let slide the fact that anyone else doing similar with prefixes would get whapped in the head by the networking community.

It's been a couple years since I've actively handled day-to-day tasks at an NSP myself. I was trying to simplify a bit; you have to flap quite a bit to actually get a route surpressed of course, which is what I was getting at in a slightly mis-stated fashion. I just did a 'sh ip bgp damp' on route-views.oregon-ix.net and took a look at a random route from that output; of 49 paths, 14 had dampening info, slightly more than I expected but not that many. Not that a lot of people selectively filter legacy A space to /20 or legacy B space to /16 anymore, so swamp space doesn't quite have the same "benefits" these days. (Verio doesn't even seem to do it anymore, and they were the only large provider doing so in the past half-dozen years. I see the same ~185k routes from them on route-views as everyone else, whereas they used to be 15 - 25% down from everyone else from filtering selectively on length.) And, yeah, the inconsistent origin ASNs as prefixes move... Ugh. No reason they couldn't just use the same ASN for all plane advertisements, even if there's no real connectivity between the nodes. They can build tunnels for the occasional packet that might want to traverse their plane-AS (plane-to-plane Skype? )

This would definitely be one decent way of doing it; advertise that /19 or whatever to one or more upstreams, then advertise the /24s to the upstreams with a no-export community on them. They'd have to use common upstreams which could be a pain, though with the current ground station locations it's quite feasible. With more ground stations they may start to run into problems getting the same 2+ providers at each site.

Yeah; I was quoting roundtrip, so 2x up and down, with some ground-transit time added in. Closer to 500 - 600ms I guess.

Building a single AS is as simple as building tunnels as I suggested. The traffic flow between nodes in the plane-AS would be very minimal. The rest of each ground station can have its own AS for other purposes (and to be the upstream to the plane-AS.) Of course then they'd have to be yelled at for horribly inconsistent advertisements across their virtual plane-AS, and they'd still be flapping as planes move around of course, but at least they'd stick with the same origin ASN.

But in short--you're right, they're violating multiple unwritten common sense BGP rules, but on a small and reasonably tolerable (for now) scale. There are enough networks out there advertising pointless deaggregates, not filtering customers properly, and so on that are even worse offenders (and don't have a cool application and at least some justification to back it up.)

GUWonder

Interesting thread.

[I don't know if the following has any bearing or not on this, but apparently Boeing has (or at least had) two types of internet connectivity systems on board the Orange Popsicle -- which they were using for demonstrations -- and one of the systems was (and maybe still is ???) giving them quite a bit of trouble (at least in the third quarter of last year). That said, the one that gave them more trouble, IIRC, had to do with what was not then being used/installed on commercially scheduled pax airlines. ]

Also, with which countries was it that Boeing was having an issue (i.e., being told it's a no-go over their airspace) in relation to Connexion? And would that explain any of the dropping?

ClueByFour

Right. I'm okay with what they are doing now, but when it scales out to a few thousand prefixes, it's a horrid example to set. In full production, I'm one of those purists who would say that the "cool application" part of things does not let one slide..... Unless, of course, Boeing wants to fund the memory upgrade for all of my edge devices in which case they'll get a pass from me .

The other thing is this: if they did this from within a single-AS, they could get around using a /24 for each plane. Burning a globally routable /24 for each aircraft is, (in the IPv4 world) a horrid waste IMHO (I say this as somebody who is constantly juggling to keep an organization with two /16s under control). If you properly aggregate everything into "some" short-ish prefix, you can assign a plane-appropriate sized prefix to each aircraft (a 737 does not need a /24, whereas a 777, 340, or 744 might in theory need something larger someday).

karthik

Yeah. I'd like to see their ARIN justification forms.

"Netblock: 10.0.0.0/19
Usage: /24s allocated to individual planes. 1 IP for general use. 253 IPs reserved for...uh...umm...modem dialup pool on plane for legacy users to connect older laptops to via seatback RJ11 jacks"

I'd think any size plane should be the same. One IP to NAT all the passengers, then figure a few for internal use, so maybe a /28 or /29 onboard plus the /30 between the ground and the plane. And things like their "live TV" feed and connectivity to onboard servers should probably be in RFC1918 space.

That "live TV" point irks me slightly too. Apparently the downlink to each plane is 20mbit; 5mbit for end users, 5mbit for airline use, 5mbit for the "live TV" (which is pretty lame, 4 channels on your browser), 5mbit "reserved." Come on, at least drop the lame "reserved" space and give us a full 10mbit. Ever heard of QoS? RSVP/MPLS-TE? I'm actually curious how they're segmenting the 20mbit into four 5mbit channels; it doesn't appear to be CAR or other simple shaping of the sort. I saturated the 5mbit user bandwidth abusing lftp's pget while tcpdumping and mtring and didn't notice abnormally high latency from a big full buffer or any significant packet loss. The overall network design is a bit silly; I could see 3 access points from where I was on the upper deck of a 747 (so figure maybe 5 - 6 in the entire plane)... all on the same channel!

dizzy

One more reason to force ourselves to fully convert to IPv6

Serious Question: I'm pretty knowledgeable when it comes to computers. But, I've only recently gotten more in depth into Internet protocol, large network stuff, etc. And I have a question

I know that large cable internet/broadband companies ran into problems a couple of years ago when a lot of IP's were being assigned to people, and under IPv4, the # of ip's worldwide was fixed [my understanding is that the first octet? (whatever the #'s in ###.xxx.etc are) was supposed to be the institution, but I digress] there was a bit of a problem looming-we we going to run out of IP addresses.

But, we were saved because of the invention of NAT/NAT transversal. My understanding is that NAT allows for more ip's inside a network than are available in the connections between that network and the internet.

So, after all of that, here's my question:

Why can't Connexion/etc. use similar tech to give everyone on the plane the IP's without messing up the internet's routers?

karthik

That's not happening any time soon.

Right, the four parts of the IP address are octets, each an unsigned 8-bit integer (so 0 - 255.) 1.0.0.0 - 223.0.0.0 are useable as addresses on the general Internet with some exceptions; see RFC 3330. Initially, many sites did have an entire /8 (so the first octet fixed, and the rest useable; that's 16777216 IP addresses.) Many of these sites didn't need them and returned them, but not very many /8s were assigned. There are /8s in use that are being used well, and a few that aren't. (For example, MIT is 18.0.0.0/8. They don't need 2^24 IPs and could get by fine with a /16, which is 2^16 IPs, 65536. Stanford had a /8 as well, 36.0.0.0/8 I think, but they were good Internet citizens and returned it in exchange for a /16 or two.)

These days addresses are much more sanely assigned; you need to fill in justification forms for the amount of IP space you request and such. There was never any shortage of IPs or any problems with anyone getting the amount of space they needed. There is plenty of empty space still left; we're 5+ years before having to worry about getting low on space with IPv4. This is just a random estimate, but I'd like to say we can easily double the number of computers on the Internet before perhaps running low on IP space. Here's some data on current status of how much IP space is assigned, allocated to registries for assignment, and actually advertised and in use.

NAT does help, but it didn't quite "save" the Internet. NAT's other important role is keeping LANs isolated from incoming connections from the Internet at large. NAT traversal, by the way, is a technique to allow protocols to work through NAT that otherwise wouldn't (primarily VPNs); it's not directly involved in helping reduce IP usage.

They do. You get a NATted IP when you're on the plane using Connexion. Any time you get an IP out of 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16 (the last being the most popular space for LANs, though 10 space isn't far behind; see RFC1918 for more information on private IP addressing.) Most random wireless networks, home LANs, etc will hand out an RFC1918 IP address to you.

The problem is that the plane still needs a public IP address in order to be able to route traffic through the closest ground station. Applying NAT to the actual plane itself does nothing (you'd have to route traffic through a "home" location; and you can accomplish this with a few public IPs anyways.)

ClueByFour

There was something done in the late 1990s which did help (indirectly) to help keep the ipv4 space from becoming full: CIDR (Classless Inter-Domain Routing).

Used to be that routers only understood 4 "classfull" subnet masks: 255.0.0.0, 255.255.0.0, 255.255.255.0, and 255.255.255.255 (eg, /8,/16/24,/32). Each location that needed IPs basically had to have at least 256 (/24), many needed the /16 (65k IPs and change) when they might only need 500 addresses (or, for that matter, 270 IPs).

CIDR allows routing of any size subnet (or, to be more correct, any sized subnet mask).

The other thing is that many providers did not, and some still do not, recognize annoucements of anything "smaller" (or "longer" in terms of subnet masking) than a /19 (32 Class C blocks, or approximately 8192 addresses). This helps to reduce the size of the internet "full" routing table that ISPs and people with multiple connections need to carry.

NAT has helped, but it took a few years to make most common services work with/thru it. NAT is much more useful (as mentioned above) because it offers a layer of "poor man's security."

karthik

Right; I've been in the industry short enough to just take CIDR for granted. It certainly prevented things from falling apart. Not to quibble too much, but if there is that type of filtering it tends to be to /20... because that's the smallest allocation out of legacy class A space (with an exception for /24 in legacy class C space of course.) Most providers are just filtering to /24 for all routes now though... (Verio for a while was even filtering legacy B space to /16 since that's the smallest allocated out of there.)

Just a quick 'sh ip bgp' on route-views.oregon-ix.net shows a ton of peers showing a handful of routes smaller than 4.0.0.0/8 in that block; a practice Genuity and now Level3 actively disallows and pursues customers doing. If they want to multihome, they're told to get their own space and not do it out of 4-space. If they weren't you'd see far more smaller routes there. By comparison, there are heaps of smaller-than-/20 routes with good global visibility inside 12.0.0.0/8 as AT&T doesn't seem to care if people advertise subnets of that space to other upstreams.

ClueByFour

The /20 filter actually came after most people started whacking anything longer than /19, but for a "quasi" practical reason:

ARIN used to (right after CIDR started--dunno what their practice is now because I've not been in the provider game for 5 or 6 years) have "starter" allocations: they would give you a /20, but "reserve" the other half of the /19 for you if you could not justify a /19. In the interest of "fairness" (loose) they'd let you announce the whole /19, but not everybody did. Ergo, some folks loosened the filter to a /20. Lots of people with /20s out of the A space took in in the shorts about 1996 or 1997 (IIRC) because of this. By 1998, you could announce a /20 from anywhere and generally expect it to be globally routed.

Some of the latter (the AT&T multihomed customers) only get global visibility because AT&T happily blow an exception into their outbound annoucements to announce both 12.0.0.0/8 and whatever longer prefix the customer announces. Somebody like Verio used to (and may still) filter the customer's /24 and only reach them via their aggregated association with 12/8. If that customer's AT&T circuit went down, they'd probably be screwed.

The most valuable thing there is anymore is a /24 out of the swamp. Fortunately, I have one in reserve .

Edited to add: the idea of "micro-allocations" is still enough to provoke a religious war among BGP zealots of various positions on the matter.