[kwlug-disc] Bandwidth aggregation

[Andrew Kohlsmith (lists)]

On February 10, 2009 03:52:44 pm L.D. Paniak wrote:
> I think it would be a great topic.  WAN connections are easily the most
> unreliable part of a modern computer system.  For things like VOIP,
> failover capability is essentially a requirement.
>
> Is it really that esoteric?  Don't most homes/businesses have access to
> cable and DSL or two channels of DSL?

Yes it is.  I said "multihomed IP space" -- that is not the same thing as two 
DSL or cable connections at all.

Multihomed IP space means an IP address which is reachable through two or more 
providers. I'm not aware of any DSL, Cable, Cell or Wifi providers which give 
you this kind of address.

If you get 55.1.2.3 from your DSL provider and 99.1.2.3 through your cable 
provider, both of those addresses are single-homed from your perspective.  If 
someone on the internet sends a packet to 55.1.2.3, it will ONLY come from 
your DSL line, and never from your cable.  If you send a request out your DSL 
connection, the response will come back through your DSL connection.  It 
won't balance between the DSL and cable connections.  You can try to fake 
your source address to have a request go out one and back the other, but any 
ISP worth their salt will be dropping packets not sourced from their own IP 
space.

What I am doing is "extruding" multihomed IP space through a number of 
single-homed connections.  Through SCS Internet, I have access to multihomed 
IP space.  (e.g. 77.1.2.0/24.)  Traffic to any of these addresses can come 
from one of (I think) three or four different providers into his rack.

Using a server in that rack (with IP 77.1.2.2, for example), I "extrude" an 
IP, say 77.1.2.3.  I use proxy-arp to as a means to capture traffic for that 
IP.  Basically that means that whenever someone asks for the ethernet address 
of 77.1.2.3, the server responds with its own MAC.

Your router at the office has a DSL and cable connection, with IP 55.1.2.3 and 
99.1.2.3 from your respective providers.  What I do now is to set up two GRE 
tunnels, one from 55.1.2.3 to 77.1.2.2 and one from 99.1.2.3 to 77.1.2.2.  So 
now your home router actually has four routes: DSL-to-world, cable-to-world, 
DSL-to-server-via-tunnel0, cable-to-server-via-tunnel1.

What I do now is to use iproute2 and iptables to set up a packet marking and 
load balancing system.  I say that traffic for the world (i.e. default route) 
is via DSL-to-server-via-tunnel1 *AND* to cable-to-server-via-tunnel1, with 
equal weighting (neither route is preferred). I also to disable route 
caching, which works against us in this scenario.  Now any traffic that the 
router wants to send out to the world will go through either tunnel and "pop 
out" as traffic from 77.1.2.3.

It may have taken the path through the cable or DSL IP, but it was 
encapsulated in the tunnel at that point.  The traffic exits from 151 Front 
with a source IP of 77.1.2.3 and heads out through any of the providers that 
he has available.  It reaches the destination, and a response comes back to 
77.1.2.3.  The server receives that traffic, sends it back through one of the 
two tunnels, and your router gets it and does what it would normally do with 
it.  All the tunnels are brought up and torn down with your standard ip-up 
style scripts that PPPoE gives, but you could also do it with the normal 
networking post-route scripts (for cable or wifi connections, for example).

If a tunnel fails, the other tunnel's there, and traffic still flows.  If you 
add more tunnels, the bandwidth is load-balanced over them.  Since the 
77.1.2.3 address is multihomed, you don't lose connectivity when one of the 
providers that SCS Internet uses has troubles, since that IP is reachable 
through several other paths.  You can play with route metrics to favour one 
tunnel over the other, and of course you can play with the routing table 
selections on the router so that you do NOT use the multihomed space for 
regular web traffic, windows updates or whatever other nonessential traffic 
you may have.

You can *also* use IPSec to encrypt everything over the tunnel.

The only real downsides are
1) bandwidth considerations at the colo
2) slightly increased latency due to tunnelling and encryption

I've used this method to provide highly-available bandwidth for the auto 
industry, where they were willing to pay for the colocated server at 151 
Front -- multihomed IP space was simply not available at some of their 
plants, and simply too expensive at others. They decided to get more bang for 
their buck by using the colocated server as an IPSec endpoint, FTP server for 
*huge* CAD drawings and a conferencing bridge instead of bringing absolutely 
all that traffic back to their home network.

Anyway -- that's why I feel it's an esoteric topic.  You can bring two DSL or 
cable connections together for home/office use, but that's not the same thing 
as what I described.

-A.