[kwlug-disc] Bandwidth aggregation
Andrew Kohlsmith (lists)
aklists at mixdown.ca
Wed Feb 11 09:59:02 EST 2009
On February 10, 2009 03:52:44 pm L.D. Paniak wrote:
> I think it would be a great topic. WAN connections are easily the most
> unreliable part of a modern computer system. For things like VOIP,
> failover capability is essentially a requirement.
> Is it really that esoteric? Don't most homes/businesses have access to
> cable and DSL or two channels of DSL?
Yes it is. I said "multihomed IP space" -- that is not the same thing as two
DSL or cable connections at all.
Multihomed IP space means an IP address which is reachable through two or more
providers. I'm not aware of any DSL, Cable, Cell or Wifi providers which give
you this kind of address.
If you get 188.8.131.52 from your DSL provider and 184.108.40.206 through your cable
provider, both of those addresses are single-homed from your perspective. If
someone on the internet sends a packet to 220.127.116.11, it will ONLY come from
your DSL line, and never from your cable. If you send a request out your DSL
connection, the response will come back through your DSL connection. It
won't balance between the DSL and cable connections. You can try to fake
your source address to have a request go out one and back the other, but any
ISP worth their salt will be dropping packets not sourced from their own IP
What I am doing is "extruding" multihomed IP space through a number of
single-homed connections. Through SCS Internet, I have access to multihomed
IP space. (e.g. 18.104.22.168/24.) Traffic to any of these addresses can come
from one of (I think) three or four different providers into his rack.
Using a server in that rack (with IP 22.214.171.124, for example), I "extrude" an
IP, say 126.96.36.199. I use proxy-arp to as a means to capture traffic for that
IP. Basically that means that whenever someone asks for the ethernet address
of 188.8.131.52, the server responds with its own MAC.
Your router at the office has a DSL and cable connection, with IP 184.108.40.206 and
220.127.116.11 from your respective providers. What I do now is to set up two GRE
tunnels, one from 18.104.22.168 to 22.214.171.124 and one from 126.96.36.199 to 188.8.131.52. So
now your home router actually has four routes: DSL-to-world, cable-to-world,
What I do now is to use iproute2 and iptables to set up a packet marking and
load balancing system. I say that traffic for the world (i.e. default route)
is via DSL-to-server-via-tunnel1 *AND* to cable-to-server-via-tunnel1, with
equal weighting (neither route is preferred). I also to disable route
caching, which works against us in this scenario. Now any traffic that the
router wants to send out to the world will go through either tunnel and "pop
out" as traffic from 184.108.40.206.
It may have taken the path through the cable or DSL IP, but it was
encapsulated in the tunnel at that point. The traffic exits from 151 Front
with a source IP of 220.127.116.11 and heads out through any of the providers that
he has available. It reaches the destination, and a response comes back to
18.104.22.168. The server receives that traffic, sends it back through one of the
two tunnels, and your router gets it and does what it would normally do with
it. All the tunnels are brought up and torn down with your standard ip-up
style scripts that PPPoE gives, but you could also do it with the normal
networking post-route scripts (for cable or wifi connections, for example).
If a tunnel fails, the other tunnel's there, and traffic still flows. If you
add more tunnels, the bandwidth is load-balanced over them. Since the
22.214.171.124 address is multihomed, you don't lose connectivity when one of the
providers that SCS Internet uses has troubles, since that IP is reachable
through several other paths. You can play with route metrics to favour one
tunnel over the other, and of course you can play with the routing table
selections on the router so that you do NOT use the multihomed space for
regular web traffic, windows updates or whatever other nonessential traffic
you may have.
You can *also* use IPSec to encrypt everything over the tunnel.
The only real downsides are
1) bandwidth considerations at the colo
2) slightly increased latency due to tunnelling and encryption
I've used this method to provide highly-available bandwidth for the auto
industry, where they were willing to pay for the colocated server at 151
Front -- multihomed IP space was simply not available at some of their
plants, and simply too expensive at others. They decided to get more bang for
their buck by using the colocated server as an IPSec endpoint, FTP server for
*huge* CAD drawings and a conferencing bridge instead of bringing absolutely
all that traffic back to their home network.
Anyway -- that's why I feel it's an esoteric topic. You can bring two DSL or
cable connections together for home/office use, but that's not the same thing
as what I described.
More information about the kwlug-disc