Lostdomain

Doing Dual ISP Load Balancing with Ubiquiti EdgeRouter

Date

Wed Apr 11

Author

Martijn

Tags

  1. networking
  2. ubiquiti
  3. wan

Ever since I moved to a new house, I’ve been stuck to a pretty bad ISP. With no fiber, a few kilometers away from the DSL termination pop (so a max of 10Mbit on ADSL), it just leaves cable. Speeds aren’t terrible, I get a 400Mbit line for E 60,- p/m. Latency spikes and jitter are horrible, but that can be expected on a cable network. Especially fun when doing internet calls. ;-)

All in all, I haven’t been enjoying my internet connection for a while and I wanted to do something about it.

Dual Connection

After dismissing the thought of moving house again, I decided to get an extra low speed, but quality connection. ADSL with a trusted provider turned out to be the best option. I planned on using VeloCloud to balance between the connections, but the new connection was actually delivered a few weeks early. I didn’t have the VeloCloud design ready yet, but did wanted to hook up the new connection and test it out.

Enter the Ubiquiti EdgeRouter I already had in place, as it appears it is able to load balancing between multiple connections.

Ubiquiti EdgeRouter

The EdgeRouter is a really good router for the network enthusiast, as it is packed with features that you can geek out with. One of those is the ability to create load balancing groups between connections, for outgoing network traffic. I linked to an article from Ubiquity themselves but found it lacking commentary and it missed a couple of extra things I wanted to do, so I decided to do a write-up of my setup.

Wanted Topology

This is the topology I was going for:

Home-Network-1

This was already in place for the cable provider, so all I did here was bring in the ADSL provider to a VLAN and transport that up to my EdgeRouter.

EdgeRouter Configuration

Let’s get into the weeds and begin with the interface configuration. All configuration is going to be done by the CLI by the way, most of this cannot be done via the UI.

Interfaces

As pictured earlier, I’m using VLANs to transport the internet connections to the EdgeRouter. This means the interface configuration looks like this:

set interfaces ethernet eth0 vif 98 address dhcp
set interfaces ethernet eth0 vif 98 description 'Internet ADSL'
set interfaces ethernet eth0 vif 98 dhcp-options default-route update
set interfaces ethernet eth0 vif 98 dhcp-options default-route-distance 210
set interfaces ethernet eth0 vif 98 dhcp-options name-server no-update
set interfaces ethernet eth0 vif 98 firewall in name Internet-ADSL-Incoming
set interfaces ethernet eth0 vif 99 address dhcp
set interfaces ethernet eth0 vif 99 description 'Internet Cable'
set interfaces ethernet eth0 vif 99 dhcp-options default-route update
set interfaces ethernet eth0 vif 99 dhcp-options default-route-distance 150
set interfaces ethernet eth0 vif 99 dhcp-options name-server no-update
set interfaces ethernet eth0 vif 99 firewall in name Internet-Cable-Incoming

A couple things to note:

  • Both connections have a DHCP IP address attached, but you can do this with a static IP as well.
  • The default-route-distance determines how to install the default route in the global routing table. In this case I opted to use the cable provider as the primary connection by giving it a lower distance then the ADSL connection.
  • I didn’t want the DHCP client to update my DNS servers for the EdgeRouter (because I have my own) and specified dhcp-options name-server no-update.

Extra Configuration

Just to make sure I paint a full picture, below is an overview of some extra configuration (firewall & NAT) which is needed to get this working:

! Firewall rules: Drop everything from outside to in, except established sessions
set firewall name Internet-Cable-Incoming default-action drop
set firewall name Internet-Cable-Incoming rule 10 action accept
set firewall name Internet-Cable-Incoming rule 10 description 'Allow all established'
set firewall name Internet-Cable-Incoming rule 10 protocol all
set firewall name Internet-Cable-Incoming rule 10 state established enable
set firewall name Internet-Cable-Incoming rule 10 state invalid disable
set firewall name Internet-Cable-Incoming rule 10 state new disable
set firewall name Internet-Cable-Incoming rule 10 state related disable
set firewall name Internet-Cable-Incoming rule 200 action drop
set firewall name Internet-Cable-Incoming rule 200 description 'Drop the rest'
set firewall name Internet-Cable-Incoming rule 200 protocol all
set firewall name Internet-ADSL-Incoming default-action drop
set firewall name Internet-ADSL-Incoming rule 10 action accept
set firewall name Internet-ADSL-Incoming rule 10 description 'Allow all established'
set firewall name Internet-ADSL-Incoming rule 10 protocol all
set firewall name Internet-ADSL-Incoming rule 10 state established enable
set firewall name Internet-ADSL-Incoming rule 10 state invalid disable
set firewall name Internet-ADSL-Incoming rule 10 state new disable
set firewall name Internet-ADSL-Incoming rule 10 state related disable
set firewall name Internet-ADSL-Incoming rule 200 action drop
set firewall name Internet-ADSL-Incoming rule 200 description 'Drop rest'
set firewall name Internet-ADSL-Incoming rule 200 protocol all

! Masquerade everything from inside to outside with SNAT
set service nat rule 5000 description 'SNAT To Cable'
set service nat rule 5000 outbound-interface eth0.99
set service nat rule 5000 protocol all
set service nat rule 5000 type masquerade
set service nat rule 5001 description 'SNAT To ADSL'
set service nat rule 5001 outbound-interface eth0.98
set service nat rule 5001 protocol all
set service nat rule 5001 type masquerade

WAN Load Balancing Configuration

So far we’ve only connected 2 connections which uses the cable connection as a primary connection and failover to the ADSL connection when the cable fails. This is also a long failure, because the DHCP lease needs to expire before the routing switches over. This is one of the issues that the WAN load balancing within the EdgeRouter fixes by doing monitoring on the connection.

First, let’s configure a load balancing group:

set load-balance group DUAL_ISP interface eth0.98 route-test initial-delay 15
set load-balance group DUAL_ISP interface eth0.98 route-test interval 5
set load-balance group DUAL_ISP interface eth0.98 route-test type ping target 1.1.1.1
set load-balance group DUAL_ISP interface eth0.98 weight 10
set load-balance group DUAL_ISP interface eth0.99 route-test initial-delay 15
set load-balance group DUAL_ISP interface eth0.99 route-test interval 5
set load-balance group DUAL_ISP interface eth0.99 route-test type ping target 1.1.1.1
set load-balance group DUAL_ISP interface eth0.99 weight 90

Here, I’ve divided up the connections with weights so the cable connection gets used for 90% of the load and the ADSL connection gets 10% of the load. This is due to the difference in speeds. Furthermore, both connections will be monitored by doing a ping to 1.1.1.1 every 5 seconds (by default it marks it as failed when 3 pings are missed).

Once you have this config in place, you can verify the monitor is working by executing this:

admin@gw:~$ show load-balance watchdog 
Group DUAL_ISP_GROUP
  eth0.98
  status: Running 
  pings: 6221
  fails: 2
  run fails: 0/3
  route drops: 0
  ping gateway: 1.1.1.1 - REACHABLE

  eth0.99
  status: Running 
  pings: 6228
  fails: 0
  run fails: 0/3
  route drops: 0
  ping gateway: 1.1.1.1 - REACHABLE

Now that you have the load balancing in place, it needs to take effect. You do this by defining a firewall modify ruleset which points to the load balancing group:

set firewall group network-group Internal-Networks network 10.0.0.0/8
set firewall group network-group Internal-Networks network 192.168.0.0/16
set firewall group network-group Internal-Networks network 172.16.0.0/12
set firewall group network-group ADDRv4_vlan98 network 192.x.x.x/24
set firewall group network-group ADDRv4_vlan99 network 77.x.x.x/24

set firewall modify ISP_BALANCE rule 10 action modify
set firewall modify ISP_BALANCE rule 10 description 'Skip LAN to LAN'
set firewall modify ISP_BALANCE rule 10 destination group network-group Internal-Networks
set firewall modify ISP_BALANCE rule 10 modify table main
set firewall modify ISP_BALANCE rule 20 action modify
set firewall modify ISP_BALANCE rule 20 description 'Skip the external IP addresses'
set firewall modify ISP_BALANCE rule 20 destination group network-group ADDRv4_vlan99
set firewall modify ISP_BALANCE rule 20 modify table main
set firewall modify ISP_BALANCE rule 30 action modify
set firewall modify ISP_BALANCE rule 30 description 'Skip the external IP addresses'
set firewall modify ISP_BALANCE rule 30 destination group network-group ADDRv4_vlan98
set firewall modify ISP_BALANCE rule 30 modify table main
set firewall modify ISP_BALANCE rule 70 action modify
set firewall modify ISP_BALANCE rule 70 modify lb-group DUAL_ISP

set interfaces ethernet eth1 vif 10 firewall in modify ISP_BALANCE

A couple things to note:

  • Rule 10, 20 & 30 make sure it doesn’t load balance between internal ranges (LAN to LAN) and not load balance towards the actual IPs of the external interfaces. (I specified a /24 there because those can change). It’s weird that the EdgeRouter doesn’t exclude these by default, yes.
  • Rule 70 is the one that actually redirects traffic to the load balancing group.
  • Interface eth1.10 is one of my internal networks. You need this line on any internal network you’d like to load balance.

Once the load balancing group is applied to an internal interface, you can verify that is it balancing traffic by executing this:

admin@gw:~$ show load-balance status 
Group DUAL_ISP
  interface   : eth0.98
  carrier     : up
  status      : active
  gateway     : 192.x.x.x
  route table : 201
  weight      : 10%
  flows
      WAN Out : 4560
      WAN In  : 910
    Local Out : 30619

  interface   : eth0.99
  carrier     : up
  status      : active
  gateway     : 77.x.x.x
  route table : 202
  weight      : 90%
  flows
      WAN Out : 17891
      WAN In  : 4657
    Local Out : 121000

admin@gw:~$ show ip route table 201
default via 192.x.x.1 dev eth0.98 
..snip..
admin@gw:~$ show ip route table 202
default via 77.x.x.1 dev eth0.99 
..snip..
admin@gw:~$

The WAN Out/In counters will go up the more they route traffic and the table 201 and 202 are filled with the routes for the different connections.

Exceptions

Of course, I wanted to make some exceptions to generally load balancing and failover. One of these examples was that my download server always used up the cable connection (with its 400Mbit) instead of bothering the ADSL line and that my own computer was to use the ADSL line as primary, only failing over to cable if the ADSL line goes down. The latter was important because the ADSL line is much better when it comes to voice calls, webinars, etc.

To facilitate a configuration where something used the ADSL connection as primary and the cable connection as secondary, I had to create a new load balancing group:

set load-balance group DUAL_ISP_ADSL_Pri interface eth0.98 route-test initial-delay 15
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.98 route-test interval 5
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.98 route-test type ping target 1.1.1.1
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.99 failover-only
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.99 route-test initial-delay 15
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.99 route-test interval 5
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.99 route-test type ping target 1.1.1.1

You’ll only notice 2 differences: I removed the weights and added failover-only to eth0.99 (which is the cable connection). This gives the result that eth0.98 will always be used unless the ping test fails, which will make it failover to eth0.99.

To effect this new load balancing group for a specific client, I’ve used this snippet:

set firewall modify ISP_BALANCE rule 50 action modify
set firewall modify ISP_BALANCE rule 50 modify lb-group DUAL_ISP_ADSL_Pri
set firewall modify ISP_BALANCE rule 50 source group address-group Martijn-iMac-Macbook

This inserts a rule in the existing ISP_BALANCE ruleset where the load balancing is being called, takes a source address-group (in this case named Martijn-iMac-Macbook) and uses that to decide to redirect the traffic to the new load balancing group. The address group simply contains the IP addresses that my iMac and Macbook have.

Another variation on this is doing this on a destination basis:

set firewall modify ISP_BALANCE rule 52 action modify
set firewall modify ISP_BALANCE rule 52 destination group address-group LD-VPN-GW-EXT
set firewall modify ISP_BALANCE rule 52 modify lb-group DUAL_ISP_ADSL_Pri

Here I make sure that a VPN tunnel endpoint on the internet (IP included in the address group LD-VPN-GW-EXT) always goes via the ADSL connection and fails back to the cable connection if needed.

Conclusion

While this is a temporary situation for me (while I figure out a network overhaul and include VeloCloud for this functionality), the EdgeRouter really does a good job with load balancing between multiple links and the possibilities to make exceptions based on source or destination are awesome.