Doing Dual ISP Load Balancing with Ubiquiti EdgeRouter

April 11, 2018 / Martijn / 17 Comments

Ever since I moved to a new house, I’ve been stuck to a pretty bad ISP. With no fiber, a few kilometers away from the DSL termination pop (so a max of 10Mbit on ADSL), it just leaves cable. Speeds aren’t terrible, I get a 400Mbit line for E 60,- p/m. Latency spikes and jitter are horrible, but that can be expected on a cable network. Especially fun when doing internet calls. 😉

All in all, I haven’t been enjoying my internet connection for a while and I wanted to do something about it.

Dual Connection

After dismissing the thought of moving house again, I decided to get an extra low speed, but quality connection. ADSL with a trusted provider turned out to be the best option. I planned on using VeloCloud to balance between the connections, but the new connection was actually delivered a few weeks early. I didn’t have the VeloCloud design ready yet, but did wanted to hook up the new connection and test it out.

Enter the Ubiquiti EdgeRouter I already had in place, as it appears it is able to load balancing between multiple connections.

Ubiquiti EdgeRouter

The EdgeRouter is a really good router for the network enthusiast, as it is packed with features that you can geek out with. One of those is the ability to create load balancing groups between connections, for outgoing network traffic. I linked to an article from Ubiquity themselves but found it lacking commentary and it missed a couple of extra things I wanted to do, so I decided to do a write-up of my setup.

Wanted Topology

This is the topology I was going for:

This was already in place for the cable provider, so all I did here was bring in the ADSL provider to a VLAN and transport that up to my EdgeRouter.

EdgeRouter Configuration

Let’s get into the weeds and begin with the interface configuration. All configuration is going to be done by the CLI by the way, most of this cannot be done via the UI.

Interfaces

As pictured earlier, I’m using VLANs to transport the internet connections to the EdgeRouter. This means the interface configuration looks like this:

set interfaces ethernet eth0 vif 98 address dhcp
set interfaces ethernet eth0 vif 98 description 'Internet ADSL'
set interfaces ethernet eth0 vif 98 dhcp-options default-route update
set interfaces ethernet eth0 vif 98 dhcp-options default-route-distance 210
set interfaces ethernet eth0 vif 98 dhcp-options name-server no-update
set interfaces ethernet eth0 vif 98 firewall in name Internet-ADSL-Incoming
set interfaces ethernet eth0 vif 99 address dhcp
set interfaces ethernet eth0 vif 99 description 'Internet Cable'
set interfaces ethernet eth0 vif 99 dhcp-options default-route update
set interfaces ethernet eth0 vif 99 dhcp-options default-route-distance 150
set interfaces ethernet eth0 vif 99 dhcp-options name-server no-update
set interfaces ethernet eth0 vif 99 firewall in name Internet-Cable-Incoming

A couple things to note:

Both connections have a DHCP IP address attached, but you can do this with a static IP as well.
The default-route-distance determines how to install the default route in the global routing table. In this case I opted to use the cable provider as the primary connection by giving it a lower distance then the ADSL connection.
I didn’t want the DHCP client to update my DNS servers for the EdgeRouter (because I have my own) and specified dhcp-options name-server no-update.

Extra Configuration

Just to make sure I paint a full picture, below is an overview of some extra configuration (firewall & NAT) which is needed to get this working:

! Firewall rules: Drop everything from outside to in, except established sessions
set firewall name Internet-Cable-Incoming default-action drop
set firewall name Internet-Cable-Incoming rule 10 action accept
set firewall name Internet-Cable-Incoming rule 10 description 'Allow all established'
set firewall name Internet-Cable-Incoming rule 10 protocol all
set firewall name Internet-Cable-Incoming rule 10 state established enable
set firewall name Internet-Cable-Incoming rule 10 state invalid disable
set firewall name Internet-Cable-Incoming rule 10 state new disable
set firewall name Internet-Cable-Incoming rule 10 state related disable
set firewall name Internet-Cable-Incoming rule 200 action drop
set firewall name Internet-Cable-Incoming rule 200 description 'Drop the rest'
set firewall name Internet-Cable-Incoming rule 200 protocol all
set firewall name Internet-ADSL-Incoming default-action drop
set firewall name Internet-ADSL-Incoming rule 10 action accept
set firewall name Internet-ADSL-Incoming rule 10 description 'Allow all established'
set firewall name Internet-ADSL-Incoming rule 10 protocol all
set firewall name Internet-ADSL-Incoming rule 10 state established enable
set firewall name Internet-ADSL-Incoming rule 10 state invalid disable
set firewall name Internet-ADSL-Incoming rule 10 state new disable
set firewall name Internet-ADSL-Incoming rule 10 state related disable
set firewall name Internet-ADSL-Incoming rule 200 action drop
set firewall name Internet-ADSL-Incoming rule 200 description 'Drop rest'
set firewall name Internet-ADSL-Incoming rule 200 protocol all

! Masquerade everything from inside to outside with SNAT
set service nat rule 5000 description 'SNAT To Cable'
set service nat rule 5000 outbound-interface eth0.99
set service nat rule 5000 protocol all
set service nat rule 5000 type masquerade
set service nat rule 5001 description 'SNAT To ADSL'
set service nat rule 5001 outbound-interface eth0.98
set service nat rule 5001 protocol all
set service nat rule 5001 type masquerade

WAN Load Balancing Configuration

So far we’ve only connected 2 connections which uses the cable connection as a primary connection and failover to the ADSL connection when the cable fails. This is also a long failure, because the DHCP lease needs to expire before the routing switches over. This is one of the issues that the WAN load balancing within the EdgeRouter fixes by doing monitoring on the connection.

First, let’s configure a load balancing group:

set load-balance group DUAL_ISP interface eth0.98 route-test initial-delay 15
set load-balance group DUAL_ISP interface eth0.98 route-test interval 5
set load-balance group DUAL_ISP interface eth0.98 route-test type ping target 1.1.1.1
set load-balance group DUAL_ISP interface eth0.98 weight 10
set load-balance group DUAL_ISP interface eth0.99 route-test initial-delay 15
set load-balance group DUAL_ISP interface eth0.99 route-test interval 5
set load-balance group DUAL_ISP interface eth0.99 route-test type ping target 1.1.1.1
set load-balance group DUAL_ISP interface eth0.99 weight 90

Here, I’ve divided up the connections with weights so the cable connection gets used for 90% of the load and the ADSL connection gets 10% of the load. This is due to the difference in speeds. Furthermore, both connections will be monitored by doing a ping to 1.1.1.1 every 5 seconds (by default it marks it as failed when 3 pings are missed).

Once you have this config in place, you can verify the monitor is working by executing this:

admin@gw:~$ show load-balance watchdog 
Group DUAL_ISP_GROUP
  eth0.98
  status: Running 
  pings: 6221
  fails: 2
  run fails: 0/3
  route drops: 0
  ping gateway: 1.1.1.1 - REACHABLE

  eth0.99
  status: Running 
  pings: 6228
  fails: 0
  run fails: 0/3
  route drops: 0
  ping gateway: 1.1.1.1 - REACHABLE

Now that you have the load balancing in place, it needs to take effect. You do this by defining a firewall modify ruleset which points to the load balancing group:

set firewall group network-group Internal-Networks network 10.0.0.0/8
set firewall group network-group Internal-Networks network 192.168.0.0/16
set firewall group network-group Internal-Networks network 172.16.0.0/12
set firewall group network-group ADDRv4_vlan98 network 192.x.x.x/24
set firewall group network-group ADDRv4_vlan99 network 77.x.x.x/24

set firewall modify ISP_BALANCE rule 10 action modify
set firewall modify ISP_BALANCE rule 10 description 'Skip LAN to LAN'
set firewall modify ISP_BALANCE rule 10 destination group network-group Internal-Networks
set firewall modify ISP_BALANCE rule 10 modify table main
set firewall modify ISP_BALANCE rule 20 action modify
set firewall modify ISP_BALANCE rule 20 description 'Skip the external IP addresses'
set firewall modify ISP_BALANCE rule 20 destination group network-group ADDRv4_vlan99
set firewall modify ISP_BALANCE rule 20 modify table main
set firewall modify ISP_BALANCE rule 30 action modify
set firewall modify ISP_BALANCE rule 30 description 'Skip the external IP addresses'
set firewall modify ISP_BALANCE rule 30 destination group network-group ADDRv4_vlan98
set firewall modify ISP_BALANCE rule 30 modify table main
set firewall modify ISP_BALANCE rule 70 action modify
set firewall modify ISP_BALANCE rule 70 modify lb-group DUAL_ISP

set interfaces ethernet eth1 vif 10 firewall in modify ISP_BALANCE

A couple things to note:

Rule 10, 20 & 30 make sure it doesn’t load balance between internal ranges (LAN to LAN) and not load balance towards the actual IPs of the external interfaces. (I specified a /24 there because those can change). It’s weird that the EdgeRouter doesn’t exclude these by default, yes.
Rule 70 is the one that actually redirects traffic to the load balancing group.
Interface eth1.10 is one of my internal networks. You need this line on any internal network you’d like to load balance.

Once the load balancing group is applied to an internal interface, you can verify that is it balancing traffic by executing this:

admin@gw:~$ show load-balance status 
Group DUAL_ISP
  interface   : eth0.98
  carrier     : up
  status      : active
  gateway     : 192.x.x.x
  route table : 201
  weight      : 10%
  flows
      WAN Out : 4560
      WAN In  : 910
    Local Out : 30619

  interface   : eth0.99
  carrier     : up
  status      : active
  gateway     : 77.x.x.x
  route table : 202
  weight      : 90%
  flows
      WAN Out : 17891
      WAN In  : 4657
    Local Out : 121000

admin@gw:~$ show ip route table 201
default via 192.x.x.1 dev eth0.98 
..snip..
admin@gw:~$ show ip route table 202
default via 77.x.x.1 dev eth0.99 
..snip..
admin@gw:~$

The WAN Out/In counters will go up the more they route traffic and the table 201 and 202 are filled with the routes for the different connections.

Exceptions

Of course, I wanted to make some exceptions to generally load balancing and failover. One of these examples was that my download server always used up the cable connection (with its 400Mbit) instead of bothering the ADSL line and that my own computer was to use the ADSL line as primary, only failing over to cable if the ADSL line goes down. The latter was important because the ADSL line is much better when it comes to voice calls, webinars, etc.

To facilitate a configuration where something used the ADSL connection as primary and the cable connection as secondary, I had to create a new load balancing group:

set load-balance group DUAL_ISP_ADSL_Pri interface eth0.98 route-test initial-delay 15
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.98 route-test interval 5
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.98 route-test type ping target 1.1.1.1
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.99 failover-only
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.99 route-test initial-delay 15
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.99 route-test interval 5
set load-balance group DUAL_ISP_ADSL_Pri interface eth0.99 route-test type ping target 1.1.1.1

You’ll only notice 2 differences: I removed the weights and added failover-only to eth0.99 (which is the cable connection). This gives the result that eth0.98 will always be used unless the ping test fails, which will make it failover to eth0.99.

To effect this new load balancing group for a specific client, I’ve used this snippet:

set firewall modify ISP_BALANCE rule 50 action modify
set firewall modify ISP_BALANCE rule 50 modify lb-group DUAL_ISP_ADSL_Pri
set firewall modify ISP_BALANCE rule 50 source group address-group Martijn-iMac-Macbook

This inserts a rule in the existing ISP_BALANCE ruleset where the load balancing is being called, takes a source address-group (in this case named Martijn-iMac-Macbook) and uses that to decide to redirect the traffic to the new load balancing group. The address group simply contains the IP addresses that my iMac and Macbook have.

Another variation on this is doing this on a destination basis:

set firewall modify ISP_BALANCE rule 52 action modify
set firewall modify ISP_BALANCE rule 52 destination group address-group LD-VPN-GW-EXT
set firewall modify ISP_BALANCE rule 52 modify lb-group DUAL_ISP_ADSL_Pri

Here I make sure that a VPN tunnel endpoint on the internet (IP included in the address group LD-VPN-GW-EXT) always goes via the ADSL connection and fails back to the cable connection if needed.

Conclusion

While this is a temporary situation for me (while I figure out a network overhaul and include VeloCloud for this functionality), the EdgeRouter really does a good job with load balancing between multiple links and the possibilities to make exceptions based on source or destination are awesome.

Share the wealth!

Categories: Networking

Tags: Networking, Ubiquiti, WAN

17 Comments

Dimitris Xalatsis
October 25, 2018 at 11:19

Very good article and to the point instructions! Wish the actual router’s manual was more like that!

Reply
Mario
December 29, 2018 at 13:07

Thanks for that article! Before this I though – that’s too much for my mind – I’m not smart enough – but with your help I’ve now configured a few exceptions to my multi WAN routing and use important clients only on the reliable (but slow) ADSL connection (so I’m in a similar situation as you – only in my case the speedy 2nd WAN is LTE – but the quality for sure aint better 🙂
thanks, greets from Austria, Mario

Reply
Rishat
January 7, 2019 at 11:20

Very useful article! Thanks for your such a good job!

Reply
Garner
August 20, 2019 at 20:41

This is very good information. Is it possible to do this using a public IP address on the internal network? Like an FTP server or an email server? That is something I’m trying to figure out how to do.

Reply

Martijn (Post author)
September 6, 2019 at 12:54

Sure, I’d suggest doing destination NAT for that. If you want to make it redundant, just put the DNAT on both ISP uplinks.

Reply

ajgamer
September 18, 2019 at 16:10

Hi! Thanks for the Article. It’s very informative. But I’m trying to achieve something which I’m unable to.

I use ER-X with Dual WAN (w/ Failover and Failback Scripts by BranoB) which facilitates that as soon as Main WAN comes back online, this script would stop using backup WAN and start using Main WAN.

Now, the thing is. I myself have Two PC Setup.

I want to use Main ISP on the PC #2 but I want to use Backup ISP on PC #1.

EdgeRouter-X handles all the DHCP. So there is no NAT in-between.

But, I don’t want to loose LAN between PC #1 and PC #2.

How can I do so?

Thanks.

Reply
John McCartan
May 4, 2020 at 14:47

Thanks much for the thorough explanation and the scripts.

My situation is very similar. I am not doing vlan on inbound connections – Cable will be on eth0 and ADSL on eth1. I am going to try to adapt your script to meet my needs. If you have any tips – much appreciated as I am just learning… is it possible to accomplish the same scenario using 2 ports (eth0, eth1)? I am assuming so but as I stated, just learning so want to confirm.

Another question I have that you might be able to help with:

I will have the two connections (separate) inbound. I have a Synology that I am considering attempting link aggregation with… that might be down the line. The other port will feed my double nat’d EERO network. Using it for all my main functionality (wireless, DHCP, etc).

1. Is this a good idea?
2. I am trying to figure out – how I get to 192.168.1.1 (ERX) from 192.168.4.x (EERO’s scope)?

Thanks!!!

Reply
Cognivator
November 9, 2020 at 04:43

This was a real treasure trove. I had virtually the same routing needs, even though my topology is slightly different. I look forward to trying these config tricks. Thanks for the post!

Reply
William Wegerson
August 13, 2021 at 22:54

My need was to route all requests to a specific IP to a specific interface (ISP), and before finding this page, I had posted to the Ubiquity forums trying to get answers. Though I got responses, they still left me high/dry.

Once I read your sentence “Another variation on this is doing this on a destination basis:” … that is what clicked. I was able to use the dashboard’s `Config Tree` and work backwards from what you did. The only thing missing on your article above is creating an address group:

`firewall/group/address-group` and give the name and ips for that group.

Thank you for this post.

Link to my forum post:

https://community.ui.com/questions/Create-A-Rule-Based-on-IP-Address-to-Route-To-Specific-Interface-on-The-Router/d46bb52d-8e6d-4a5f-971a-b7cc9c250ef2

Reply
sennai
February 18, 2022 at 10:31

Awesome!!!
This post helped me clearly understand and configure the concept of load balance / failover, source & destination PBR. Was able to achieve what I wanted. Used your post as a reference along with unifi help pages. My setup is for Starling – DSL LB-failover and DSL for devices that I do the MS teams and zoom calls. Did it all with the Edgerouter-x GUI, and just used CLI to check the watchdog & status.
Perfect!!!

Thank you very much.

Reply
Alan
July 23, 2022 at 14:54

Hi,

Are you able to say how you know what the default values for:

load-balance group G interface ethX route-test count success

You mention that it is three (pings) but how do you know that?

I have not manually set it, but if I enter configure mode, and execute:

show load-balance group G interface ethX

I get the response:

Configuration under specified path is empty

which makes sense, but does not tell me what the default value is.

Thanks in advance!

Alan.

Reply

Martijn (Post author)
July 23, 2022 at 15:01

It was in the documentation. But, if you’re already thinking about those values, just set it them to your desired value in your config.

Reply

Alan
July 23, 2022 at 15:50

Hi Martijn,

Thanks for your quick response. I have actually just copied your values for now (I can always adjust them later if I find it necessary).

My question was actually more generic – I may be missing something obvious, but I could not find any documentation for my ER-X that sets out all the default values for things I have not set, and was hoping you might know how to find that 🙂

If not, then no worries, and thank you for the article above – useful to have an actual implementation to cross reference against the generic example online.

Thanks,

Alan

Reply

Martijn (Post author)
July 23, 2022 at 15:58

Gotcha! You might be able to get those defaults from the CLI itself with the show commands (and going into details). I don’t run this setup anymore, so I can’t check.

Reply

Alan
July 25, 2022 at 12:14

That was the first thing I tried, but the ‘show’ commands return:

Configuration under specified path is empty

I’ll keep digging around the net – a list will be out there somewhere 🙂

Thanks again,

Alan.

Reply

West
October 27, 2022 at 19:27

Does anyone know how to write a rule to specifically treat certain MAC addresses to certain rules, load balancing, rather than doing it by group and IP address?

Thanks!

Reply

Martijn (Post author)
October 28, 2022 at 11:59

As far as I know, the EdgeRouter can’t do L2 conditional forwarding. I’d suggest binding the MAC addresses to DHCP static IPs and putting those IPs into a group.

Reply

Lostdomain

Tagcloud