FreeBSD Network Gateway on EdgeRouter Lite
EdgeRouter Lite is a great device to run at the edge of a home network. It becomes even better when it's running FreeBSD. This guide documents how to setup such a gateway. There are accompanying git repos to somewhat automate the process as well.
Why?
FreeBSD has all the required ingredients to make a secure and easy-to-administer firewall. Security projects like pfSense and OPNsense are based on it and have gained significant popularity among enthusiasts and professionals alike. Its open source nature has additional benefits over a proprietary operating system.
EdgeRouter Lite is a small and simple device. It has three network interfaces and a serial console. It has the ability to boot alternative operating systems making it perfect for enthusiasts. Be forewarned: you may void warranty if you proceed with the steps in this guide. I'll abbreviate EdgeRouter Lite as ERL in the rest of the post.
My brother sent me one ERL for my birthday in 2016. I thought it would be a great way to learn more about FreeBSD if I could use it as my home network's gateway/firewall. This blog post is the culmination of hours upon hours I spent on this project. I want to thank the community for providing very helpful documentation on setting up various pieces of software. And I want to especially thank my brother, Mansoor, for getting me started on this project.
Build Disk Image
Colin Percival has written a great blog post on the subject, titled FreeBSD on EdgeRouter Lite - no serial port required . In it he provides and describes a shell script to build a bootable image of FreeBSD to be run on ERL, available from GitHub in the freebsd-ERL-build repo.
I have built a Vagrant-based workflow to automate the building of the drive image. It's available on GitHub in the freebsd-edgerouterlite-ansible repo. It uses the build script Percival wrote.
Hardware Setup
Once you've built the disk image it's time to write it to a USB drive. There are two options: overwrite the original drive in the ERL or buy a new drive.
I tried the second option first and wrote to a new Sandrive Ultra Fit 32GB USB 3.0 Flash Drive (SDCZ43-032G-GAM46). It did not work and I later found on some blog that those drives do not work. I have not tried another third party drive since.
It is easy to take the drive out of the ERL enclosure. Take out three screws from the back to open the enclosure. Wiggle the drive up and down with a little more than gentle force and it'll slide out.
Before overwriting the original drive I created a backup image using dd. On
my macOS it showed up as /dev/disk2 upon insertion. Replace rdrive596870 with
the name of the drive on your machine.
$ diskutil list $ diskutil unmountdrive /dev/rdrive596870 $ sudo dd if=/dev/rdrive596870 of=original-erl.img
With the backup created it was easy to overwrite the drive with the newly built FreeBSD image.
$ sudo dd if=erl-freebsd.img of=/dev/rdrive596870 bs=1m && sync $ diskutil eject /dev/rdrive596870
Reinsert the drive, put back the enclosure, and put back the screws.
I found it helpful to use a USB-to-serial-console cable to monitor the boot process. It's what helped me figure out the third party drive I was using was not working.
Operating System Setup
Once the device is powered up and FreeBSD is booted successfully, Percival's script has configured it in this way: network interface (NIC) 1 (octe0) is the WAN connection, NIC 2 (octe1) is for LAN, and NIC 3 (octe2) is also LAN. The pf firewall allows SSH connections only from octe2. I haven't changed this original setting since I found the idea useful to use octe2 as an admin interface.
Connect your computer to octe2 and it should receive an IPv4 address from the DHCP server in the 192.168.2.0/24 range. octe2 has the IP address 192.168.2.1.
I have made it easier for myself and made it reusable for others by managing the config using Ansible. It's available on GitHub in the freebsd-on-erl-home-router repo.
The rest of this post will document my decisions in writing the roles and playbook.
Network Design
Comcast is my internet service provider (ISP) and thus the Ansible stuff is configured to work with it. Be especially careful with the IPv6 stuff since your provider may be different.
I wanted both IPv4 and IPv6 running in my network and to reach out to the internet.
To troubleshoot my internet service I use a USB-to-ethernet dongle and its MAC address is what's configured with Comcast on my cable modem. I use the same MAC address to override any gateway device that connects to the cable modem since I can swap devices without having to contact Comcast to update settings at their end.
For the wireless LAN (WLAN) or WiFi part of the network I'm using a simple access point (AP) that provides no routing, DHCP, or other services. It acts as a dumb AP providing WiFi services only.
Initial rc.conf
The initial config of rc.conf looks like this, mostly unchanged from Percival's scripts
hostname="erl" growfs_enable="YES" tmpfs="YES" tmpsize="50M" sendmail_enable="NONE" sshd_enable="YES" ntpd_enable="YES" ntpd_sync_on_start="YES" svscan_enable="YES"
Override octe0 MAC Address
The MAC address of an interface in FreeBSD 12.0-CURRENT is overriden with a file in /etc called start_if.*interfacename*, which in this case is start_if.octe0.
The contents of my /etc/start_if.octe0 are (with dummy values)
ifconfig octe0 ether OV:ER:RI:DE:00:00 # original ether OR:IG:IN:AL:MA:C0
IPv4 Network Config in rc.conf
ifconfig_octe0="SYNCDHCP" ifconfig_octe1="192.168.1.1 netmask 255.255.255.0" ifconfig_octe2="192.168.2.2 netmask 255.255.255.0" gateway_enable="YES"
I ran into an issue where WAN interface (octe0) would intermittently stop passing packets. I would log into the box and ping an IP on the internet and things would start working. Sometimes that didn't help and I had to bring octe0 down and up again to fix the issue. I found a solution on the FreeBSD forums: dhclient and intermittent connection.
I also tried SYNCDHCP instead of DHCP in ifconfig_octe0="SYNCDHCP"
above but it didn't help. I suspect it's a network driver issue but have no
confirmation.
IPv6 Network Config in rc.conf
rtsold_enable="NO" ipv6_gateway_enable="YES" ipv6_cpe_wanif="octe0" ifconfig_octe0_ipv6="inet6 -ifdisabled accept_rtadv" ifconfig_octe1_ipv6="inet6 -ifdisabled auto_linklocal" ifconfig_octe2_ipv6="inet6 -ifdisabled auto_linklocal" ipv6_privacy="YES" ip6addrctl_enable="YES" ip6addrctl_policy="AUTO"
Notice I set rtsold to disabled. A router node cannot run rtsold; it must only run on a non-router node.
dhcp6
Install dhcp6
$ sudo pkg install -y dhcp6
Configure dhcp6c with the following contents in /usr/local/etc/dhcp6c.conf. Remember, your ISP may require different settings. For example, Time Warner Road Runner has different settings.
interface octe0 {
send ia-pd 0;
send ia-pd 1;
send ia-na 1;
};
id-assoc na 1 {
};
id-assoc pd 0 {
prefix ::/64 infinity;
prefix-interface octe1 {
sla-id 1;
sla-len 0;
};
};
id-assoc pd 1 {
prefix ::/64 infinity;
prefix-interface octe2 {
sla-id 1;
sla-len 0;
};
};
Append these lines to /etc/rc.conf
dhcp6c_enable="YES" dhcp6c_interfaces="octe0"
Start dhcp6c
$ sudo service dhcp6c start
If unlike me you have a "smart" WiFI system, where instead of a dumb AP your AP does routing, firewalling, DHCP, etc., you may want to configure dhcp6s.conf as needed. I have not done it so can't really help on how to.
rtadvd
Configure rtadvd with an empty file in /etc/rtadvd.conf. The reason is that it is able to use the delegated prefixes assigned by dhcp6c to network interfaces to advertise downstream. When I tried using a non-empty config file, IPv6 didn't work as expected.
Append these lines to /etc/rc.conf
rtadvd_enable="YES" rtadvd_interfaces="octe1 octe2"
Start rtadvd
$ sudo service rtadvd start
Firewall with pf
I use pf firewall on my FreeBSD installs.
Create /etc/pf.conf with these settings, updated as needed. I have curated these rules from many sources and did not keep track of those sources. My apologies and if you can point out the source for the rule I'll reference them. I also make no claim that these rules have created a secure firewall.
# Interface layout
# octe0: WAN
# octe1: LAN 1
# octe1: LAN 2 (SSH access)
# pflog0: target interface for blocked packets
# Never touch loopback interfaces
set skip on lo
# Scrub all incoming traffic
scrub in
# NAT outgoing traffic
nat on octe0 inet from { octe1:network, octe2:network } to any -> (octe0:0)
# Default to blocking incoming traffic but allowing outgoing traffic
block all
pass out all
# DHCPv6 client: make IA_PD requests and receive responses to them
pass out quick on octe0 inet6 proto udp from octe0 to any port dhcpv6-server
pass in quick on octe0 inet6 proto udp to octe0 port dhcpv6-client
# Allow LAN to access the rest of the world
pass in on { octe1, octe2 } from any to any
block in on { octe1, octe2 } from any to self
# Allow LAN to ping us
pass in on { octe1, octe2 } inet proto icmp to any icmp-type { echoreq, echorep }
# Allow LAN to access DNS, DHCP, and NTP
pass quick on { octe0, octe1, octe2 } inet proto udp from { octe1:network, octe2:network } to any port { 53, 67, 123 }
pass quick on { octe0, octe1, octe2 } inet6 proto udp from { octe1, octe2 } to any port { 53, 67, 123 }
# Allow SSH to self
pass in on { octe2 } proto { tcp, udp } to self port 22
# Pass all outbound IPv6 traffic
pass out quick on octe0 inet6 from { octe0, octe1, octe2 } modulate state
# RFC 4890: Recommendations for Filtering ICMPv6 Messages in Firewalls
pass quick on { octe0, octe1, octe2 } inet6 proto icmp6 icmp6-type { 1, 2, 128, 129, 133, 134, 135, 136, 137 }
pass quick on { octe0, octe1, octe2 } inet6 proto icmp6 icmp6-type 3 code 0
pass quick on { octe0, octe1, octe2 } inet6 proto icmp6 icmp6-type 3 code 1
pass quick on { octe0, octe1, octe2 } inet6 proto icmp6 icmp6-type 4 code 0
pass quick on { octe0, octe1, octe2 } inet6 proto icmp6 icmp6-type 4 code 1
pass quick on { octe0, octe1, octe2 } inet6 proto icmp6 icmp6-type 4 code 2
Append these lines to /etc/rc.conf
pf_enable="YES" pf_rules="/etc/pf.conf" pf_flags="" pflog_enable="YES" pflog_logfile="/var/log/pflog" pflog_flags=""
Start pf
$ sudo service pf start
Unbound DNS Server
I decided on Unbound for DNS services in my LAN. Big thanks to Unbound DNS Tutorial for all the help it provided in configuring Unbound.
Create /usr/local/etc/unbound/unbound.conf with these contents. Modify as needed.
## Authoritative, validating, recursive caching DNS
## modified form of unbound.conf from https://calomel.org retrieved on 2016-10-24
#
server:
# log verbosity
verbosity: 1
# specify the interfaces to answer queries from by ip-address. The default
# is to listen to localhost (127.0.0.1 and ::1). specify 0.0.0.0 and ::0 to
# bind to all available interfaces. specify every interface[@port] on a new
# 'interface:' labeled line. The listen interfaces are not changed on
# reload, only on restart.
interface: 127.0.0.1
interface: ::1
interface: 192.168.1.1
interface: 192.168.2.1
# port to answer queries from
port: 53
# Enable IPv4, "yes" or "no".
do-ip4: yes
# Enable IPv6, "yes" or "no".
do-ip6: yes
# Enable UDP, "yes" or "no".
do-udp: yes
# Enable TCP, "yes" or "no". If TCP is not needed, Unbound is actually
# quicker to resolve as the functions related to TCP checks are not done.i
# NOTE: you may need tcp enabled to get the DNSSEC results from *.edu domains
# due to their size.
do-tcp: yes
# control which client ips are allowed to make (recursive) queries to this
# server. Specify classless netblocks with /size and action. By default
# everything is refused, except for localhost. Choose deny (drop message),
# refuse (polite error reply), allow (recursive ok), allow_snoop (recursive
# and nonrecursive ok)
access-control: 10.0.0.0/16 allow
access-control: 127.0.0.0/8 allow
access-control: 192.168.0.0/16 allow
# Read the root hints from this file. Default is nothing, using built in
# hints for the IN class. The file has the format of zone files, with root
# nameserver names and addresses only. The default may become outdated,
# when servers change, therefore it is good practice to use a root-hints
# file. get one from ftp://FTP.INTERNIC.NET/domain/named.cache
#root-hints: "/var/unbound/etc/root.hints"
# enable to not answer id.server and hostname.bind queries.
hide-identity: yes
# enable to not answer version.server and version.bind queries.
hide-version: yes
# Will trust glue only if it is within the servers authority.
# Harden against out of zone rrsets, to avoid spoofing attempts.
# Hardening queries multiple name servers for the same data to make
# spoofing significantly harder and does not mandate dnssec.
harden-glue: yes
# Require DNSSEC data for trust-anchored zones, if such data is absent, the
# zone becomes bogus. Harden against receiving dnssec-stripped data. If you
# turn it off, failing to validate dnskey data for a trustanchor will trigger
# insecure mode for that zone (like without a trustanchor). Default on,
# which insists on dnssec data for trust-anchored zones.
harden-dnssec-stripped: yes
# Use 0x20-encoded random bits in the query to foil spoof attempts.
# http://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00
# While upper and lower case letters are allowed in domain names, no significance
# is attached to the case. That is, two names with the same spelling but
# different case are to be treated as if identical. This means calomel.org is the
# same as CaLoMeL.Org which is the same as CALOMEL.ORG.
use-caps-for-id: yes
# the time to live (TTL) value lower bound, in seconds. Default 0.
# If more than an hour could easily give trouble due to stale data.
cache-min-ttl: 3600
# the time to live (TTL) value cap for RRsets and messages in the
# cache. Items are not cached for longer. In seconds.
cache-max-ttl: 86400
# perform prefetching of close to expired message cache entries. If a client
# requests the dns lookup and the TTL of the cached hostname is going to
# expire in less than 10% of its TTL, unbound will (1st) return the ip of the
# host to the client and (2nd) pre-fetch the dns request from the remote dns
# server. This method has been shown to increase the amount of cached hits by
# local clients by 10% on average.
prefetch: yes
# number of threads to create. 1 disables threading. This should equal the number
# of CPU cores in the machine. Our example machine has 4 CPU cores.
num-threads: 2
## Unbound Optimization and Speed Tweaks ###
# the number of slabs to use for cache and must be a power of 2 times the
# number of num-threads set above. more slabs reduce lock contention, but
# fragment memory usage.
#msg-cache-slabs: 4
#rrset-cache-slabs: 4
#infra-cache-slabs: 4
#key-cache-slabs: 4
# Increase the memory size of the cache. Use roughly twice as much rrset cache
# memory as you use msg cache memory. Due to malloc overhead, the total memory
# usage is likely to rise to double (or 2.5x) the total cache memory. The test
# box has 4gig of ram so 256meg for rrset allows a lot of room for cacheed objects.
#rrset-cache-size: 256m
#msg-cache-size: 128m
# buffer size for UDP port 53 incoming (SO_RCVBUF socket option). This sets
# the kernel buffer larger so that no messages are lost in spikes in the traffic.
#so-rcvbuf: 1m
## Unbound Optimization and Speed Tweaks ###
# Enforce privacy of these addresses. Strips them away from answers. It may
# cause DNSSEC validation to additionally mark it as bogus. Protects against
# 'DNS Rebinding' (uses browser as network proxy). Only 'private-domain' and
# 'local-data' names are allowed to have these private addresses. No default.
private-address: 192.168.0.0/16
private-address: 172.16.0.0/12
private-address: 10.0.0.0/8
# Allow the domain (and its subdomains) to contain private addresses.
# local-data statements are allowed to contain private addresses too.
private-domain: home.lan
# If nonzero, unwanted replies are not only reported in statistics, but also
# a running total is kept per thread. If it reaches the threshold, a warning
# is printed and a defensive action is taken, the cache is cleared to flush
# potential poison out of it. A suggested value is 10000000, the default is
# 0 (turned off). We think 10K is a good value.
unwanted-reply-threshold: 10000
# IMPORTANT FOR TESTING: If you are testing and setup NSD or BIND on
# localhost you will want to allow the resolver to send queries to localhost.
# Make sure to set do-not-query-localhost: yes . If yes, the above default
# do-not-query-address entries are present. if no, localhost can be queried
# (for testing and debugging).
do-not-query-localhost: no
# File with trusted keys, kept up to date using RFC5011 probes, initial file
# like trust-anchor-file, then it stores metadata. Use several entries, one
# per domain name, to track multiple zones. If you use forward-zone below to
# query the Google DNS servers you MUST comment out this option or all DNS
# queries will fail.
# auto-trust-anchor-file: "/var/unbound/etc/root.key"
# Should additional section of secure message also be kept clean of unsecure
# data. Useful to shield the users of this validator from potential bogus
# data in the additional section. All unsigned data in the additional section
# is removed from secure messages.
val-clean-additional: yes
# Blocking Ad Server domains. Google's AdSense, DoubleClick and Yahoo
# account for a 70 percent share of all advertising traffic. Block them.
# local-zone: "doubleclick.net" redirect
# local-data: "doubleclick.net A 127.0.0.1"
# local-zone: "googlesyndication.com" redirect
# local-data: "googlesyndication.com A 127.0.0.1"
# local-zone: "googleadservices.com" redirect
# local-data: "googleadservices.com A 127.0.0.1"
# local-zone: "google-analytics.com" redirect
# local-data: "google-analytics.com A 127.0.0.1"
# local-zone: "ads.youtube.com" redirect
# local-data: "ads.youtube.com A 127.0.0.1"
# local-zone: "adserver.yahoo.com" redirect
# local-data: "adserver.yahoo.com A 127.0.0.1"
# local-zone: "ask.com" redirect
# local-data: "ask.com A 127.0.0.1"
# Unbound will not load if you specify the same local-zone and local-data
# servers in the main configuration as well as in this "include:" file. We
# suggest commenting out any of the local-zone and local-data lines above if
# you suspect they could be included in the unbound_ad_servers servers file.
#include: "/etc/unbound/unbound_ad_servers"
# locally served zones can be configured for the machines on the LAN.
local-zone: "home.lan." static
local-data: device1.home.lan. IN A 192.168.1.55
local-data: device2.home.lan. IN A 192.168.1.97
local-data-ptr: 192.168.1.55 device1.home.lan
local-data-ptr: 192.168.1.97 device2.home.lan
# Use the following forward-zone to forward all queries to Google DNS,
# OpenDNS.com or your local ISP's dns servers for example. To test resolution
# speeds use "drill calomel.org @8.8.8.8" and look for the "Query time:" in
# milliseconds.
#
forward-zone:
name: "."
forward-addr: 50.116.40.226 # OpenDNS
forward-addr: 8.8.4.4 # Google
forward-addr: 2604:180:1:22a::8c53 # OpenDNS
Append these lines to /etc/rc.conf
unbound_enable="YES"
Start unbound
$ sudo service unbound start
dhcpd
Install isc-dhcp43-server
$ sudo pkg install -y isc-dhcp43-server
Create /usr/local/etc/dhcpd.conf with these contents. I like to have static assignments for IPv4 addresses based on MAC addresses. Modify as needed.
authoritative;
option domain-name "home.lan";
default-lease-time 43200;
max-lease-time 90000;
subnet 192.168.1.0 netmask 255.255.255.0 {
range 192.168.1.21 192.168.1.254;
option routers 192.168.1.1;
option broadcast-address 192.168.1.255;
option domain-name-servers 192.168.1.1, 50.116.40.226, 8.8.4.4;
}
subnet 192.168.2.0 netmask 255.255.255.0 {
range 192.168.2.2 192.168.2.254;
option routers 192.168.2.2;
option broadcast-address 192.168.2.255;
option domain-name-servers 192.168.1.2, 50.116.40.226, 8.8.4.4;
}
host device1 {
hardware ethernet 00:00:00:00:be:ef;
fixed-address 192.168.1.55;
}
host device2 {
hardware ethernet 00:00:00:01:be:ef;
fixed-address 192.168.1.97;
}
Append these lines to /etc/rc.conf
dhcpd_enable="YES" dhcpd_ifaces="octe1 octe2"
Start isc-dhcpd
$ sudo service isc-dhcpd start
Conclusion
I'm pretty happy with ERL and FreeBSD. There is great community documentation on how to configure all the pieces of software that make a FreeBSD-based home network gateway possible. I can tweak things as needed and upgrade when newer versions become available.
My plan on upgrading the base OS is to get a third party USB drive that works, write a newer FreeBSD image to it, and replace the drive in the ERL enclosure. This way I can keep a bunch of drives in rotation. Upgrades to newer builds or reverts to last known good version are as easy as swapping USB drives.
Configuration with Ansible means I don't have to manually do things again and again. As the configs change they'll be tracked in git so I get version control as well.
ERL is simply a great piece of network hardware. I'm tempted to try Ubiquiti's WiFi products instead of a mixture of DD-WRT and OpenWRT devices I have now. But that is for another day and perhaps another blog post.