OpenBSD Network Gateway on EdgeRouter Lite

(updated )

Source

EdgeRouter Lite is a great device to run at the edge of a home network. It becomes even better when it's running OpenBSD. This guide documents how to setup such a gateway. There are accompanying git repos to somewhat automate the process as well.

Why?

This is a follow up to FreeBSD Network Gateway on EdgeRouter Lite. I observed some issues, possibly related to network drivers, where the network would become unresponsive after some time. I first tried OpenWRT on the same hardware and didn't observe any unresponsiveness. I wanted to use a BSD instead of Linux so tried OpenBSD.

OpenBSD has all the required ingredients to make a secure and easy-to-administer firewall. I find the syntax of pf much better than iptables and would always prefer it if I had a choice.

Before you read this guide make sure you have read these resources which I used as well for this guide.

I initially wrote this post with OpenBSD 6.1. I have updated it to work with OpenBSD 6.4.

Required Tools

The following two cables connect to each other so you can connect from USB on your Mac to the serial port on ERL3.

  • USB to Serial interface cable
  • Serial to RJ45 Console Adapter Cable for Cisco Routers

You also need:

  • screen installed on your Mac
  • Phillips head screwdriver size 0
  • Wired network with DHCP server to use during install

Download OpenBSD

At the time of writing the latest release of OpenBSD was 6.4 so that's what I downloaded.

$ curl -O http://ftp.openbsd.org/pub/OpenBSD/6.4/octeon/miniroot64.fs

Hardware Setup and First Boot

Once you've downloaded the file it's time to write it to a USB drive. There are two options: overwrite the original drive in the ERL or buy a new drive.

I tried the second option first and wrote to a new Sandrive Ultra Fit 32GB USB 3.0 Flash Drive (SDCZ43-032G-GAM46). It did not work and I later found on some blog that those drives do not work. I bought another drive -- Samsung 32GB USB 3.0 Flash Drive Fit (MUF-32BB/AM) -- and that did work. I later bought another drive -- Samsung BAR Plus 32GB - 200MB/s USB 3.1 Flash Drive Titan Gray (MUF-32BE4/AM) -- which worked great as well.

It is easy to take the drive out of the ERL enclosure. Take out three screws from the back to open the enclosure. Wiggle the drive up and down with a little more than gentle force and it'll slide out.

Before overwriting the original drive I created a backup image using dd. On my macOS it showed up as /dev/disk2 upon insertion. Replace rdrive596870 with the name of the drive on your machine.

$ diskutil list
$ diskutil unmountDisk /dev/rdrive596870
$ sudo dd if=/dev/rdrive596870 of=original-erl.img

With the backup created it was easy to overwrite the drive with the downloaded file.

$ sudo dd if=miniroot64.fs of=/dev/rdrive596870 bs=1m && sync
$ diskutil eject /dev/rdrive596870

Reinsert the drive, put back the enclosure, and put back the screws.

DO NOT power on ERL3 just yet.

Connect an ethernet cable to eth0 that's connected to your network. We need a DHCP server to give an IP address to ERL3.

Connect to the serial port.

Check which device your Mac has identified for the serial connection. In my case it was /dev/tty.usbserial.

$ ls -ltr /dev/*usb* | grep tty

Start a screen session on port 115200.

$ screen /dev/tty.usbserial 115200

Power on ERL3 and watch it boot on the screen session.

Install OpenBSD

Once the device is powered up and ERL3 has booted successfully, you'll see this prompt.

Octeon ubnt_e100#

Load the miniroot.

Octeon ubnt_e100# fatload usb 0 $loadaddr bsd.rd

Boot the image.

Octeon ubnt_e100# bootoctlinux

After a lot of text scrolling across you'll be asked to make a choice from the following menu.

Welcome to the OpenBSD/octeon 6.4 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell?

Follow each prompt and answer appropriately. A sample session of my install is below.

Welcome to the OpenBSD/octeon 6.4 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell? i
At any prompt except password prompts you can escape to a shell by
typing '!'. Default answers are shown in []'s and are selected by
pressing RETURN.  You can exit this program at any time by pressing
Control-C, but this can leave your system in an inconsistent state.

Terminal type? [vt220]
System hostname? (short form, e.g. 'foo') erl

Available network interfaces are: cnmac0 cnmac1 cnmac2 vlan0.
Which network interface do you wish to configure? (or 'done') [cnmac0]
IPv4 address for cnmac0? (or 'dhcp' or 'none') [dhcp]
cnmac0: no link ..... got link
cnmac0: bound to 10.10.10.74 from 10.10.10.1 (REDACTED)
IPv6 address for cnmac0? (or 'autoconf' or 'none') [none] autoconf
Available network interfaces are: cnmac0 cnmac1 cnmac2 vlan0.
Which network interface do you wish to configure? (or 'done') [done]
Using DNS domainname my.example.com
Using DNS nameservers at 10.10.10.1

Password for root account? (will not echo)
Password for root account? (again)
Start sshd(8) by default? [yes]
Setup a user? (enter a lower-case loginname, or 'no') [no] ubnt
Full name for user ubnt? [ubnt]
Password for user ubnt? (will not echo)
Password for user ubnt? (again)
WARNING: root is targeted by password guessing attacks, pubkeys are safer.
Allow root ssh login? (yes, no, prohibit-password) [no] yes
What timezone are you in? ('?' for list) [UTC] UTC

Available disks are: sd0.
Which disk is the root disk? ('?' for details) [sd0]
Disk: sd0       geometry: 3900/255/63 [62656641 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
#: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
*0: 0C      0   1   2 -      1 103  38 [          64:       22528 ] FAT32L
1: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
2: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
3: 00      0   0   0 -      0   0   0 [           0:           0 ] unused
Use (W)hole disk or (E)dit the MBR? [whole]
Creating a FAT partition and an OpenBSD partition for rest of sd0...done.
/dev/rsd0i: 65372 sectors in 16343 FAT16 clusters (2048 bytes/cluster)
bps=512 spc=4 res=1 nft=2 rde=512 mid=0xf8 spf=64 spt=63 hds=255 hid=64 bsec=65536
The auto-allocated layout for sd0 is:
#                size           offset  fstype [fsize bsize   cpg]
a:          1024.0M            65600  4.2BSD   2048 16384     1 # /
b:           768.0M          2162752    swap
c:         30594.1M                0  unused
d:          1750.4M          3735616  4.2BSD   2048 16384     1 # /tmp
e:          2729.4M          7320416  4.2BSD   2048 16384     1 # /var
f:          1919.0M         12910208  4.2BSD   2048 16384     1 # /usr
g:           995.4M         16840320  4.2BSD   2048 16384     1 # /usr/X11R6
h:          4081.0M         18878880  4.2BSD   2048 16384     1 # /usr/local
i:            32.0M               64   MSDOS
j:          1707.6M         27236768  4.2BSD   2048 16384     1 # /usr/src
k:          5935.2M         30733920  4.2BSD   2048 16384     1 # /usr/obj
l:          9652.1M         42889184  4.2BSD   2048 16384     1 # /home
Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a]
/dev/rsd0a: 1024.0MB in 2097152 sectors of 512 bytes
6 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0l: 9652.1MB in 19767456 sectors of 512 bytes
48 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0d: 1750.4MB in 3584800 sectors of 512 bytes
9 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0f: 1919.0MB in 3930112 sectors of 512 bytes
10 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0g: 995.4MB in 2038560 sectors of 512 bytes
5 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0h: 4081.0MB in 8357888 sectors of 512 bytes
21 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0k: 5935.2MB in 12155264 sectors of 512 bytes
30 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0j: 1707.6MB in 3497152 sectors of 512 bytes
9 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0e: 2729.4MB in 5589792 sectors of 512 bytes
14 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/sd0a (a6dc8ad8dc03899c.a) on /mnt type ffs (rw, asynchronous, local)
/dev/sd0l (a6dc8ad8dc03899c.l) on /mnt/home type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0d (a6dc8ad8dc03899c.d) on /mnt/tmp type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0f (a6dc8ad8dc03899c.f) on /mnt/usr type ffs (rw, asynchronous, local, nodev)
/dev/sd0g (a6dc8ad8dc03899c.g) on /mnt/usr/X11R6 type ffs (rw, asynchronous, local, nodev)
/dev/sd0h (a6dc8ad8dc03899c.h) on /mnt/usr/local type ffs (rw, asynchronous, local, nodev)
/dev/sd0k (a6dc8ad8dc03899c.k) on /mnt/usr/obj type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0j (a6dc8ad8dc03899c.j) on /mnt/usr/src type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0e (a6dc8ad8dc03899c.e) on /mnt/var type ffs (rw, asynchronous, local, nodev, nosuid)

Let's install the sets!
Location of sets? (disk http nfs or 'done') [http]
HTTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none]
HTTP Server? (hostname, list#, 'done' or '?') [ftp.OpenBSD.org]
Server directory? [pub/OpenBSD/6.4/octeon]

Select sets by entering a set name, a file name pattern or 'all'. De-select
sets by prepending a '-', e.g.: '-game*'. Selected sets are labelled '[X]'.
    [X] bsd           [X] comp64.tgz    [X] xbase64.tgz   [X] xserv64.tgz
    [X] bsd.rd        [X] man64.tgz     [X] xshare64.tgz
    [X] base64.tgz    [X] game64.tgz    [X] xfont64.tgz
Set name(s)? (or 'abort' or 'done') [done] -game64.tgz
[X] bsd           [X] comp64.tgz    [X] xbase64.tgz   [X] xserv64.tgz
[X] bsd.rd        [X] man64.tgz     [X] xshare64.tgz
[X] base64.tgz    [ ] game64.tgz    [X] xfont64.tgz
Set name(s)? (or 'abort' or 'done') [done]
Get/Verify SHA256.sig   100% |**************************|  1365       00:00
Signature Verified
Get/Verify bsd          100% |**************************|  5714 KB    00:04
Get/Verify bsd.rd       100% |**************************|  8626 KB    00:06
Get/Verify base64.tgz   100% |**************************| 87690 KB    00:29
Get/Verify comp64.tgz   100% |**************************| 50446 KB    00:19
Get/Verify man64.tgz    100% |**************************|  7093 KB    00:04
Get/Verify xbase64.tgz  100% |**************************| 17055 KB    00:09
Get/Verify xshare64.tgz 100% |**************************|  4432 KB    00:03
Get/Verify xfont64.tgz  100% |**************************| 39348 KB    00:24
Get/Verify xserv64.tgz  100% |**************************|  5046 KB    00:04
Installing bsd          100% |**************************|  5714 KB    00:00
Installing bsd.rd       100% |**************************|  8626 KB    00:00
Installing base64.tgz   100% |**************************| 87690 KB    00:50
Extracting etc.tgz      100% |**************************|   259 KB    00:00
Installing comp64.tgz   100% |**************************| 50446 KB    00:38
Installing man64.tgz    100% |**************************|  7093 KB    00:07
Installing xbase64.tgz  100% |**************************| 17055 KB    00:12
Extracting xetc.tgz     100% |**************************|  6961       00:00
Installing xshare64.tgz 100% |**************************|  4432 KB    00:07
Installing xfont64.tgz  100% |**************************| 39348 KB    00:19
Installing xserv64.tgz  100% |**************************|  5046 KB    00:03
Location of sets? (disk http nfs or 'done') [done]
Time appears wrong.  Set to 'Sat Nov 17 20:47:46 UTC 2018'? [yes]
Saving configuration files...done.
Making all device nodes...done.
Relinking to create unique kernel... done.

CONGRATULATIONS! Your OpenBSD install has been successfully completed!

When you login to your new system the first time, please read your mail
using the 'mail' command.


INSTALL.octeon describes how to configure U-Boot to boot OpenBSD.
Exit to (S)hell, (H)alt or (R)eboot? [reboot]

I configured root SSH access because I use Ansible later on and found it worked better when it logged in as root. Feel free to disable root SSH for better security.

I created a regular user ubnt to use for SSH access.

I did not install game64.tgz just because. I didn't feel it necessary to install it.

Time to reboot.

Swap the Bootloader

Once ERL3 has rebooted you'll be at the Octeon ubnt_e100# prompt again because the bootloader is not at the expected location.

We'll swap the bootloader. But first, let's get a list of the existing environment variables. Save this output and keep in a safe place. You may need to undo risky steps ahead.

Octeon ubnt_e100# printenv
baudrate=115200
download_baudrate=115200
nuke_env=protect off $(env_addr) +$(env_size);erase $(env_addr) +$(env_size)
autoload=n
ethact=octeth0
bootdelay=5
bootcmd=fatload usb 0 $loadaddr vmlinux.64;bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom)
loadaddr=0x9f00000
numcores=2
stdin=serial
stdout=serial
stderr=serial
env_addr=0x1fbfe000
env_size=0x2000
flash_base_addr=0x1f800000
flash_size=0x400000
uboot_flash_addr=0x1f880000
uboot_flash_size=0x70000
flash_unused_addr=0x1f8f0000
flash_unused_size=0x310000
bootloader_flash_update=bootloaderupdate

Notice bootcmd above. That's what we'll swap.

Octeon ubnt_e100# setenv old_bootcmd "${bootcmd}"
Octeon ubnt_e100# setenv bootcmd 'fatload usb 0 $loadaddr bsd;bootoctlinux rootdev=/dev/sd0'
Octeon ubnt_e100# setenv bootdelay 5
Octeon ubnt_e100# saveenv
Octeon ubnt_e100# reset

After ERL3 reboots it should happily boot OpenBSD.

A sample session is below.

U-Boot 1.1.1 (UBNT Build ID: REDACTED) (Build time: May 27 2014 - 11:16:22)

BIST check passed.
UBNT_E100 r1:2, r2:18, f:4/71, serial #: REDACTED
MPR 13-00318-18
Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM:  512 MB
Clearing DRAM....... done
Flash:  4 MB
Net:   octeth0, octeth1, octeth2

USB:   (port 0) scanning bus for devices... 1 USB Devices found
    scanning bus for storage devices...
Device 0: Vendor: Samsung  Prod.: Flash Drive      Rev: 1100
            Type: Removable Hard Disk
            Capacity: 30594.0 MB = 29.8 GB (62656641 x 512)
 4  3  2  1  0
reading bsd
.............................

5851467 bytes read
ELF file is 64 bit
Allocating memory for ELF segment: addr: 0xffffffff81000000 (adjusted to: 0x1000000), size 0x588c30
Allocated memory for ELF segment: addr: 0xffffffff81000000, size 0x588c30
Processing PHDR 0
Loading 4f7ef8 bytes at ffffffff81000000
Clearing 90d38 bytes at ffffffff814f7ef8
## Loading Linux kernel with entry point: 0xffffffff81000000 ...
Bootloader: Done loading app on coremask: 0x1
bootmem desc 0x24108 version 3.0

avail phys mem 0x0000000000100290 - 0x0000000000fffbe0

avail phys mem 0x0000000001588c30 - 0x0000000008100000

avail phys mem 0x0000000008100010 - 0x000000000fffdc00

avail phys mem 0x0000000410000000 - 0x000000041ff00000

Total DRAM Size 0x0000000020000000

mem_layout[0] page 0x0000000000000041 -> 0x00000000000003FF

mem_layout[1] page 0x0000000000000563 -> 0x0000000000002040

mem_layout[2] page 0x0000000000002041 -> 0x0000000000003FFFInitial setup done, switching console.

boot_desc->desc_ver:7

boot_desc->desc_size:400

boot_desc->stack_top:0

boot_desc->heap_start:0

boot_desc->heap_end:0

boot_desc->argc:2

boot_desc->flags:0x5

boot_desc->core_mask:0x1

boot_desc->dram_size:512

boot_desc->phy_mem_desc_addr:0

boot_desc->debugger_flag_addr:0xa44

boot_desc->eclock:500000000

boot_desc->boot_info_addr:0x1001f0

boot_info->ver_major:1

boot_info->ver_minor:2

boot_info->stack_top:0

boot_info->heap_start:0

boot_info->heap_end:0

boot_info->boot_desc_addr:0

boot_info->exception_base_addr:0x1000

boot_info->stack_size:0

boot_info->flags:0x5

boot_info->core_mask:0x1

boot_info->dram_size:512

boot_info->phys_mem_desc_addr:0x24108

boot_info->debugger_flags_addr:0

boot_info->eclock:500000000

boot_info->dclock:266000000

boot_info->board_type:20002

boot_info->board_rev_major:2

boot_info->board_rev_minor:18

boot_info->mac_addr_count:3

boot_info->cf_common_addr:0

boot_info->cf_attr_addr:0

boot_info->led_display_addr:0

boot_info->dfaclock:0

boot_info->config_flags:0x8

Copyright (c) 1982, 1986, 1989, 1991, 1993

    The Regents of the University of California.  All rights reserved.

Copyright (c) 1995-2018 OpenBSD. All rights reserved.  https://www.OpenBSD.org


OpenBSD 6.4 (GENERIC) #0: Sat Oct 13 03:41:10 UTC 2018

    visa@octeon:/usr/src/sys/arch/octeon/compile/GENERIC

real mem = 536870912 (512MB)

avail mem = 523878400 (499MB)

mainbus0 at root: board 20002 rev 2.18

cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation

cpu0: cache L1-I 32KB 4 way D 16KB 64 way, L2 128KB 8 way

clock0 at mainbus0: int 5

octcrypto0 at mainbus0

iobus0 at mainbus0

simplebus0 at iobus0: "soc"

octciu0 at simplebus0

cn30xxsmi0 at simplebus0

com0 at simplebus0: ns16550a, 64 byte fifo

com0: console

dwctwo0 at iobus0 base 0x1180068000000 irq 56

usb0 at dwctwo0: USB revision 2.0

uhub0 at usb0 configuration 1 interface 0 "Octeon DWC2 root hub" rev 2.00/1.00 addr 1

octrng0 at iobus0 base 0x1400000000000 irq 0

cn30xxgmx0 at iobus0 base 0x1180008000000

cnmac0 at cn30xxgmx0: RGMII, address REDACTED

atphy0 at cnmac0 phy 7: AR8035 10/100/1000 PHY, rev. 2

cnmac1 at cn30xxgmx0: RGMII, address REDACTED

atphy1 at cnmac1 phy 6: AR8035 10/100/1000 PHY, rev. 2

cnmac2 at cn30xxgmx0: RGMII, address REDACTED

atphy2 at cnmac2 phy 5: AR8035 10/100/1000 PHY, rev. 2

/dev/ksyms: Symbol table not valid.

umass0 at uhub0 port 1 configuration 1 interface 0 "Samsung Flash Drive" rev 2.10/11.00 addr 2

umass0: using SCSI over Bulk-Only

scsibus0 at umass0: 2 targets, initiator 0

sd0 at scsibus0 targ 1 lun 0: <Samsung, Flash Drive, 1100> SCSI4 0/direct removable serial.REDACTED

sd0: 30594MB, 512 bytes/sector, 62656641 sectors

vscsi0 at root

scsibus1 at vscsi0: 256 targets

softraid0 at root

scsibus2 at softraid0: 256 targets

boot device: sd0

root on sd0a (a6dc8ad8dc03899c.a) swap on sd0b dump on sd0b

WARNING: No TOD clock, believing file system.

WARNING: CHECK AND RESET THE DATE!

Automatic boot in progress: starting file system checks.
/dev/sd0a (a6dc8ad8dc03899c.a): file system is clean; not checking
/dev/sd0l (a6dc8ad8dc03899c.l): file system is clean; not checking
/dev/sd0d (a6dc8ad8dc03899c.d): file system is clean; not checking
/dev/sd0f (a6dc8ad8dc03899c.f): file system is clean; not checking
/dev/sd0g (a6dc8ad8dc03899c.g): file system is clean; not checking
/dev/sd0h (a6dc8ad8dc03899c.h): file system is clean; not checking
/dev/sd0k (a6dc8ad8dc03899c.k): file system is clean; not checking
/dev/sd0j (a6dc8ad8dc03899c.j): file system is clean; not checking
/dev/sd0e (a6dc8ad8dc03899c.e): file system is clean; not checking
setting tty flags
pf enabled
starting network
cnmac0: no link.... got link
cnmac0: bound to 10.10.10.190 from 10.10.10.1 (REDACTED)
reordering libraries: done.
openssl: generating isakmpd/iked RSA keys... done.
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519
starting early daemons: syslogd pflogd ntpd.
starting RPC daemons:.
savecore: /bsd: kvm_read: version misread
checking quotas: done.
kvm_mkdb: can't open /dev/ksyms
clearing /tmp
kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sshd smtpd sndiod.
running rc.firsttime
Path to firmware: http://firmware.openbsd.org/firmware/6.4/
No devices found which need firmware files to be downloaded.
starting local daemons: cron.
Sat Nov 17 20:51:23 UTC 2018

OpenBSD/octeon (octeon1) (console)

login:

Apply Patches

Read Apply Patch to OpenBSD on how to apply security and other patches before continuing. Since patches gets released as needed, be sure to keep your system updated.

Network Design

Comcast is my internet service provider (ISP) and thus the Ansible stuff is configured to work with it. Be especially careful with the IPv6 stuff since your provider may be different.

I wanted both IPv4 and IPv6 running in my network and to reach out to the internet.

For the wireless LAN (WLAN) or WiFi part of the network I'm using a simple access point (AP) that provides no routing, DHCP, or other services. It acts as a dumb AP providing WiFi services only.

Networking

ERL3 has three network interfaces: eth0, eth1, and eth2. They are recognized by OpenBSD as cnmac0, cnmac1, and cnmac2 respectively.

I'll use cnmac0 (eth0) as the WAN port. I'll bridge cnmac1 (eth1) and cnmac2 (eth2) for LAN.

cnmac0

To troubleshoot my internet service I use a USB-to-ethernet dongle and its MAC address is what's configured with Comcast on my cable modem. I use the same MAC address to override any gateway device that connects to the cable modem since I can swap devices without having to contact Comcast to update settings at their end.

I override the MAC address of cnmac0 in my config but you don't have to.

I was confused for a while and got it wrong on getting an IPv6 non-temporary address (NA) via dhcp6c. Just like I would expect a client OS to ask a DHCPv6 server to hand it an IPv6 address, I was trying to do it on this router. I've explained what I was doing wrong in a section below.

What I needed to understand was that the WAN interface (cnmac0) only needed IPv6 autoconf to get a link-local address going. Then it was a matter of adding a default route. With this setup ERL could reach any globally routable address through the ISP-provided gateway.

# vi /etc/hostname.cnmac0
dhcp lladdr OV:ER:RI:DE:00:00
up
inet6 autoconf
!/usr/sbin/rcctl start dhcpcd

It took me a while to get this right but it's very important that you understand why I did it this way.

cnmac1

# vi /etc/hostname.cnmac1
up

cnmac2

# vi /etc/hostname.cnmac2
up

vether0

Create a virtual ethernet interface to be used in bridging cnmac1 and cnmac2.

# vi /etc/hostname.vether0
inet 192.168.1.1 255.255.255.0 192.168.1.255

bridge0

Create a bridge for LAN ports.

# vi /etc/hostname.bridge0
add vether0
add cnmac1
add cnmac2
blocknonip vether0
blocknonip cnmac1
blocknonip cnmac2
up

dhcpcd

Install dhcpcd.

# pkg_add -U dhcpcd

Add the following lines at the end of /etc/dhcpcd.conf. Leave other lines untouched.

# vi /etc/dhcpcd.conf
nohook resolv.conf
release
denyinterfaces cnmac1 cnmac2
noipv4
interface cnmac0
    ipv6rs
    ia_na 1
    ia_pd 2 vether0/0/64

Enable and start dhcpcd.

# rcctl enable dhcpcd
# rcctl start dhcpcd

dhcp6c

WARNING: DO NOT USE dhcp6c. USE dhcpcd INSTEAD. KEEPING THIS SECTION HERE FOR INFORMATIONAL PURPOSES ONLY.

Install dhcp6.

# pkg_add -U wide-dhcpv6

Add a line to /etc/rc.conf.local. Leave other lines, if present, untouched.

 # vi /etc/rc.conf.local
 dhcp6c_flags=cnmac0

Append a line (!/usr/sbin/rcctl restart dhcp6c) at the end of /etc/hostname.cnmac0. Leave other lines untouched.

That makes the entire contents of /etc/hostname.cnmac0 to be.

# vi /etc/hostname.cnmac0
dhcp lladdr OV:ER:RI:DE:00:00
up
inet6 autoconf
!/sbin/route add ::/0 -ifp cnmac0 fe80::
!/usr/sbin/rcctl restart dhcp6c

Make sure /etc/dhcp6c.conf looks like below for Comcast. You may need to alter some settings depending on your network and ISP.

interface cnmac0 {
    send ia-pd 0;
};

id-assoc pd 0 {
    prefix ::/64 infinity;
    prefix-interface vether0 {
        sla-id 1;
        sla-len 0;
    };
};

Create an init script.

# vi /etc/rc.d/dhcp6c
#!/bin/sh

daemon="/usr/local/sbin/dhcp6c"

. /etc/rc.d/rc.subr

rc_reload=NO

rc_cmd $1

Change permissions of the init script.

# chmod ugo+rx /etc/rc.d/dhcp6c

Enable and start dhcp6c.

# rcctl enable dhcp6c
# rcctl start dhcp6c

The "Wrong" Config

This is the "wrong" way to configure dhcp6c.conf for a router.

interface cnmac0 {
    send ia-pd 0;
    send ia-na 1;
};

id-assoc na 1 {
};

id-assoc pd 0 {
    prefix ::/64 infinity;
    prefix-interface vether0 {
        sla-id 1;
        sla-len 0;
    };
};

Correspondingly, /etc/hostname.cnmac0 was configured thusly.

# vi /etc/hostname.cnmac0
dhcp lladdr OV:ER:RI:DE:00:00
up
inet6 autoconf
!/usr/sbin/rcctl restart dhcp6c

Notice how I have send ia-na 1; and id-assoc na 1 { }; in the "wrong" config above? You need these when the device is on the edge of the network, for example a server or a desktop. This tells the DHCPv6 server to assign your WAN a non-temporary IPv6 address. You would use this address to access the device (client) from the WAN side. You don't necessarily have to do the same on a router, primarily because you can access it from the LAN (vether0) side.

I reached out to misc@ mailing list with a cry for help (WAN interface loses IPv6 NA address after pltime/vltime expire).

I was running into a problem where -- with the above config -- WAN was getting an IPv6 address (IA_NA) when dhcp6c was started. The remaining lease time was provided by pltime and vltime values in ifconfig cnmac0. When that time expired, dhcp6c would renew the address but it never got applied to the network interface (cnmac0). This resulted in my entire network losing IPv6 connectivity to the outside world. All devices would have an IPv6 address but they couldn't get to the Internet. When I restarted dhcp6c (rcctl restart dhcp6c), cnmac0 would get an IPv6 address again and things would start working. I ended up creating a cron entry to restart dhcp6c every hour.

As Stuart Henderson notes in his reply to my email,

There was a problem with vltime/pltime exported from the kernel being
incorrect which was fixed after 6.4

I appeared to have been running into this bug since I was using OpenBSD 6.4.

I'll try wide-dhcpv6 on OpenBSD 6.2 or later when they become available to see if my original problem is fixed. Otherwise it may be time to give dhcpcd a try instead, as suggested by Henderson. It seems to be a more actively maintained project and thus merits a deeper look.

Ideally, WAN should also be assigned a globally routable IPv6 address by the ISP so it can be reached from outside. It works for me when I run LEDE (OpenWRT fork) or RouterOS. It does work in this case, too, but obviously there are deficiencies and bugs. Once the kinks are ironed out I'd consider this to not be the "wrong" config. It's "wrong" as long as there are problems using it.

Unbound DNS Server

I decided on Unbound for DNS services in my LAN. Big thanks to Unbound DNS Tutorial for all the help it provided in configuring Unbound.

Create /var/unbound/etc/unbound.conf with these contents. Modify as needed.

# vi /var/unbound/etc/unbound.conf
## Authoritative, validating, recursive caching DNS
## modified form of unbound.conf from https://calomel.org retrieved on 2016-10-24
#
server:
  # log verbosity
    verbosity: 1

  # specify the interfaces to answer queries from by ip-address.  The default
  # is to listen to localhost (127.0.0.1 and ::1).  specify 0.0.0.0 and ::0 to
  # bind to all available interfaces.  specify every interface[@port] on a new
  # 'interface:' labeled line.  The listen interfaces are not changed on
  # reload, only on restart.
    interface: 127.0.0.1
    interface: ::1
    interface: 192.168.1.1
    interface: ::0

  # port to answer queries from
    port: 53

  # Enable IPv4, "yes" or "no".
    do-ip4: yes

  # Enable IPv6, "yes" or "no".
    do-ip6: yes

  # Enable UDP, "yes" or "no".
    do-udp: yes

  # Enable TCP, "yes" or "no". If TCP is not needed, Unbound is actually
  # quicker to resolve as the functions related to TCP checks are not done.i
  # NOTE: you may need tcp enabled to get the DNSSEC results from *.edu domains
  # due to their size.
    do-tcp: yes

  # control which client ips are allowed to make (recursive) queries to this
  # server. Specify classless netblocks with /size and action.  By default
  # everything is refused, except for localhost.  Choose deny (drop message),
  # refuse (polite error reply), allow (recursive ok), allow_snoop (recursive
  # and nonrecursive ok)
    access-control: 10.0.0.0/8 allow
    access-control: 127.0.0.0/8 allow
    access-control: 192.168.0.0/16 allow

  # Read  the  root  hints from this file. Default is nothing, using built in
  # hints for the IN class. The file has the format of  zone files,  with  root
  # nameserver  names  and  addresses  only. The default may become outdated,
  # when servers change,  therefore  it is good practice to use a root-hints
  # file.  get one from ftp://FTP.INTERNIC.NET/domain/named.cache
    #root-hints: "/var/unbound/etc/root.hints"

  # enable to not answer id.server and hostname.bind queries.
    hide-identity: yes

  # enable to not answer version.server and version.bind queries.
    hide-version: yes

  # Will trust glue only if it is within the servers authority.
  # Harden against out of zone rrsets, to avoid spoofing attempts.
  # Hardening queries multiple name servers for the same data to make
  # spoofing significantly harder and does not mandate dnssec.
    harden-glue: yes

  # Require DNSSEC data for trust-anchored zones, if such data is absent, the
  # zone becomes  bogus.  Harden against receiving dnssec-stripped data. If you
  # turn it off, failing to validate dnskey data for a trustanchor will trigger
  # insecure mode for that zone (like without a trustanchor).  Default on,
  # which insists on dnssec data for trust-anchored zones.
    harden-dnssec-stripped: yes

  # Use 0x20-encoded random bits in the query to foil spoof attempts.
  # http://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00
  # While upper and lower case letters are allowed in domain names, no significance
  # is attached to the case. That is, two names with the same spelling but
  # different case are to be treated as if identical. This means calomel.org is the
  # same as CaLoMeL.Org which is the same as CALOMEL.ORG.
    use-caps-for-id: yes

  # the time to live (TTL) value lower bound, in seconds. Default 0.
  # If more than an hour could easily give trouble due to stale data.
    cache-min-ttl: 3600

  # the time to live (TTL) value cap for RRsets and messages in the
  # cache. Items are not cached for longer. In seconds.
    cache-max-ttl: 86400

  # perform prefetching of close to expired message cache entries.  If a client
  # requests the dns lookup and the TTL of the cached hostname is going to
  # expire in less than 10% of its TTL, unbound will (1st) return the ip of the
  # host to the client and (2nd) pre-fetch the dns request from the remote dns
  # server. This method has been shown to increase the amount of cached hits by
  # local clients by 10% on average.
    prefetch: yes

  # number of threads to create. 1 disables threading. This should equal the number
  # of CPU cores in the machine. Our example machine has 4 CPU cores.
    num-threads: 2


  ## Unbound Optimization and Speed Tweaks ###

  # the number of slabs to use for cache and must be a power of 2 times the
  # number of num-threads set above. more slabs reduce lock contention, but
  # fragment memory usage.
    #msg-cache-slabs: 4
    #rrset-cache-slabs: 4
    #infra-cache-slabs: 4
    #key-cache-slabs: 4

  # Increase the memory size of the cache. Use roughly twice as much rrset cache
  # memory as you use msg cache memory. Due to malloc overhead, the total memory
  # usage is likely to rise to double (or 2.5x) the total cache memory. The test
  # box has 4gig of ram so 256meg for rrset allows a lot of room for cacheed objects.
    #rrset-cache-size: 256m
    #msg-cache-size: 128m

  # buffer size for UDP port 53 incoming (SO_RCVBUF socket option). This sets
  # the kernel buffer larger so that no messages are lost in spikes in the traffic.
    #so-rcvbuf: 1m

  ## Unbound Optimization and Speed Tweaks ###


  # Enforce privacy of these addresses. Strips them away from answers.  It may
  # cause DNSSEC validation to additionally mark it as bogus.  Protects against
  # 'DNS Rebinding' (uses browser as network proxy).  Only 'private-domain' and
  # 'local-data' names are allowed to have these private addresses. No default.
    private-address: 192.168.0.0/16
    private-address: 172.16.0.0/12
    private-address: 10.0.0.0/8

  # Allow the domain (and its subdomains) to contain private addresses.
  # local-data statements are allowed to contain private addresses too.
    private-domain: home.lan

  # If nonzero, unwanted replies are not only reported in statistics, but also
  # a running total is kept per thread. If it reaches the threshold, a warning
  # is printed and a defensive action is taken, the cache is cleared to flush
  # potential poison out of it.  A suggested value is 10000000, the default is
  # 0 (turned off). We think 10K is a good value.
    unwanted-reply-threshold: 10000

  # IMPORTANT FOR TESTING: If you are testing and setup NSD or BIND  on
  # localhost you will want to allow the resolver to send queries to localhost.
  # Make sure to set do-not-query-localhost: yes . If yes, the above default
  # do-not-query-address entries are present.  if no, localhost can be queried
  # (for testing and debugging).
    do-not-query-localhost: no

  # File with trusted keys, kept up to date using RFC5011 probes, initial file
  # like trust-anchor-file, then it stores metadata.  Use several entries, one
  # per domain name, to track multiple zones. If you use forward-zone below to
  # query the Google DNS servers you MUST comment out this option or all DNS
  # queries will fail.
  # auto-trust-anchor-file: "/var/unbound/etc/root.key"

  # Should additional section of secure message also be kept clean of unsecure
  # data. Useful to shield the users of this validator from potential bogus
  # data in the additional section. All unsigned data in the additional section
  # is removed from secure messages.
    val-clean-additional: yes

  # Blocking Ad Server domains. Google's AdSense, DoubleClick and Yahoo
  # account for a 70 percent share of all advertising traffic. Block them.
  # local-zone: "doubleclick.net" redirect
  # local-data: "doubleclick.net A 127.0.0.1"
  # local-zone: "googlesyndication.com" redirect
  # local-data: "googlesyndication.com A 127.0.0.1"
  # local-zone: "googleadservices.com" redirect
  # local-data: "googleadservices.com A 127.0.0.1"
  # local-zone: "google-analytics.com" redirect
  # local-data: "google-analytics.com A 127.0.0.1"
  # local-zone: "ads.youtube.com" redirect
  # local-data: "ads.youtube.com A 127.0.0.1"
  # local-zone: "adserver.yahoo.com" redirect
  # local-data: "adserver.yahoo.com A 127.0.0.1"
  # local-zone: "ask.com" redirect
  # local-data: "ask.com A 127.0.0.1"


  # Unbound will not load if you specify the same local-zone and local-data
  # servers in the main configuration as well as in this "include:" file. We
  # suggest commenting out any of the local-zone and local-data lines above if
  # you suspect they could be included in the unbound_ad_servers servers file.
  #include: "/etc/unbound/unbound_ad_servers"

  # locally served zones can be configured for the machines on the LAN.

    local-zone: "home.lan." static

    local-data: device1.home.lan.  IN A 192.168.1.55
    local-data: device2.home.lan.  IN A 192.168.1.97

    local-data-ptr: 192.168.1.55 device1.home.lan
    local-data-ptr: 192.168.1.97 device2.home.lan

  # Use the following forward-zone to forward all queries to Google DNS,
  # OpenDNS.com or your local ISP's dns servers for example. To test resolution
  # speeds use "drill calomel.org @8.8.8.8" and look for the "Query time:" in
  # milliseconds.
  #
   forward-zone:
      name: "."
      forward-addr: 50.116.40.226        # OpenDNS
      forward-addr: 8.8.4.4        # Google
      forward-addr: 2604:180:1:22a::8c53        # OpenDNS

Append these lines to /etc/rc.conf.local. Leave other lines untouched.

# vi /etc/rc.conf.local
unbound_flags=""

Edit /etc/dhclient.conf so it doesn't overwrite the local nameserver. Leave other lines untouched.

# vi /etc/dhclient.conf
ignore domain-name-servers;

Enable and start unbound.

# rcctl enable unbound
# rcctl start unbound

DHCP

Append these lines to /etc/rc.conf.local. Leave other lines untouched.

# vi /etc/rc.conf.local
dhcpd_flags="vether0"

Create /etc/dhcpd.conf with these settings. I like to have static assignments for IPv4 addresses based on MAC addresses. Modify as needed.

authoritative;
option domain-name "my.example.com";
option routers 192.168.1.1;
option broadcast-address 192.168.1.255;
option domain-name-servers 192.168.1.1;
default-lease-time 43200;
max-lease-time 90000;

subnet 192.168.1.0 netmask 255.255.255.0 {
    range 192.168.1.21 192.168.1.200;
}

host device1 {
    hardware ethernet 00:00:00:00:be:ef;
    fixed-address 192.168.1.238;
}

host device2 {
    hardware ethernet 00:00:00:01:be:ef;
    fixed-address 192.168.1.244;
}

Notice that the range for dynamic assignment does not include the IPs provided in static assignment. This is on purpose since otherwise dhcpd complains that the same address has been assigned twice.

Enable and start dhcpd.

# rcctl enable dhcpd
# rcctl start dhcpd

Routing

Enable IPv4 and IPv6 forwarding.

Append these lines to /etc/sysctl.conf. Leave other lines untouched.

# vi /etc/sysctl.conf
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

Enable these settings.

# xargs sysctl < /etc/sysctl.conf

Append these lines to /etc/resolv.conf.tail. Leave other lines untouched. Modify as needed.

# vi /etc/resolv.conf.tail
search my.example.com
nameserver 127.0.0.1
nameserver 8.8.8.8
lookup file bind

rad

OpenBSD 6.4 introduced rad as a replacement for KAME rtadvd (radvd). For OpenBSD 6.3, for example, read the section in this post about rtadvd.

Configure rad by editing /etc/rad.conf. In this instance, we only send out router advertisements over the vether0 interface.

# vi /etc/rad.conf
interface vether0

Append these lines to /etc/rc.conf.local. Leave other lines untouched.

# vi /etc/rc.conf.local
rad_flags=

Enable and start rad.

# rcctl enable rad
# rcctl start rad

rtadvd (radvd)

WARNING: OPENBSD 6.4 REPLACED KAME rtadvd (radvd) WITH ANOTHER DAEMON, rad, THAT THE OPENBSD PROJECT WROTE.

Configure rtadvd with an empty file in /etc/rtadvd.conf. The reason is that it is able to use the delegated prefixes assigned by dhcp6c to network interfaces to advertise downstream. When I tried using a non-empty config file, IPv6 didn't work as expected.

# truncate -s 0 /etc/rtadvd.conf

Append these lines to /etc/rc.conf.local. Leave other lines untouched.

# vi /etc/rc.conf.local
rtadvd_flags="vether0"

Enable and start rtadvd.

# rcctl enable rtadvd
# rcctl start rtadvd

Firewall with pf

Replace /etc/pf.conf with these settings, updated as needed. I have curated these rules from many sources, primarily The modern OpenBSD home router. I also make no claim that these rules have created a secure firewall.

# vi /etc/pf.conf
### ~~~ Interface layout ~~~ ###

# cnmac0: 802.3ab (ethernet) to cable modem
# cnmac1: 802.3ab (ethernet) to internal switch
# cnmac2: 802.3ab (ethernet) to internal switch
# vether0: persists address 192.168.1.0/255.255.255.0
# bridge0: Ethernet bridge over cnmac1 and cnmac2
# pflog0: target interface for blocked packets

### ~~~ Constants and variables ~~~ ###

# All addresses associated with this host
self = "{ (egress), (vether0) }"

self_lan = "{ (vether0) }"

# RFC 6890: Special-Purpose IP Address Registries:
# https://www.iana.org/assignments/iana-ipv4-special-registry/
# https://www.iana.org/assignments/iana-ipv6-special-registry/

# Included below are all address blocks with either Forwardable = False,
# Global = False, or both, but excluding 2001::/23 because it is often
# superseded by more specific allocations, as of 2015-08-05.

table <martians> const { \
    0.0.0.0/8, \
    10.0.0.0/8, \
    100.64.0.0/10, \
    127.0.0.0/8, \
    169.254.0.0/16, \
    172.16.0.0/12, \
    192.0.0.0/24, \
    192.0.2.0/24, \
    192.168.0.0/16, \
    198.18.0.0/15, \
    198.51.100.0/24, \
    203.0.113.0/24, \
    240.0.0.0/4, \
    255.255.255.255/32, \
    ::1/128, \
    ::/128, \
    ::ffff:0:0/96, \
    100::/64, \
    2001::/32, \
    2001:2::/48, \
    2001:db8::/32, \
    fc00::/7 \
}

### ~~~ Default rules ~~~ ###

# Never touch loopback interfaces
set skip on lo

# Normalise packets, especially IPv4 DF and Identification
match in all scrub (no-df random-id)

# Limit the MSS on PPPoE to 1440 octets
# match on pppoe0 scrub (max-mss 1440)

# Block all packets by default, logging them to pflog0
block log

### ~~~ Link-scoped services ~~~ ###

# DHCPv6 client: make IA_PD requests and receive responses to them
pass out quick on egress inet6 proto udp from (egress) to ff02::1:2 port dhcpv6-server
pass in quick on egress inet6 proto udp to (egress) port dhcpv6-client

### ~~~ Bulk pass rules ~~~ ###

# Pass all traffic on internal interfaces
# vether0 is necessary here, but bridge0 is not
pass quick on { vether0 cnmac1 cnmac2 }

# Pass all outbound IPv6 traffic
pass out quick on egress inet6 from { egress, (vether0:network) } modulate state

# Pass all outbound IPv4 traffic from this host
pass out quick on egress inet from (egress) modulate state

# NAT all outbound IPv4 traffic from the rest of our network
pass out quick on egress inet from (vether0:network) nat-to (egress) modulate state

### ~~~ Block undesirable traffic ~~~ ###

# These rules must not precede the DHCPv6 client or NAT rules above
block log quick on egress from { no-route, urpf-failed, <martians> }
block log quick on egress to { no-route, <martians> }

### ~~~ Pass some ICMP and ICMPv6 traffic ~~~ ####

# Pass all inbound ICMP echo requests
pass quick on { egress vether0 } inet proto icmp icmp-type { echoreq, echorep }
pass quick on { egress vether0 } inet6 proto icmp6 icmp6-type { echoreq, echorep }

# RFC 4890: Recommendations for Filtering ICMPv6 Messages in Firewalls
pass log quick on { egress vether0 } inet6 proto icmp6

pass log quick on egress inet6 proto udp from (egress) to fe80::d62c:44ff:fe
pass log quick on egress inet6 proto udp from fe80::d62c:44ff:fe to (egress)

### ~~~ Open services on this router ~~~ ###

# OpenSSH server
pass in on egress proto { tcp, udp } to $self_lan port ssh

# Allow LAN to access DNS, DHCP, and NTP
pass quick on egress inet proto udp from (vether0:network) to any port { 53, 67, 123 }
pass quick on egress inet proto udp from $self to any port { 53, 67, 123 }
pass quick on egress inet6 proto udp from (vether0:network) to any port { 53, 67, 123, 546 }
pass quick on egress inet6 proto udp from $self to any port { 53, 67, 123, 546 }

Reload rules.

# pfctl -f /etc/pf.conf

Ansible

You can use Ansible roles I created for this post if you're so inclined. They're available on GitHub - openbsd-on-erl.

Conclusion

I'm pretty happy with ERL and OpenBSD. There is great community documentation on how to configure all the pieces of software that make a OpenBSD-based home network gateway possible. I can tweak things as needed and upgrade when newer versions become available.

My plan on upgrading the base OS is to get a third party USB drive that works, write a newer OpenBSD image to it, and replace the drive in the ERL enclosure. This way I can keep a bunch of drives in rotation. Upgrades to newer builds or reverts to last known good version are as easy as swapping USB drives.

Configuration with Ansible means I don't have to manually do things again and again. As the configs change they'll be tracked in git so I get version control as well.