OpenBSD on Ubiquiti USG
Ubiquiti Unifi Security Gateway (USG) is a great device to run at the edge of a home network. It becomes even better when it's running OpenBSD. This guide documents how to setup such a gateway. There are accompanying git repos to somewhat automate the process as well.
Why?
This is a follow up to OpenBSD Network Gateway on EdgeRouter Lite. Since the EdgeRouter Lite 3 (ERL) and USG are very similar devices, the install process is nearly identical as well.
Required Tools
The following two cables connect to each other so you can connect from USB on your Mac to the serial port on USG.
USB to Serial interface cable
Serial to RJ45 Console Adapter Cable for Cisco Routers
You also need:
screeninstalled on your MacPhillips head screwdriver size 0
Wired network with DHCP server to use during install
Download OpenBSD
At the time of writing the latest release of OpenBSD was 6.3 so that's what I downloaded.
$ curl -O http://ftp.openbsd.org/pub/OpenBSD/6.3/octeon/miniroot63.fs
Hardware Setup and First Boot
Once you've downloaded the file it's time to write it to a USB drive. There are two options: overwrite the original drive in the USG or buy a new drive.
I tried the second option and wrote to a new Samsung 32GB USB 3.0 Flash Drive Fit (MUF-32BB/AM).
It is easy to take the drive out of the USG enclosure. Take off the rubber feet and take out four screws from the back to open the enclosure. Wiggle the USB drive up and down with a little more than gentle force and it'll slide out.
Even though I didn't use the original drive that came in the USG, I still created a backup image using dd. On my macOS it showed up as /dev/disk2 upon insertion. Replace rdrive596870 with the name of the drive on your machine.
$ diskutil list $ diskutil unmountdrive /dev/rdrive596870 $ sudo dd if=/dev/rdrive596870 of=original-erl.img
Overwrite the replacement drive (Samsung in my case) with the downloaded OpenBSD mini root file.
$ sudo dd if=miniroot63.fs of=/dev/rdrive596870 bs=1m && sync $ diskutil eject /dev/rdrive596870
Reinsert the drive, put back the enclosure, and put back the screws.
DO NOT power on USG just yet.
Connect an ethernet cable to WLAN 1 that's connected to your network. We need a DHCP server to give an IP address to USG.
Connect to the serial port.
Check which device your Mac has identified for the serial connection. In my case it was /dev/tty.usbserial.
$ ls -ltr /dev/*usb* | grep tty
Start a screen session on port 115200.
$ screen /dev/tty.usbserial 115200
Power on USG and watch it boot on the screen session.
Install OpenBSD
Once the device is powered up and USG has booted successfully, you'll see this prompt.
Octeon ubnt_e120#
Load the miniroot.
Octeon ubnt_e120# fatload usb 0 $loadaddr bsd.rd reading bsd.rd .......................... ................. 8789365 bytes read
Boot the image.
Octeon ubnt_e120# bootoctlinux
After a lot of text scrolling across you'll be asked to make a choice from the following menu.
Welcome to the OpenBSD/octeon 6.3 installation program. (I)nstall, (U)pgrade, (A)utoinstall or (S)hell?
Follow each prompt and answer appropriately. A sample session of my install is below.
Welcome to the OpenBSD/octeon 6.3 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell? i
At any prompt except password prompts you can escape to a shell by
typing '!'. Default answers are shown in []'s and are selected by
pressing RETURN. You can exit this program at any time by pressing
Control-C, but this can leave your system in an inconsistent state.
Terminal type? [vt220]
System hostname? (short form, e.g. 'foo') usg
Available network interfaces are: cnmac0 cnmac1 cnmac2 vlan0.
Which network interface do you wish to configure? (or 'done') [cnmac0] cnmac0
IPv4 address for cnmac0? (or 'dhcp' or 'none') [dhcp] dhcp
cnmac0: no link ..... got link
DHCPDISCOVER on cnmac0 - interval 1
DHCPOFFER from 10.10.10.1 (REDACTED_FOR_PRIVACY)
DHCPREQUEST on cnmac0 to 255.255.255.255
DHCPACK from 10.10.10.1 (REDACTED_FOR_PRIVACY)
bound to 10.10.10.74 -- renewal in 21600 seconds.
IPv6 address for cnmac0? (or 'autoconf' or 'none') [none] autoconf
Available network interfaces are: cnmac0 cnmac1 cnmac2.
Which network interface do you wish to configure? (or 'done') [done] done
DNS domain name? (e.g. 'bar.com') [my.domain] my.example.com
Using DNS nameservers at 10.10.10.1 8.8.8.8
Password for root account? (will not echo)
Password for root account? (again)
Start sshd(8) by default? [yes]
Setup a user? (enter a lower-case loginname, or 'no') [no] ubnt
Full name for user ubnt? [ubnt]
Password for user ubnt? (will not echo)
Password for user ubnt? (again)
WARNING: root is targeted by password guessing attacks, pubkeys are safer.
Allow root ssh login? (yes, no, prohibit-password) [no] yes
What timezone are you in? ('?' for list) [UTC] America/Los_Angeles
Available disks are: sd0.
Which disk is the root disk? ('?' for details) [sd0]
Disk: sd0 geometry: 487/255/63 [7831552 Sectors]
Offset: 0 Signature: 0xAA55
Starting Ending LBA Info:
#: id C H S - C H S [ start: size ]
-------------------------------------------------------------------------------
*0: 0C 0 1 2 - 1 103 38 [ 64: 22528 ] FAT32L
1: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
2: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
3: 00 0 0 0 - 0 0 0 [ 0: 0 ] unused
Use (W)hole disk or (E)dit the MBR? [whole] W
Creating a FAT partition and an OpenBSD partition for rest of sd0...done.
/dev/rsd0i: 65372 sectors in 16343 FAT16 clusters (2048 bytes/cluster)
bps=512 spc=4 res=1 nft=2 rde=512 mid=0xf8 spf=64 spt=63 hds=255 hid=64 bsec=65536
The auto-allocated layout for sd0 is:
# size offset fstype [fsize bsize cpg]
a: 1024.0M 65600 4.2BSD 2048 16384 1 # /
b: 768.0M 2162752 swap
c: 30592.0M 0 unused
d: 1914.1M 3735616 4.2BSD 2048 16384 1 # /tmp
e: 2995.4M 7655616 4.2BSD 2048 16384 1 # /var
f: 2021.3M 13790112 4.2BSD 2048 16384 1 # /usr
g: 1024.0M 17929696 4.2BSD 2048 16384 1 # /usr/X11R6
h: 4387.9M 20026848 4.2BSD 2048 16384 1 # /usr/local
i: 32.0M 64 MSDOS
j: 1748.5M 29013216 4.2BSD 2048 16384 1 # /usr/src
k: 3969.0M 32594176 4.2BSD 2048 16384 1 # /usr/obj
l: 10707.8M 40722752 4.2BSD 2048 16384 1 # /home
Use (A)uto layout, (E)dit auto layout, or create (C)ustom layout? [a] a
Rounding size to bsize (32 sectors): 3920000
Rounding size to bsize (32 sectors): 6134496
Rounding size to bsize (32 sectors): 4139584
Rounding size to bsize (32 sectors): 8986368
Rounding size to bsize (32 sectors): 3580960
Rounding size to bsize (32 sectors): 8128576
/dev/rsd0a: 1024.0MB in 2097152 sectors of 512 bytes
6 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0l: 10707.8MB in 21929664 sectors of 512 bytes
53 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0d: 1914.1MB in 3920000 sectors of 512 bytes
10 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0f: 2021.3MB in 4139584 sectors of 512 bytes
10 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0g: 1024.0MB in 2097152 sectors of 512 bytes
6 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0h: 4387.9MB in 8986368 sectors of 512 bytes
22 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0k: 3969.0MB in 8128576 sectors of 512 bytes
20 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0j: 1748.5MB in 3580960 sectors of 512 bytes
9 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/rsd0e: 2995.4MB in 6134496 sectors of 512 bytes
15 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each
/dev/sd0a (33d72d1625866818.a) on /mnt type ffs (rw, asynchronous, local)
/dev/sd0l (33d72d1625866818.l) on /mnt/home type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0d (33d72d1625866818.d) on /mnt/tmp type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0f (33d72d1625866818.f) on /mnt/usr type ffs (rw, asynchronous, local, nodev)
/dev/sd0g (33d72d1625866818.g) on /mnt/usr/X11R6 type ffs (rw, asynchronous, local, nodev)
/dev/sd0h (33d72d1625866818.h) on /mnt/usr/local type ffs (rw, asynchronous, local, nodev)
/dev/sd0k (33d72d1625866818.k) on /mnt/usr/obj type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0j (33d72d1625866818.j) on /mnt/usr/src type ffs (rw, asynchronous, local, nodev, nosuid)
/dev/sd0e (33d72d1625866818.e) on /mnt/var type ffs (rw, asynchronous, local, nodev, nosuid)
Let's install the sets!
Location of sets? (disk http nfs or 'done') [http] http
HTTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none]
HTTP Server? (hostname, list#, 'done' or '?') [ftp.OpenBSD.org]
Server directory? [pub/OpenBSD/6.3/octeon]
Select sets by entering a set name, a file name pattern or 'all'. De-select
sets by prepending a '-', e.g.: '-game*'. Selected sets are labelled '[X]'.
[X] bsd [X] comp63.tgz [X] xbase63.tgz [X] xserv63.tgz
[X] bsd.rd [X] man63.tgz [X] xshare63.tgz
[X] base63.tgz [X] game63.tgz [X] xfont63.tgz
Set name(s)? (or 'abort' or 'done') [done] -game63.tgz
Set name(s)? (or 'abort' or 'done') [done] -game63.tgz
[X] bsd [X] comp63.tgz [X] xbase63.tgz [X] xserv63.tgz
[X] bsd.rd [X] man63.tgz [X] xshare63.tgz
[X] base63.tgz [ ] game63.tgz [X] xfont63.tgz
Set name(s)? (or 'abort' or 'done') [done]
Get/Verify SHA256.sig 100% |**************************| 1376 00:00
Signature Verified
Get/Verify bsd 100% |**************************| 5654 KB 00:04
Get/Verify bsd.rd 100% |**************************| 8583 KB 00:06
Get/Verify base63.tgz 100% |**************************| 86597 KB 00:29
Get/Verify comp63.tgz 100% |**************************| 49971 KB 00:17
Get/Verify man63.tgz 100% |**************************| 7031 KB 00:05
Get/Verify xbase63.tgz 100% |**************************| 16789 KB 00:06
Get/Verify xshare63.tgz 100% |**************************| 4421 KB 00:03
Get/Verify xfont63.tgz 100% |**************************| 39349 KB 00:13
Get/Verify xserv63.tgz 100% |**************************| 5036 KB 00:04
Installing bsd 100% |**************************| 5654 KB 00:00
Installing bsd.rd 100% |**************************| 8583 KB 00:02
Installing base63.tgz 100% |**************************| 86597 KB 01:13
Extracting etc.tgz 100% |**************************| 260 KB 00:01
Installing comp63.tgz 100% |**************************| 49971 KB 00:54
Installing man63.tgz 100% |**************************| 7031 KB 00:13
Installing xbase63.tgz 100% |**************************| 16789 KB 00:21
Extracting xetc.tgz 100% |**************************| 7049 00:00
Installing xshare63.tgz 100% |**************************| 4421 KB 00:09
Installing xfont63.tgz 100% |**************************| 39349 KB 00:50
Installing xserv63.tgz 100% |**************************| 5036 KB 00:05
Location of sets? (disk http nfs or 'done') [done]
Time appears wrong. Set to 'Wed Apr 4 22:14:34 PDT 2018'? [yes]
Saving configuration files...done.
Making all device nodes...done.
Relinking to create unique kernel...done.
CONGRATULATIONS! Your OpenBSD install has been successfully completed!
When you login to your new system the first time, please read your mail
using the 'mail' command.
INSTALL.octeon describes how to configure U-Boot to boot OpenBSD.
Exit to (S)hell, (H)alt or (R)eboot? [reboot] reboot
I configured root ssh access because I use Ansible later on and found it worked better when it logged in as root. Feel free to disable root ssh for better security.
I created a regular user ubnt to use for ssh access.
I did not install game63.tgz just because. I didn't feel it necessary to install it.
Swap the Bootloader
Once USG has rebooted you'll be at the Octeon ubnt_e120# prompt again because the bootloader is not at the expected location.
We'll swap the bootloader. But first, let's get a list of the existing environment variables. Save this output and keep in a safe place. You may need to undo risky steps ahead.
Octeon ubnt_e120# printenv bootcmd=fatload usb 0 $loadaddr vmlinux.64;bootoctlinux $loadaddr coremask=0x3 root=/dev/sda2 rootdelay=15 rw rootsqimg=squashfs.img rootsqwdir=w mtdparts=phys_mapped_flash:512k(boot0),512k(boot1),64k@1024k(eeprom) bootdelay=0 baudrate=115200 download_baudrate=115200 nuke_env=protect off $(env_addr) +$(env_size);erase $(env_addr) +$(env_size) autoload=n ethact=octeth0 loadaddr=0x9f00000 numcores=2 stdin=serial stdout=serial stderr=serial env_addr=0x1fbfe000 env_size=0x2000 flash_base_addr=0x1f400000 flash_size=0x800000 uboot_flash_addr=0x1f480000 uboot_flash_size=0x70000 flash_unused_addr=0x1f4f0000 flash_unused_size=0x710000 bootloader_flash_update=bootloaderupdate
Notice bootcmd above. That's what we'll swap.
Octeon ubnt_e120# setenv old_bootcmd ${bootcmd}
Octeon ubnt_e120# setenv bootcmd 'fatload usb 0 $loadaddr bsd;bootoctlinux rootdev=/dev/sd0'
Octeon ubnt_e120# setenv bootdelay 5
Octeon ubnt_e120# saveenv
Octeon ubnt_e120# reset
After USG reboots it should happily boot OpenBSD.
A sample session is below.
U-Boot 1.1.1 (UBNT Build ID: REDACTED) (Build time: Jun 9 2014 - 14:38:01)
BIST check passed.
UBNT_E120 r1:0, r2:16, f:8/135, serial #: REDACTED_FOR_PRIVACY
MPR 13-02044-16
Core clock: 500 MHz, DDR clock: 266 MHz (532 Mhz data rate)
DRAM: 512 MB
Clearing DRAM....... done
Flash: 8 MB
Net: octeth0, octeth1, octeth2
USB: (port 0) scanning bus for devices... 1 USB Devices found
scanning bus for storage devices...
Device 0: Vendor: Samsung Prod.: Flash Drive FIT Rev: 1100
Type: Removable Hard Disk
Capacity: 30592.0 MB = 29.8 GB (62652416 x 512) 0
reading bsd
.............................
5790035 bytes read
ELF file is 64 bit
Allocating memory for ELF segment: addr: 0xffffffff81000000 (adjusted to: 0x1000000), size 0x5796f0
Allocated memory for ELF segment: addr: 0xffffffff81000000, size 0x5796f0
Processing PHDR 0
Loading 4ebb88 bytes at ffffffff81000000
Clearing 8db68 bytes at ffffffff814ebb88
## Loading Linux kernel with entry point: 0xffffffff81000000 ...
Bootloader: Done loading app on coremask: 0x1
bootmem desc 0x24108 version 3.0
avail phys mem 0x0000000000100290 - 0x0000000000fffce0
avail phys mem 0x00000000015796f0 - 0x0000000008100000
avail phys mem 0x0000000008100010 - 0x000000000fffdc00
avail phys mem 0x0000000410000000 - 0x000000041ff00000
Total DRAM Size 0x0000000020000000
mem_layout[0] page 0x0000000000000041 -> 0x00000000000003FF
mem_layout[1] page 0x000000000000055F -> 0x0000000000002040
mem_layout[2] page 0x0000000000002041 -> 0x0000000000003FFF
mem_layout[3] page 0x0000000000104000 -> 0x0000000000107FC0
boot_desc->Initial setup done, switching console.
boot_desc->desc_ver:7
boot_desc->desc_size:400
boot_desc->stack_top:0
boot_desc->heap_start:0
boot_desc->heap_end:0
boot_desc->argc:2
boot_desc->flags:0x5
boot_desc->core_mask:0x1
boot_desc->dram_size:512
boot_desc->phy_mem_desc_addr:0
boot_desc->debugger_flag_addr:0xa44
boot_desc->eclock:500000000
boot_desc->boot_info_addr:0x1001f0
boot_info->ver_major:1
boot_info->ver_minor:2
boot_info->stack_top:0
boot_info->heap_start:0
boot_info->heap_end:0
boot_info->boot_desc_addr:0
boot_info->exception_base_addr:0x1000
boot_info->stack_size:0
boot_info->flags:0x5
boot_info->core_mask:0x1
boot_info->dram_size:512
boot_info->phys_mem_desc_addr:0x24108
boot_info->debugger_flags_addr:0
boot_info->eclock:500000000
boot_info->dclock:266000000
boot_info->board_type:20004
boot_info->board_rev_major:0
boot_info->board_rev_minor:16
boot_info->mac_addr_count:3
boot_info->cf_common_addr:0
boot_info->cf_attr_addr:0
boot_info->led_display_addr:0
boot_info->dfaclock:0
boot_info->config_flags:0x8
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2018 OpenBSD. All rights reserved. https://www.OpenBSD.org
OpenBSD 6.3 (GENERIC) #0: Sun Mar 25 19:48:22 UTC 2018
visa@octeon:/usr/src/sys/arch/octeon/compile/GENERIC
real mem = 536870912 (512MB)
avail mem = 523943936 (499MB)
mainbus0 at root: board 20004 rev 0.16
cpu0 at mainbus0: CN50xx CPU rev 0.1 500 MHz, Software FP emulation
cpu0: cache L1-I 32KB 4 way D 16KB 64 way, L2 128KB 8 way
clock0 at mainbus0: int 5
iobus0 at mainbus0
simplebus0 at iobus0: "soc"
octciu0 at simplebus0
cn30xxsmi0 at simplebus0
com0 at simplebus0: ns16550a, 64 byte fifo
com0: console
dwctwo0 at iobus0 base 0x1180068000000 irq 56
usb0 at dwctwo0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Octeon DWC2 root hub" rev 2.00/1.00 addr 1
octrng0 at iobus0 base 0x1400000000000 irq 0
cn30xxgmx0 at iobus0 base 0x1180008000000
cnmac0 at cn30xxgmx0: RGMII, address REDACTED_FOR_PRIVACY
atphy0 at cnmac0 phy 7: AR8035 10/100/1000 PHY, rev. 2
cnmac1 at cn30xxgmx0: RGMII, address REDACTED_FOR_PRIVACY
atphy1 at cnmac1 phy 6: AR8035 10/100/1000 PHY, rev. 2
cnmac2 at cn30xxgmx0: RGMII, address REDACTED_FOR_PRIVACY
atphy2 at cnmac2 phy 5: AR8035 10/100/1000 PHY, rev. 2
/dev/ksyms: Symbol table not valid.
umass0 at uhub0 port 1 configuration 1 interface 0 "Samsung Flash Drive FIT" rev 2.10/11.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus0 at umass0: 2 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: <Samsung, Flash Drive FIT, 1100> SCSI4 0/direct removable serial.REDACTED_FOR_PRIVACY
sd0: 30592MB, 512 bytes/sector, 62652416 sectors
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
boot device: sd0
root on sd0a (33d72d1625866818.a) swap on sd0b dump on sd0b
WARNING: No TOD clock, believing file system.
WARNING: CHECK AND RESET THE DATE!
Automatic boot in progress: starting file system checks.
/dev/sd0a (33d72d1625866818.a): file system is clean; not checking
/dev/sd0l (33d72d1625866818.l): file system is clean; not checking
/dev/sd0d (33d72d1625866818.d): file system is clean; not checking
/dev/sd0f (33d72d1625866818.f): file system is clean; not checking
/dev/sd0g (33d72d1625866818.g): file system is clean; not checking
/dev/sd0h (33d72d1625866818.h): file system is clean; not checking
/dev/sd0k (33d72d1625866818.k): file system is clean; not checking
/dev/sd0j (33d72d1625866818.j): file system is clean; not checking
/dev/sd0e (33d72d1625866818.e): file system is clean; not checking
setting tty flags
pf enabled
starting network
cnmac0: no link .... got link
cnmac0: bound to 10.10.10.190 from 10.10.10.1 (REDACTED_FOR_PRIVACY)
reordering libraries: done.
openssl: generating isakmpd/iked RSA keys... done.
ssh-keygen: generating new host keys: RSA DSA ECDSA ED25519
starting early daemons: syslogd pflogd ntpd.
starting RPC daemons:.
savecore: /bsd: kvm_read: version misread
checking quotas: done.
kvm_mkdb: can't open /dev/ksyms
clearing /tmp
kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sshd smtpd sndiod.
running rc.firsttime
Path to firmware: http://firmware.openbsd.org/firmware/6.3/
No devices found which need firmware files to be downloaded.
starting local daemons: cron.
Wed Apr 4 22:18:55 PDT 2018
OpenBSD/octeon (octeon1) (console)
login:
Configure as Gateway and Firewall
The device is configured exactly as OpenBSD Network Gateway on EdgeRouter Lite.
You can use Ansible roles I created for that post if you're so inclined. They're available on GitHub - openbsd-on-erl.
Conclusion
I'm pretty happy with USG and OpenBSD. Its setup was nearly identical to ERL 3. Using Ansible means identical configuration can be applied to both kinds of devices to get the same functionality.