Revision as of 01:24, 16 July 2020

All the setup involved for bespin, a Ubuntu 18.04 desktop server.

Inital Setup

Aptitude update

During installation, we allow setup to join the wifi network. On first boot, the network manager will be running and will be connected to the same wifi network. We will disable network manager eventually, but first get some software.

sudo apt-get update
sudo apt-get -y install vim gnome-tweak-tool net-tools

Set caps lock as a control key.

Allow sudo for user

Create wheel group:

sudo groupadd wheel

Add user to group:

sudo usermod -a -G wheel <your-username-here>

Allow wheel group users passwordless sudo, first use visudo to edit the sudoers file:

EDITOR=vi visudo

Now add this line to the end:

%wheel ALL=(ALL) NOPASSWD: ALL

Install ssh

Install ssh and server:

sudo apt-get install ssh

Start the server:

sudo service ssh start

Install trusted ssh key

If you want, set up a machine to securely SSH into the Ubuntu server.

From the machine you want to SSH FROM:

cat ~/.ssh/id_rsa.pub

Copy this text. Now in another terminal, ssh into the Ubuntu server. Paste the output of the above command into the file:

~/.ssh/authorized_keys

Now verify that SSHing into the Ubuntu server will not ask you for a password.

Configure WPA Supplicant

We want to configure wifi manually, and disable the network manager. This requires some preparation to manually join a wifi network with wpa supplicant.

First set your wpa supplicant to join a wifi network.

/etc/wpa_supplicant/wpa_supplicant.conf

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="yournetworkhere"
    proto=RSN
    key_mgmt=WPA-PSK
    pairwise=CCMP TKIP
    group=CCMP TKIP
    psk="yourpskhere"
}

Don't forget the WPA supplicant service (two steps down)

Name Network Interfaces

Ubuntu 18.04 does this annoying thing where the wifi interfaces are awful to type and impossible to remember because they contain the ENTIRE MAC ADDRESS OF THE DEVICE.

To fix this, rename the network devices. The following file will not exist on a fresh Ubuntu install, so create it with the following contents (one line per network device you want to rename):

/etc/udev/rules.d/70-persistent-net.rules

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="02:01:02:03:04:05", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="wlan0"
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="02:01:02:03:04:06", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="wlan1"

Configure Network Interfaces

Next add the 2 usb wifi devices to network interfaces file.

wlan0 will be joining an existing wifi network
wlan1 will be in manual mode so it can be used as an AP

The following lines should be APPENDED to any existing network interfaces file:

/etc/network/interfaces

allow-hotplug wlan0
iface wlan0 inet dhcp
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

allow-hotplug wlan1
iface wlan1 inet static
    address 192.168.10.1
    netmask 255.255.255.0
    gateway 192.168.10.1

WPA Supplicant Startup Service

Copy a wpa supplicant service template:

sudo cp /lib/systemd/system/wpa_supplicant.service /etc/systemd/system/wpa_supplicant.service

Edit the file

sudo vim /etc/systemd/system/wpa_supplicant.service

Change this line from this:

ExecStart=/sbin/wpa_supplicant -u -s -O /run/wpa_supplicant

to this:

ExecStart=/sbin/wpa_supplicant -u -s -c /etc/wpa_supplicant/wpa_supplicant.conf -i wlan0

Also, remove the following line if it is present:

Alias=dbus-fi.w1.wpa_supplicant1.service # DELETE ME!

Now enable this service to start on boot:

sudo systemctl enable wpa_supplicant.service

Dhclient on Startup

The dhclient command must be run on startup after the wifi is set up so that bespin will get an IP address.

Create an rc.local startup service:

/etc/systemd/system/rc-local.service

[Unit]
Description=/etc/rc.local
ConditionPathExists=/etc/rc.local

[Service]
Type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Now add the dhclient command to rc.local:

/etc/rc.local

#!/bin/bash
/sbin/dhclient
exit 0

Make it executable:

chmod 744 /etc/rc.local

Now enable the rc-local service:

sudo systemctl enable rc-local.service

Verify it works okay:

sudo systemctl start rc-local.service
sudo systemctl status rc-local.service

Requesting Static IP

If you want to request a static IP from the router, add this to the dhclient config file:

/etc/dhcp/dhclient.conf

interface "wlan0" {
     send dhcp-requested-address 192.168.0.122;
}

Disable Network Manager

Next step is to disable the network manager. It takes a lot of commands. Too many commands.

sudo systemctl stop NetworkManager.service
sudo systemctl disable NetworkManager.service

and three more services:

sudo systemctl stop NetworkManager-wait-online.service
sudo systemctl disable NetworkManager-wait-online.service

sudo systemctl stop NetworkManager-dispatcher.service
sudo systemctl disable NetworkManager-dispatcher.service

sudo systemctl stop network-manager.service
sudo systemctl disable network-manager.service

Don't uninstall it, though, because that will uninstall a bunch of other important gnome packages and you'll be left with a stupid broken ubuntu.

Now reboot

sudo reboot now

Run sudo service --status-all and verify network manager is not running.

Test Wifi

Test that everything is working as expected by running the ifconfig and iwconfig commands.

ifconfig should show an IP address for the wlan0 interface that the wpa_supplicant connects with
iwconfig should show the name of the wifi network that wlan0 is connected to (same one defined in wpa_supplicant.conf)

Troubleshooting

If you don't have an IPv4 address, troubleshoot with the following commands:

Check if you can reach the internet:

ping google.com

Check logs from dhcp service started by rc.local (this gets an IP address from the router and is the most likely culprit):

sudo service rc-local status

Check logs from wpa supplicant:

sudo service wpa_supplicant status

Gnome Setup

Now time to tweak the Gnome theme/appearance. You already have Tweak tool installed (see above apt-get install line).

Install Macbuntu Theme

Set it up to look like a mac by installing the macbuntu theme - see http://www.linuxandubuntu.com/home/macbuntu-transform-ubuntu-1604-to-mac-os-x

First activate the PPA from noobslab and pull in the new packages:

sudo add-apt-repository ppa:noobslab/macbuntu
sudo apt-get update

Now install the macbuntu icons and themes:

sudo apt-get -y install macbuntu-os-icons-v1804 macbuntu-os-ithemes-v1804

Pick Theme in Tweak

Now you can open the Tweak tool and pick the Macbuntu theme. You might have to log out and log in, or refresh Gnome (Alt + F2, then enter "r" as the command to run, to refresh Gnome).

Install Plank

Plank is a dock replacement. To install it:

sudo apt-get -y install plank

Noobslab also provides a Macbuntu theme for Plank, to make it look more like the Mac dock. You will need the PPA from noobslab set up (see above add-apt-repository step).

sudo apt-get -y install macbuntu-os-plank-theme-v1804

Start it at boot by creating a file in ~/.config/autostart.

First create the folder:

mkdir -p ~/.config/autostart

Now create a Plank startup script:

~/.config/autostart/plank.desktop

[Desktop Entry]
Type=Application
Exec=plank
Hidden=false
NoDisplay=false
X-GNOME-Autostart-enabled=true
Name=Plank

To open Plank preferences, right-click on the right or left side of the Plank dock and choose Preferences.

Disable Launcher

Run these commands to disable the ubuntu-provided launcher dock:

cd /usr/share/gnome-shell/extensions/
sudo mv ubuntu-dock@ubuntu.com{,.bak}

It will take effect when you log out or restart. If you can't wait to see it go, type Alt + F2, then enter the letter "r" in the input box. This will refresh gnome.

Install Albert

Albert is a spotlight/quicksilver replacement.

Install it like so:

sudo add-apt-repository ppa:noobslab/macbuntu
sudo apt-get update
sudo apt-get -y install albert

Start it at boot by creating the following file:

~/.config/autostart/albert.desktop

[Desktop Entry]
Type=Application
Exec=albert
Hidden=false
NoDisplay=false
X-GNOME-Autostart-enabled=true
Name=Albert

Albert Shortcut

We want to assign the shortcut Alt + Space to Albert, but this is currently occupied by a default system keyboard shortcut to open the window menu.

Let's remap that to Super + Space, then map Alt + Space to Albert.

Open Settings, then click Keyboard, and find the Windows section. Look for the Open Window Menu which is assigned the keyboard shortcut Alt + Space. Reassign it to Super + Space. By default this is assigned to something else, which can be disabled.

Now run albert from the command line: albert

Open the albert preferences. Assign Alt + Space to the Albert shortcut.

Disable Super Key Launcher Shortcut

Once you have Albert and Plank, you don't want the super key to open the Launcher because you don't need the launcher. Also, we disabled it.

Disable the Super key shortcut, where pressing Super by itself opens the launcher, by running this command:

gsettings set org.gnome.mutter overlay-key 'Alt_R'

This works immediately and will survive reboot.

Source: https://askubuntu.com/a/1037679

Test with Restart

Restart and test that the startup services start okay:

sudo reboot now

You should see:

The Plank dock on the bottom of the screen
No launcher on the left side
The super key should not open the launcher
(If you have set the Alt + Space shortcut for albert) Alt + Space should open albert

Disable Crash Reporting

Apport

Disable apport crash reporting:

/etc/default/apport

Change this:

enabled=1

to this:

enabled=0

Disable the service:

sudo systemctl disable apport
sudo systemctl stop apport

Whoopsie

Disable whoopsie crash reporting:

/etc/default/whoopsie

report_crashes=false

Disable the service:

sudo systemctl disable whoopsie
sudo systemctl stop whoopsie

Good riddance ya dumb piecea software

sudo rm -f /etc/init.d/whoopsie

Ansible Setup

Note that it is easiest and safest to perform these steps on a local network (e.g., a wifi router network) that has internet connectivity but that won't expose the Ubuntu server to the public internet.

Now use https://github.com/charlesreid1-com/charlesreid1-ansible to run the provision and base plays against the server. Note that this requires root access via ssh, so you have to edit your ssh config file to include this line:

/etc/ssh/sshd_config

PermitRootLogin yes

Now restart the ssh service:

sudo service ssh restart

and test that you can log in as root without a password:

ssh root@<ip-for-bespin>

ONLY ENABLE THIS WHILE YOU RUN ANSIBLE! DISABLE IT WHEN YOU ARE DONE!

Now run ansible from a different machine:

# run this from a different machine!
git clone git@github.com:charlesreid1-com/charlesreid1-ansible.git ansible
cd ansible

# provision (python3 install)
ANSIBLE_CONFIG="local.cfg" ansible-playbook --extra-vars "machine_name=bespin" provision.yml

# set up with all the things
ANSIBLE_CONFIG="local.cfg" ansible-playbook --extra-vars "machine_name=bespin" --vault-password-file=.vault_secret base.yml

Last, edit your ssh config to disable root login:

/etc/ssh/sshd_config

Remove this line!

PermitRootLogin yes # REMOVE ME

and replace with this one:

PermitRootLogin no

Now restart the ssh service:

sudo service ssh restart

Wifi Access Point Setup

In this section we set up Bespin to operate a wifi access point with hostapd. This involves the following:

Install and configure hostapd
Install and configure dnsmasq
Configure access point network interface
Start hostapd
Create hostapd startup service

Sysctl Changes

Enable bespin to forward packets, a necessary role of an access point router, in the sysctl settings:

/etc/sysctl.conf

Add this line in the .conf file, it should be there but commented out, add it in:

net.ipv4.ip_forward=1

To reload sysctl, run this command:

sysctl --system

Install hostapd

Start by installing hostapd:

sudo apt-get -y install hostapd

Configure hostapd

Create the hostapd config file:

/etc/hostapd/hostapd.conf

# Set this to wlan0 if you are using an ethernet cable
# to connect your Pi to the internet
interface=wlan1
driver=nl80211
hw_mode=g
channel=1
macaddr_acl=0
ignore_broadcast_ssid=0

# LAN10
ssid=LAN10
wpa_passphrase=cow-doctor-horse-building-5
auth_algs=1
wpa=3
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP
rsn_pairwise=CCMP

Now set the hostapd configuration file:

/etc/default/hostapd

DAEMON_CONF="/etc/hostapd/hostapd.conf"

At this point you should be able to start hostapd with this command:

sudo hostapd /etc/hostapd/hostapd.conf

You will be able to see the wifi network, and clients will be able to authenticate, but they won't be able to join it because they won't get IP addresses.

Side Note: Bridged Network with Shared DHCP

If you want your AP (wlan1) to share a network with the existing wifi connection (wlan0), and offload the DHCP functionality to the DHCP server of the existing wifi network, you can use the network bridge that hostapd creates (that wlan1 is already attached to) and attach wlan0 to the bridge. This allows clients sending packets to wlan1 to have those packets seen by wlan0.

Here are the steps to bring up the bridge:

sudo ifconfig wlan0 inet 0.0.0.0  # <-- is this necessary???
sudo brctl addif br0 wlan0
sudo ifconfig br0 up
sudo dhclient br0

Breakdown:

The bridge already exists, and is already connected to wlan1, so we add the wlan0 device to it
We bring up the bridge
We ask for an IP address for the bridge device

Here are the steps to bring down the bridge:

sudo ifconfig br0 down
sudo brctl delif br0 wlan1
sudo brctl delif br0 wlan0
sudo ifconfig br0 down
sudo dhclient wlan0

This is all you have to do, and you're done - no need for a DHCP server.

dnsmasq for DHCP and DNS

We are going to install dnsmasq to provide a DHCP and DNS service for the AP network.

Start by installing it with apt-get:

sudo apt-get -y install dnsmasq

This will try to start dnsmasq, but it will fail and print red text. This is fine - the system's built-in systemd-resolved is already listening on port 53 so we will need to disable this service. (But we need dnsmasq installed FIRST, because disabling systemd-resolved will cause DNS queries to fail, so the internet connection will break. We need dnsmasq ready to start and take over the DNS duties.)

sudo systemctl disable systemd-resolved
sudo systemctl stop systemd-resolved

Preserve the original dnsmasq config file, which has a lot of useful information:

sudo mv /etc/dnsmasq.conf{,.orig}

Now create the dnsmasq config file:

/etc/dnsmasq.conf

interface=wlan1
dhcp-range=192.168.10.100,192.168.10.150,255.255.255.0,24h

Here is a much better version:

# don't send external traffic that is missing a domain
domain-needed
# don't send external traffic that has bogus private ip
bogus-priv
# set the local domain
domain=mynet
local=/mynet/
# listen on these interfaces and only these interfaces
listen-address=127.0.0.1
listen-address=192.168.10.1
bind-interfaces
# define range of IP addresses to hand out
dhcp-range=192.168.10.100,192.168.10.150,255.255.255.0,24h
# define what to do if no name resolution
no-resolv
server=8.8.8.8
# send dnsmasq logs to a single place
log-facility=/var/log/dnsmasq.log

Now we are ready to start up dnsmasq:

sudo systemctl enable dnsmasq
sudo systemctl start dnsmasq

Set Preferred DNS Nameservers

(Following instructions from RaspberryPi/Hotspot)

Remove the existing file at /etc/resolv.conf (a symlink to a network manager thing). Create a new version of the file that specifies preferred nameservers:

nameserver 1.1.1.1
nameserver 8.8.8.8

NOTE: this file will be overwritten at boot by Network Manager. If your DNS is broken and dnsmasq does not seem to be able to find a preferred nameserver, circle back and double-check that network manager has been disabled.

Improved Logging

Dnsmasq configuration file has a log-facility option to control where logs go. Add this to the config file:

log-facility=/var/log/dnsmasq.log

Testing the AP

Before testing everything out, do a reboot and make sure that all services are running as expected (dnsmasq in paarticular) and that network interfaces are configured as expected.

To test the AP, we will run hostapd manually first. Checklist for testing if the access point works:

is dnsmasq service running? (Yes)
is hostapd service running? (No)
is static ip set for the wifi card creating the AP in /etc/network/interfaces? (Yes, 192.168.10.1)
does ifconfig show that IP? (Yes)
can cients see to wifi network? (Yes)
can clients connect to wifi network? (Yes)
does handshake process succeed? (Yes)
does client receive an IP? (Yes)
can client access internet? (No)

Now run hostapd manually:

sudo hostapd /etc/hostapd/hostapd.conf

To add more debug info:

sudo hostapd -d /etc/hostapd/hostapd.conf

To log it to a file,

sudo hostapd -d /etc/hostapd/hostapd.conf 2>&1 | tee hostapd.log

From a phone or another computer, connect to the wireless network and verify you get an IP in the lease range specified in the dnsmasq config file.

Starting the Service

To start hostap in the background as a service, and enable it to start on boot, run these commands:

sudo systemctl unmask hostapd
sudo systemctl enable hostapd
sudo systemctl start hostapd

Modifying Startup Service to Retry

The hostapd service is very fickle, failing the first time it is run but succeeding when run a few seconds later, so we can set up the hostapd service to retry 5 times, waiting 30 seconds each time.

To do that, edit the startup service file at /lib/systemd/system/hostapd.service

Add these two lines to the [Unit] section:

[Unit]
StartLimitInterval=200
StartLimitBurst=5

Add these two lines to the [Service] section:

[Service]
Restart=always
RestartSec=30

The final version of the hostapd startup service is given below:

/lib/systemd/system/hostapd.service

[Unit]
Description=Advanced IEEE 802.11 AP and IEEE 802.1X/WPA/WPA2/EAP Authenticator
After=network.target
StartLimitInterval=200
StartLimitBurst=5

[Service]
Type=forking
PIDFile=/run/hostapd.pid
EnvironmentFile=/etc/default/hostapd
ExecStart=/usr/sbin/hostapd -P /run/hostapd.pid -B $DAEMON_OPTS ${DAEMON_CONF}
Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target

VPN Tunnel

In this section we set up a VPN tunnel using OpenVPN and a pre-existing OpenVPN server. For this specific example we cover the use of Private Internet Access, a third-party VPN provider.

Install software

sudo apt-get -y install openvpn

Obtain OpenVPN profile

For PIA:

cd /tmp
wget https://www.privateinternetaccess.com/openvpn/openvpn.zip
unzip -d openvpn openvpn.zip
cd openvpn

This directory contains multiple OpenVPN profiles for each of PIA's server regions.

RUN THE FOLLOWING COMMANDS AS ROOT.

To use the openvpn profile, server cert, and client key:

PROFILE="Belgium"
cp ca.rsa.2048.crt /etc/openvpn/.
cp crl.rsa.2048.pem /etc/openvpn/.
cp ${PROFILE}.ovpn /etc/openvpn/.

Now add login credentials to a login file:

touch /etc/openvpn/login
echo "USERNAME" >> /etc/openvpn/login
echo "PASSWORD" >> /etc/openvpn/login

Modify the OpenVPN profile to use the credentials file we just made, and to point to the correct locations of the certificate and key.

Also modify the OpenVPN profile to name the tunnel device it will create. We will set the tunnel device name as tun1:

sed -i 's+^auth-user-pass+& /etc/openvpn/login+' /etc/openvpn/${PROFILE}.ovpn
sed -i 's+^ca ca.rsa.2048.crt+& /etc/openvpn/ca.rsa.2048.crt+' /etc/openvpn/${PROFILE}.ovpn
sed -i 's+^crl-verif crl.rsa.2048.pem+& /etc/openvpn/crl.rsa.2048.pem+' /etc/openvpn/${PROFILE}.ovpn
sed -i 's+dev tun$+dev tun1+' /etc/openvpn/${PROFILE}.ovpn

Testing OpenVPN Tunnel

Test the VPN connection by running curl -4 icanhazip.com before and after you bring the VPN up to verify your IP has changed:

openvpn --config /etc/openvpn/${PROFILE}.ovpn

Note that you may have a config file (.conf) instead, in which case, use the config file instead of the .ovpn file.

Use curl -6 icanhazip.com to test whether your IPv6 address has changed.

You can also run ifconfig and you should see a tun1 device with an IP address like 10.X.Y.Z. This is the VPN connection. To run curl using specifically that network interface, use the --interface flag with curl:

curl --interface 10.96.10.6 -4 icanhazip.com

The VPN should be capturing and routing ALL traffic through the VPN tunnel. Double check by running this command, which should fail to do anything:

curl --interface 192.168.0.199 -4 icanhazip.com

where 192.168.0.199 is the IP address of bespin on the wifi network.

Modifying OpenVPN Service

Before we can automatically start OpenVPN, we need to modify the startup service to look for .ovpn files instead of .conf files. Use this sed one-liner to do that:

sed -i 's|\.conf|.ovpn|' /lib/systemd/system/openvpn@.service

Enabling OpenVPN Service

To start the process automatically in the background, and on boot, enable the OpenVPN service. It will be called openvpn@${PROFILE}. So, to use the example of Belgium.ovpn,

sudo service openvpn@Belgium start

This will look for a file named Belgium.ovpn and start an OpenVPN client process connecting to that server.

Once again you can check your public IP with

curl -4 icanhazip.com

and look up the IP address location to determine whether your traffic is being routed properly through Belgium, or wherever you chose.

Connect AP to VPN Tunnel

The last step here is to provide an internet gateway for the AP, and to do it via the VPN tunnel.

The current network setup is as follows:

wlan0 on 192.168.0.0/24 - provides internet connection
wlan1 on 192.168.10.0/24 - access point network
tun1 on 10.96.10.0/24 - private IP for VPN tunnel

Bridged network vs routed network

There are (at least) two ways we can do this:

Bridged network - this uses a virtual network device called a bridge to allow two network interfaces to see traffic from each other. This can be thought of as connecting two network interfaces to a switch, done virtually on the local machine.
Routed network - this keeps the two network interfaces separate, and uses iptables to forward traffic from one device to another. This uses masquerading, which means that the server takes packets destined for external networks and proxies them, sending them out over the VPN tunnel, and returning the result to the client when it arrives.

Bridged networks are useful if you want clients on the AP to obtain IP addresses from the wifi router providing bespin with internet. In this scenario, clients would see the 192.168.0.0/24 network, just like bespin does, and would receive IP addresses on that network instead of 192.168.10.0/24.

Routed networks keep the networks associated with each network interface isolated. Packets only pass from one network interface to another if iptables has a rule to do that.

We will use a routed network for this setup.

Creating the routed network

Start by installing the netfilter-persistent tool, which will make it easy to save the iptables configuration:

sudo apt-get -y install netfilter-persistent

Creating the routed network requires packet forwarding to be enabled (see AP setup for instructions). Set up the routed network by adding iptables rules with the following script:

#!/bin/bash
set -e

ipt="sudo /sbin/iptables"

# start by flushing all rules and setting defaults
$ipt -F
# should we do this?
#$ipt -P INPUT DROP
#$ipt -P FORWARD DROP
$ipt -P INPUT ACCEPT
$ipt -P FORWARD ACCEPT
$ipt -P OUTPUT ACCEPT
$ipt -t nat -F
$ipt -t mangle -F
$ipt -F
$ipt -X

##################################
# PIA VPN Tunnels

# These are PIA tunnels that handle traffic from APs
PIA_AP_TUNNELS="tun1"
for TUN in TUNNELS; do
    # Accept all traffic coming in from tunnel
    $ipt -A INPUT -i ${TUN} -j ACCEPT
    # Masquaerade outgoing traffic leaving via the tunnel
    $ipt -t nat -A POSTROUTING -o ${TUN} -j MASQUERADE
done

##################################
# AP-PIA Tunneling

# Forward outgoing traffic for APs through tunnel
AP="wlan1"
TUN="tun1"
$ipt -A FORWARD -i ${AP} -o ${TUN} -j ACCEPT
$ipt -A FORWARD -i ${TUN} -o ${AP} -m state --state ESTABLISHED,RELATED -j ACCEPT

# Make rules persistent
sudo netfilter-persistent save

hat tip:

This stores the iptables configuration in /etc/iptables/

Testing Joining AP

Join the wifi network from another laptop or phone.

Verify that you receive an IP address and that your device can be pinged from bespin.

Check your IP address from the device (whatsmyip.org) to ensure it is coming from the PIA region specified.

Troubleshooting EAPOL Timeout

If your test computer/laptop/phone connects to the network and authenticates okay but the EAPOL handshake step keeps timing out, it's a problem with the DNS server (dnsmasq) not being set up properly. You can see the EAPOL handshake timeout messages when you run hostapd in debug mode (sudo hostapd -d /etc/hostapd/hostapd.conf) and try to join the wifi network with another computer.

Troubleshooting Joining Wifi

View the kernel IP routing table and review it to make sure things are wired up correctly:

netstat -rn

Check iptables rules with this command:

sudo iptables -S
# or
sudo iptables -L

To limit to input/output rules only, do this:

sudo iptables -L INPUT
sudo iptables -L OUTPUT

To check that traffic is flowing okay:

On bespin, run tcpdump -i tun1 (monitoring the openvpn tunnel) and tcpdump -i wlan1 (monitoring traffic on the AP) in side by side windows. Then join the AP from the phone or device and try to access the internet.

You should see packets related to the request that show up in both the tun1 and wlan1 traffic streams, which verifies that traffic is correctly being forwarded from the AP client through bespin and on to the final destination.

The packets should also be going in both directions - to and from the AP client. If they are only going one direction (from the client to the destination) and none are returning, double-check the iptables rules.

Help from here: [5]

More Troubleshooting

If you restart the networking service on bespin, like this

sudo service networking restart

then you'll lose your wifi connection. This is because the networking interface reverts back to looking for the old network interface name (the one with the entire mac address in the name), instead of the renamed version.

We specified the device name as part of udev, specifically the file /etc/udev/rules.d/70-persistent-net.rules

So we need to reload udev:

sudo udevadm control --reload-rules && udevadm trigger

Well crap, that doesn't work. If you reload the networking service, wifi breaks because wpa_supplicant reverts to a stupid network interface scheme, and apparently you're hosed until you restart.

PiHole

Run PiHole in a Docker container, and install it between dnsmasq and the VPN tunnel (so that DNS queries will be filtered by the PiHole).

Ubuntu/Bespin/PiHole

OpenVPN Server

Real simple: just set up an OpenVPN server.

Ubuntu/Bespin/OpenVPN Server

47 pages later: man that was was intense

Brushup of dnsmasq Config

Updated the dnsmasq config file to the one shown here.

Key changes:

Specifying the interfaces and listen addresses together (only need one or the other, but just to be sure)
Using the expanded notation for the server keyword, and using it to specify which DNS nameserver to use for traffic from different sources. Now different LANs can use different DNS.