FreeBSD Jails with VLAN HOWTO

26 Mar 2016    articles

Last Updated: May 4, 2016

This article discusses how to set up jails on a FreeBSD 11-CURRENT system utilizing VIMAGE (aka VNET) to provide a virtualized independent network stack with support for broad stroke VLAN tagging for each jail. Steps covered here do not employ the use of jail management frameworks, such as, iocage or ez-jail. This allows one to understand the underlying process and be able to fine tune it as needed.

A separate article will cover optimizations that can be utilized by virtue of being on ZFS.

Prerequisites

  • You have a machine installed with FreeBSD 11-CURRENT on ZFS.
  • We will be building world and kernel and using that as the base for the jails. Hence basic knowledge of FreeBSD system administration is assumed. If you’ve never compiled and installed a FreeBSD base system and kernel, this article may be hard to follow. Refer to the FreeBSD Handbook, especially chapter 8: ‘Configuring the FreeBSD Kernel’ and chapter 23: ‘Updating and Upgrading FreeBSD’.

Assumptions

  • Your host’s ethernet interface is em0.
  • Your IP network is 192.168.6.0/24 with gateway at 192.168.6.1.
  • The host will be assigned IP 192.168.6.66.
  • The guest jails will be assigned IPs in the range 192.168.6.100-254.
  • VLAN ID for all network interfaces will be 6.
  • Jails will be stored in ZFS datasets under /jail directory.

Configure the host system

Steps include:

  • Rebuild world and kernel
  • Update rc.conf
  • Enable jails related sysctl variables
  • Configure jails support

Rebuild world and kernel

On the host system, we need to re-compile the kernel to include EPAIR(4) and IF_BRIDGE(4) devices and the VIMAGE option to enable virtualized network stack capabilities. We also enable NULLFS so that we can mount ports and the src tree inside jails.

# Virtual networking for jail
options         VIMAGE
device          epair
device          if_bridge

# Enable nullFS to mount src and port directories
options         NULLFS

It’s not a bad idea to fetch the latest 11-CURRENT sources to get the latest fixes and rebuild and install world along with kernel at this point.

Update rc.conf

First, let’s set up the networking configuration. To enable VLANs, we need to create a cloned interface to set the VLAN parameters. We will also create a bridge that the paired networked interfaces for the jails will be trunked on.

ifconfig_em0="up"
cloned_interfaces="vlan0 bridge0"
ifconfig_vlan0="inet 192.168.6.66 netmask 255.255.255.0 vlan 6 vlandev em0"
defaultrouter="192.168.6.1"
ifconfig_bridge0="addm vlan0 up"

This setup will enable the host to access the network via vlan0 interface through the em0 physical adapater. Each paired interface will allow the guest to access the network by having its traffic flow to vlan0 via bridge0. As an added benefit, any traffic on the system, host and jails included, automatically get tagged with VLAN 6. See note at the end of the article for alternate configuration.

Next, we need to enable jails support:

jail_enable="YES"
jail_list=""

We also need to restrict some of the services on the host so that they don’t interfere with the jails:

dumpdev="AUTO"                  # Set to AUTO to enable crash dump, otherwise NO
rpcbind_enable="NO"             # Disable the RPC daemon
cron_flags="$cron_flags -J 15"  # Prevent jails running cron jobs at same time
syslogd_flags="-ss"             # Disable syslogd listening for incoming conns
sendmail_enable="NONE"          # Completely disable sendmail
clear_tmp_enable="YES"          # Clear /tmp at startup
inetd_flags="-wW -a $EXT_IP"    # Restrict inetd to not interfere with jails

To enable bridge to work with epair devices, a few settings need to be enabled via sysctl:

net.inet.ip.forwarding=1
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_member=0
security.bsd.unprivileged_read_msgbuf=0
security.jail.sysvipc_allowed=1

Configure jails support

FreeBSD implements jails v2 which requires configuration to be in /etc/jail.conf. To get started, we will create /etc/jail.conf with the default configuration applicable to all jails:

# Jail configuration - /etc/jail.conf

allow.raw_sockets = "0";
allow.set_hostname = "0";
allow.sysvipc = "1";
allow.mount.devfs;
  
host.hostname  = "${name}";
path  = "/jail/${name}";
mount.fstab  = "/etc/jails/fstab.${name}";
mount.devfs  = "1";
devfs_ruleset  = "4";
exec.consolelog  = "/var/log/jail_${name}_console.log";

Jail specific configuration will be added to /etc/jail.conf as we create each jail below.

Each jail will have it’s own fstab which will be placed under /etc/jails/, so create that directory:

sudo mkdir /etc/jails

Reboot your system or restart networking manually:

sudo service netif restart
sudo service routing restart

Running ifconfig should result in output similar to this (lo0 omitted for brevity):

em0:flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=42098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
    ether 68:05:ca:36:3c:26
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active

vlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 68:05:ca:36:3c:26
    inet 192.168.6.66 netmask 0xffffff00 broadcast 192.168.6.255 
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 6 parent interface: em0
    groups: vlan

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:0b:47:0d:33:00
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vlan0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000000

Here’s what the routing table looks like:

$ netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.6.1        UGS       vlan0
localhost          link#3             UH          lo0
192.168.6.0/24     link#4             U         vlan0
192.168.6.66       link#4             UHS         lo0

Create the jail

For demonstration purposes, we will create a jail called webserver whose intent is to run a web server. The installation and configuration of the web server is out of scope for this guide.

Create ZFS dataset for the jail

We will create a ZFS dataset with quota for 2GB (substitute your zpool name accordingly):

sudo zfs create zroot/webserver
sudo zfs set mountpoint=/jail/webserver zroot/webserver
sudo zfs set quota=2g zroot/webserver

Populate filesystem

We will use the pre-built system binaries to populate our jail as follows:

cd /usr/src
sudo make installworld DESTDIR=/jail/webserver
sudo make distribution DESTDIR=/jail/webserver

If you chose to use a release distribution for the jail instead of 11-CURRENT binaries, extract the content from an appropriate ISO image accordingly. You may also wish to build the binaries from sources other than 11-CURRENT.

Create the jail’s rc.conf

We set up the jail’s rc.conf similar to the host:

hostname="webserver"
sendmail_enable="NONE"          # Completely disable sendmail
inetd_flags="-wW -a 192.168.6.100"    # Restrict inetd to not interfere with jails
rpcbind_enable="NO"             # Disable the RPC daemon
cron_flags="$cron_flags -J 15"  # Prevent jails running cron jobs at same time
syslogd_flags="-ss"             # Disable syslogd listening for incoming conns

Create the jail’s fstab on host

Create a new file /etc/jails/fstab.webserver and set it’s contents to:

/usr/src              /jail/webserver/usr/src            nullfs      rw     0     0
/usr/ports            /jail/webserver/usr/ports          nullfs      rw     0     0

Create the jail specific configuration in jail.conf

Finally, we need to set up the jail specific configuration in jail.conf. Here, we are assigning 192.168.6.100 to the jail. We specify that the jail will be using its own virtual networking stack with the vnet command and set the vnet interface is set to epair100b. Since the traffic flows through the bridge to the vlan0 interface on the host, all packets are automatically tagged with our VLAN ID of 6. There is no need for additional VLAN configuration within the jail.

# VIMAGE jail with upnp-based dlna server with its own NIC address
webserver {
  $if  = "100";
  $ip_addr  = "192.168.6.${if}";   # Jail ip address
  $ip_route  = "192.168.6.1";  # Gateway or host's ip address
  $mask = "255.255.255.0";  # Netmask
  vnet;
  vnet.interface  = "epair${if}b";

  # Commands to run on host before jail is created
  exec.prestart  = "ifconfig epair${if} create up";
  exec.prestart  += "ifconfig epair${if}a up";
  exec.prestart  += "ifconfig bridge0 addm epair${if}a up";

  # Commands to run in jail after it is created
  exec.start  = "/sbin/ifconfig lo0 127.0.0.1 up";
  exec.start  += "/sbin/ifconfig epair${if}b up";
  exec.start  += "/sbin/ifconfig epair${if}b ${ip_addr} netmask ${mask} up";
  exec.start  += "/sbin/route add default ${ip_route}";
  exec.start  += "/bin/sh /etc/rc";

  exec.stop  = "/bin/sh /etc/rc.shutdown";
  exec.poststop  = "ifconfig bridge0 deletem epair${if}a";
  exec.poststop  += "ifconfig epair${if}a destroy";
  persist;
}

Notes:

  • Prior to the jail being created (see exec.start), we create the epair(4) interfaces and add the epair100a interface to bridge0 on the host. This is the virtualized network adapter for the jail.
  • Once the jail is up (see exec.start), we activate the lo0 and epair100b interfaces.

Start the jail

sudo service jail start webserver

If all went well, you should see the jail listed (not IP Address is not shown for VNET based jails):

$ jls
   JID  IP Address      Hostname                      Path
     1                  webserver                     /jail/webserver

On the host, ifconfig should report the newly created epair100a interface along with it being a member of bridge0:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>     metric 0 mtu 1500
    ether 02:0b:47:0d:33:00
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair100a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: vlan flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
        ifmaxaddr 0 port 1 priority 128 path cost 2000000

epair100a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8<VLAN_MTU>
    ether 02:ff:70:00:07:0a
    inet6 fe80::ff:70ff:fe00:70a%epair100a prefixlen 64 scopeid 0x7
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair

And ifconfig output inside the jail should report the corresponding epair100b interface:

epair100b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:c0:00:0a:0b
    inet6 fe80::ff:c0ff:fe00:a0b%epair101b prefixlen 64 tentative scopeid 0x2
    inet 192.168.6.100 netmask 0xffffff00 broadcast 192.168.6.255
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair

Alternate Configuration

I initially wrote this article detailing my attempt to set up a jail environment in a DMZ. In the original post, I had bridged the em0 interface (without vlan or ip configuration) with the jail epair interfaces via bridge0. I left the responsibility of tagging packets with the VLAN ID to vlan0 on the host and to a separate vlan interface in each jail. @helio on freenode pointed out that if a jail were to be compromised, one could modify the vlan interface within the jail potentially causing havoc. Luckily, all traffic from this host was being routed through smart switches with aggressive ingress and egress filtering so it was not a concern, however, can’t ignore best practices.

If you are not in a similar situation, you may want to provide the flexibility of each jail specifying which VLAN it belongs to. In that case, remove vlan0 and add em0 to the bridge. The host traffic will still flow through vlan0 but this makes the phsyical raw interface available to the jails which they can then tag as appropriate. VLAN setup within the jail is similar to that of the host.

  freebsd networking