Network Virtualization for FreeBSD Jails
Lecture note
With the increasing power of computer systems, it is now possible to have services running
simultaneously that once would have required several separate machines. There are two
basic ways in which the increased power of computing has been harnessed. One way is to
virtualize the underlying hardware, introducing a layer of software on which several
complete and isolated systems can execute at the same time. Hardware virtualization is not
a new idea, but systems are now inexpensive and powerful enough that the use of
virtualization software is common in the industry. Another way to harness the power of
modern computing systems for multiple disparate purposes is to virtualize the services
themselves. Virtualization is what an operating system does with the underlying hardware,
making it appear to multiple programs that each of them has exclusive use of the machine.
One type of software virtualization are jails, which are containers for entire sets of programs
running on top of FreeBSD.
The networking and interprocess-communication subsystems in FreeBSD have been
virtualized so that many copies of the network subsystem can run in parallel. The framework
that virtualizes the network subsystems is referred to as VIMAGE. Each virtual network stack
is a world unto itself, with its own set of sockets and network interfaces. The
implementation of the FreeBSD network stack relies on a collection of kernel global variables
that maintain the data structures for all the network services. With the introduction of
VIMAGE, each data structure had to be virtualized, meaning that if there are N instances of
the network stack there are also N instances of each global variable. The global variables
defined by the stack are collected in a special linker set, which is a collection of global
variables that are encapsulated by the linker when a program, such as the kernel, is built.
The kernel uses the linker set, set_vnet, to create new instances of the network stack’s
global state whenever a new vnet instance is created. To reduce the overhead of finding a
global variable in a particular instance, a simple offset is used from the base of the memory
containing the virtualized global variables. A memory offset is the fastest way to effect a
lookup of the proper variable but it requires that the memory blocks containing the global
variables be exactly the same size and laid out the same way in memory.
If kernel developers had to do all this work themselves, it would be both tedious and error
prone. A small set of macros are used to declare variables that are global to the network
stack. The VNET_DEFINE macro is used throughout the kernel to set up global variables to be
used by VIMAGE. When modules need to refer to externally defined variables, they use the
VNET_DECLARE macro. Each virtualized global variable name is preceded by the characters
V_, which is a convention used in the kernel to denote virtualized global variables. A
complete set of each virtual stack’s global state is kept in a vnet structure shown below.
1
All vnets are kept on a singly-linked list and contain a count of the number of interfaces and
sockets that are currently in use by the virtual network instance. The global variables are
accessed via the vnet_data_mem pointer. Programmers do not access the global data
members directly but instead use the macros discussed above to indicate the global variable
that they are trying to access. When VIMAGE is not compiled into a kernel, all the macros
that handle the indirection and variable lookup are null and empty, meaning there is no
performance penalty for variable access when only a single network stack is in use.
Virtualized network stacks in FreeBSD 10 are inextricably tied to jails. A vnet is created via a
call to vnet_alloc() that is called from the jail_set system call. Each jail may contain only one
vnet. All the network stack’s global state is initialized using the same kernel routines in the
virtualized and nonvirtualized cases, with the VNET_ macros handling the proper indexing
and offsets at run time.
Mapping IPC-related system calls to vnet instances is handled in the kernel using the
credential structure associated with a thread. If a jail has been created with a vnet instance
then every process in the jail has a valid pointer to a vnet instance in its prison structure. The
system call then executes using the global variables ultimately pointed to from the prison
structure. User applications and system-management programs, such as netstat do not
expose vnet IDs to users of the system, but instead they, too, are a part of the jail and
cannot see any data structures not already encapsulated in the jail. When a user outside the
jail, such as a system administrator, wishes to look at the vnet instance inside a jail, he or she
uses the jexec command, which executes the requested program from within the jail,
thereby removing the need for anyone using the system to know the VNET ID of a vnet
instance.
Reference
M. K. McKusick, G. V. Neville-Neil and R. N.M. Watson. “The Design and Implementation of
the FreeBSD Operating System”. Second Edition. Addison-Wesley. pp. 644-646. November
2014.
