








           JJaaiillss:: CCoonnffiinniinngg tthhee oommnniippootteenntt rroooott..


            _P_o_u_l_-_H_e_n_n_i_n_g _K_a_m_p _<_p_h_k_@_F_r_e_e_B_S_D_._o_r_g_>
         _R_o_b_e_r_t _N_. _M_. _W_a_t_s_o_n _<_r_w_a_t_s_o_n_@_F_r_e_e_B_S_D_._o_r_g_>
                    The FreeBSD Project


                          _A_B_S_T_R_A_C_T

          The traditional UNIX security model is simple
     but inexpressive.  Adding fine-grained access con-
     trol improves the expressiveness, but  often  dra-
     matically  increases  both the cost of system man-
     agement and implementation complexity.   In  envi-
     ronments  with  a  more  complex management model,
     with delegation of some  management  functions  to
     parties  under  varying degrees of trust, the base
     UNIX model and most natural extensions  are  inap-
     propriate  at  best.   Where multiple mutually un-
     trusting parties are introduced, ``inappropriate''
     rapidly transitions to ``nightmarish'', especially
     with regards to data integrity and privacy protec-
     tion.

          The  FreeBSD  ``Jail''  facility provides the
     ability to partition the operating system environ-
     ment, while maintaining the simplicity of the UNIX
     ``root'' model.  In  Jail,  users  with  privilege
     find  that  the scope of their requests is limited
     to the jail,  allowing  system  administrators  to
     delegate  management capabilities for each virtual
     machine environment.  Creating virtual machines in
     this manner has many potential uses; the most pop-
     ular thus  far  has  been  for  providing  virtual
     machine  services  in  Internet  Service  Provider
     environments.


-----------
This  paper was presented at the 2nd International
System Administration  and  Networking  Conference
"SANE  2000"  May  22-25,  2000 in Maastricht, The
Netherlands and is published in the proceedings.
This       work       was       sponsored       by
http://www.servetheweb.com/  and  donated  to  the
FreeBSD Project for inclusion in the  FreeBSD  OS.
FreeBSD  4.0-RELEASE was the first release includ-
ing this code.  Follow-on work  was  sponsored  by
Safeport    Network   Services,   http://www.safe-
port.com/









                             -2-


11..  IInnttrroodduuccttiioonn

     The UNIX access control mechanism is  designed  for  an
environment with two types of users: those with, and without
administrative  privilege.   Within  this  framework,  every
attempt  is  made  to  provide an open system, allowing easy
sharing of files and inter-process communication.  As a mem-
ber  of  the  UNIX  family,  FreeBSD inherits these security
properties.  Users of FreeBSD in non-traditional UNIX  envi-
ronments must balance their need for strong application sup-
port, high network performance and  functionality,  and  low
total  cost of ownership with the need for alternative secu-
rity models that are difficult or  impossible  to  implement
with the UNIX security mechanisms.

     One  such  consideration is the desire to delegate some
(but not all) administrative functions to untrusted or  less
trusted   parties,  and  simultaneously  impose  system-wide
mandatory  policies  on  process  interaction  and  sharing.
Attempting  to create such an environment in the current-day
FreeBSD security environment is both difficult  and  costly:
in  many  cases,  the  burden of implementing these policies
falls on user applications, which means an increase  in  the
size and complexity of the code base, in turn translating to
higher development and maintenance cost,  as  well  as  less
overall flexibility.

     This abstract risk becomes more clear when applied to a
practical, real-world example: many  web  service  providers
turn  to  the  FreeBSD operating system to host customer web
sites, as it provides  a  high-performance,  network-centric
server  environment.  However, these providers have a number
of concerns on their plate, both in terms of protecting  the
integrity  and  confidentiality  of their own files and ser-
vices from their customers, as well as protecting the  files
and  services  of  one  customer  from (accidental or inten-
tional) access by any other customer.  At the same  time,  a
provider  would like to provide substantial autonomy to cus-
tomers, allowing them to  install  and  maintain  their  own
software,  and  to  manage  their  own services, such as web
servers and other content-related daemon programs.

     This problem space points strongly in the direction  of
a  partitioning  solution,  in  which customer processes and
storage are isolated from those of other customers, both  in
terms  of  accidental disclosure of data or process informa-
tion, but also in terms of the ability to  modify  files  or
processes  outside  of a compartment.  Delegation of manage-
ment functions within the system must be possible,  but  not
at the cost of system-wide requirements, including integrity
and privacy protection between partitions.

     However, UNIX-style access control makes it notoriously
difficult    to   compartmentalise   functionality.    While









                             -3-


mechanisms such as chroot(2) provide a modest level compart-
mentalisation,  it  is well known that these mechanisms have
serious shortcomings, both in terms of the  scope  of  their
functionality,   and  effectiveness  at  what  they  provide
[CHROOT].

     In the case of the chroot(2) call, a process's visibil-
ity  of  the  file  system name-space is limited to a single
subtree.  However, the compartmentalisation does not  extend
to  the  process  or  networking  spaces  and therefore both
observation of and interference with processes outside their
compartment is possible.

     To  this  end,  we  describe  the  new FreeBSD ``Jail''
facility, which provides  a  strong  partitioning  solution,
leveraging  existing  mechanisms, such as chroot(2), to what
effectively amounts to a virtual machine environment.   Pro-
cesses  in a jail are provided full access to the files that
they may manipulate, processes they may influence, and  net-
work  services  they can make use of, and neither access nor
visibility of files, processes or network  services  outside
their partition.

     Unlike other fine-grained security solutions, Jail does
not substantially increase the  policy  management  require-
ments  for  the system administrator, as each Jail is a vir-
tual FreeBSD environment permitting local policy to be inde-
pendently managed, with much the same properties as the main
system itself, making Jail easy to use for  the  administra-
tor, and far more compatible with applications.

22..  TTrraaddiittiioonnaall UUNNIIXX SSeeccuurriittyy,, oorr,, ````GGoodd,, rroooott,, wwhhaatt ddiiffffeerr--
eennccee??"" [[UUFF]]..

     The traditional UNIX access model assigns numeric  uids
to  each user of the system. In turn, each process ``owned''
by a user will be tagged with that user's uid in an unforge-
able  manner.   The  uids  serve  two  purposes: first, they
determine how discretionary access control  mechanisms  will
be  applied,  and second, they are used to determine whether
special privileges are accorded.

     In the case of discretionary access controls, the  pri-
mary  object protected is a file.  The uid (and related gids
indicating group membership) are mapped to a set  of  rights
for each object, courtesy the UNIX file mode, in effect act-
ing as a limited form of access control list.  Jail  is,  in
general,  not concerned with modifying the semantics of dis-
cretionary access control  mechanisms,  although  there  are
important implications from a management perspective.

     For  the purposes of determining whether special privi-
leges are accorded to a process, the check is  simple:  ``is
the  numeric  uid  equal  to  0  ?''.  If so, the process is









                             -4-


acting with ``super-user privileges'', and all access checks
are  granted,  in effect allowing the process the ability to
do whatever it wants to 1.

     For the purposes of human convenience, uid 0 is canoni-
cally  allocated  to the ``root'' user [ROOT].  For the pur-
poses of jail, this behaviour is extremely relevant: many of
these  privileged  operations  can  be used to manage system
hardware and configuration, file system name-space, and spe-
cial network operations.

     Many  limitations  to this model are immediately clear:
the root user is a single, concentrated source of  privilege
that  is  exposed to many pieces of software, and as such an
immediate target for attacks.  In the event of a  compromise
of  the  root capability set, the attacker has complete con-
trol over the system.  Even without an attacker,  the  risks
of a single administrative account are serious: delegating a
narrow scope of capability to an inexperienced administrator
is  difficult,  as  the granularity of delegation is that of
all system management abilities.  These  features  make  the
omnipotent  root  account  a  sharp, efficient and extremely
dangerous tool.

     The BSD family of operating  systems  have  implemented
the ``securelevel'' mechanism which allows the administrator
to block certain configuration and management functions from
being  performed  by root, until the system is restarted and
brought up into single-user mode.  While this  does  provide
some  amount  of protection in the case of a root compromise
of the machine, it does nothing to address the need for del-
egation of certain root abilities.

33..  OOtthheerr SSoolluuttiioonnss ttoo tthhee RRoooott PPrroobblleemm

     Many operating systems attempt to address these limita-
tions by providing fine-grained access controls  for  system
resources [BIBA].  These efforts vary in degrees of success,
but almost all suffer from at least  three  serious  limita-
tions:

     First,  increasing the granularity of security controls
increases the complexity of the administration  process,  in
turn  increasing both the opportunity for incorrect configu-
ration, as well as the  demand  on  administrator  time  and
resources.   In many cases, the increased complexity results
in significant frustration for the administrator, which  may
result  in  two disastrous types of policy: ``all doors open
as it's too much trouble'', and ``trust that the  system  is
secure, when in fact it isn't''.


-----------
  1 ... no matter how patently stupid it may be.









                             -5-


     The  extent  of  the trouble is best illustrated by the
fact that an entire niche  industry  has  emerged  providing
tools to manage fine grained security controls [UAS].

     Second, usefully segregating capabilities and assigning
them to running code and  users  is  very  difficult.   Many
privileged  operations  in UNIX seem independent, but are in
fact closely related, and the handing out of  one  privilege
may, in effect, be transitive to the many others.  For exam-
ple, in some trusted operating systems, a system  capability
may be assigned to a running process to allow it to read any
file, for the purposes of backup.  However, this  capability
is,  in  effect,  equivalent to the ability to switch to any
other account, as the ability to access  any  file  provides
access to system keying material, which in turn provides the
ability to authenticate as any user.  Similarly, many  oper-
ating  systems  attempt to segregate management capabilities
from auditing capabilities.  In a number of these  operating
systems,  however,  ``management  capabilities''  permit the
administrator to assign ``auditing capabilities'' to itself,
or  another  account, circumventing the segregation of capa-
bility.

     Finally,  introducing  new  security   features   often
involves  introducing  new  security  management APIs.  When
fine-grained capabilities  are  introduced  to  replace  the
setuid  mechanism  in  UNIX-like operating systems, applica-
tions that previously did an  ``appropriateness  check''  to
see  if  they were running as root before executing must now
be changed to know that they need not run as root.   In  the
case  of  applications  running with privilege and executing
other programs, there is now a new set  of  privileges  that
must  be  voluntarily given up before executing another pro-
gram.  These change can introduce significant  incompatibil-
ity  for existing applications, and make life more difficult
for application developers who may not be aware of differing
security semantics on different systems [POSIX1e].

44..  TThhee JJaaiill PPaarrttiittiioonniinngg SSoolluuttiioonn

     Jail  neatly  side-steps the majority of these problems
through  partitioning.   Rather  than  introduce  additional
fine-grained   access  control  mechanism,  we  partition  a
FreeBSD  environment  (processes,   file   system,   network
resources)  into  a  management  environment, and optionally
subset Jail environments.  In doing  so,  we  simultaneously
maintain the existing UNIX security model, allowing multiple
users and a privileged root user in each jail, while  limit-
ing  the  scope  of  root's  activities to his jail.  Conse-
quently the administrator of a FreeBSD machine can partition
the  machine  into separate jails, and provide access to the
super-user account in each of these without  losing  control
of the over-all environment.










                             -6-


     A process in a partition is referred to as ``in jail''.
When a FreeBSD system is booted up after a fresh install, no
processes  will  be  in jail.  When a process is placed in a
jail, it, and any descendents of the process  created  after
the  jail  creation, will be in that jail.  A process may be
in only one jail, and after creation, it can not  leave  the
jail.  Jails are created when a privileged process calls the
jail(2) syscall, with a description of the jail as an  argu-
ment  to the call.  Each call to jail(2) creates a new jail;
the only way for a new process  to  enter  the  jail  is  by
inheriting  access  to the jail from another process already
in that jail.  Processes may never leave the jail they  cre-
ated, or were created in.
Fig. 1 -- Schematic diagram of machine with two configured jails


     Membership in a jail involves a number of restrictions:
access to the file name-space is restricted in the style  of
chroot(2),  the ability to bind network resources is limited
to a specific IP address, the ability to  manipulate  system
resources  and perform privileged operations is sharply cur-
tailed, and the ability to interact with other processes  is
limited to only processes inside the same jail.

     Jail  takes  advantage of the existing chroot(2) behav-
iour to limit access  to  the  file  system  name-space  for
jailed  processes.  When a jail is created, it is bound to a
particular file system root.  Processes are unable to manip-
ulate  files  that  they  cannot  address,  and  as such the
integrity and confidentiality of files outside of  the  jail
file  system root are protected.  Traditional mechanisms for
breaking  out  of  chroot(2)  have  been  blocked.   In  the
expected and documented configuration, each jail is provided
with its exclusive file system root,  and  standard  FreeBSD
directory  layout, but this is not mandated by the implemen-
tation.

     Each jail is bound to a single  IP  address:  processes
within the jail may not make use of any other IP address for
outgoing or incoming connections; this includes the  ability
to  restrict  what  network  services  a particular jail may
offer.  As FreeBSD distinguishes attempts  to  bind  all  IP
addresses  from  attempts to bind a particular address, bind
requests for all IP addresses are redirected to the individ-
ual  Jail  address.   Some  network functionality associated
with privileged calls are  wholesale  disabled  due  to  the
nature  of  the functionality offered, in particular facili-
ties which would allow ``spoofing'' of IP numbers or disrup-
tive traffic to be generated have been disabled.

     Processes  running  without root privileges will notice
few, if any differences between a jailed environment or  un-
jailed  environment.  Processes running with root privileges
will find that many restrictions  apply  to  the  privileged









                             -7-


calls  they  may make.  Some calls will now return an access
error -- for example, an attempt to  create  a  device  node
will  now  fail.  Others will have a more limited scope than
normal -- attempts to bind a reserved  port  number  on  all
available  addresses will result in binding only the address
associated with the jail.  Other calls will succeed as  nor-
mal: root may read a file owned by any uid, as long as it is
accessible through the jail file system name-space.

     Processes within the  jail  will  find  that  they  are
unable to interact or even verify the existence of processes
outside the jail --  processes within the jail are prevented
from  delivering  signals  to processes outside the jail, as
well as connecting to those  processes  with  debuggers,  or
even  see them in the sysctl or process file system monitor-
ing mechanisms.  Jail does not prevent, nor is  it  intended
to  prevent,  the  use  of covert channels or communications
mechanisms via accepted interfaces -- for example, two  pro-
cesses  may  communicate  via  sockets  over  the IP network
interface.  Nor does it attempt to provide  scheduling  ser-
vices based on the partition; however, it does prevent calls
that interfere with normal process operation.

     As a result of these attempts to  retain  the  standard
FreeBSD  API and framework, almost all applications will run
unaffected.  Standard system services such as  Telnet,  FTP,
and SSH all behave normally, as do most third party applica-
tions, including the popular Apache web server.

55..  JJaaiill IImmpplleemmeennttaattiioonn

     Processes running with root privileges in the jail find
that there are serious restrictions on what it is capable of
doing -- in particular, activities that would extend outside
of the jail:

     +o  Modifying  the  running  kernel by direct access and
     loading kernel modules is prohibited.

     +o Modifying any of the  network  configuration,  inter-
     faces, addresses, and routing table is prohibited.

     +o Mounting and unmounting file systems is prohibited.

     +o Creating device nodes is prohibited.

     +o  Accessing raw, divert, or routing sockets is prohib-
     ited.

     +o Modifying kernel runtime  parameters,  such  as  most
     sysctl settings, is prohibited.

     +o  Changing  securelevel-related  file flags is prohib-
     ited.









                             -8-


     +o Accessing network resources not associated  with  the
     jail is prohibited.

     Other  privileged  activities  are permitted as long as
they are limited to the scope of the jail:

     +o Signalling any process within the jail is  permitted.

     +o  Changing  the  ownership and mode of any file within
     the jail is permitted, as long as the file flags permit
     this.

     +o  Deleting  any  file within the jail is permitted, as
     long as the file flags permit this.

     +o Binding reserved TCP and  UDP  port  numbers  on  the
     jails  IP  address is permitted.  (Attempts to bind TCP
     and UDP ports using INADDR_ANY will  be  redirected  to
     the jails IP address.)

     +o  Functions which operate on the uid/gid space are all
     permitted since  they  act  as  labels  for  filesystem
     objects  of proceses which are partitioned off by other
     mechanisms.

     These restrictions on root access limit  the  scope  of
root  processes,  enabling  most applications to run un-hin-
dered, but preventing calls that might allow an  application
to  reach  beyond  the jail and influence other processes or
system-wide configuration.


66..  IImmpplleemmeennttaattiioonn jjaaiill iinn tthhee FFrreeeeBBSSDD kkeerrnneell..

66..11..  TThhee jjaaiill((22)) ssyysstteemm ccaallll,, aallllooccaattiioonn,,  rreeffccoouunnttiinngg  aanndd
ddeeaallllooccaattiioonn ooff struct prison..

     The  jail(2)  system  call  is  implemented  as  a non-
optional system call in FreeBSD.   Other  system  calls  are
controlled  by compile time options in the kernel configura-
tion file, but due to  the  minute  footprint  of  the  jail
implementation,  it was decided to make it a standard facil-
ity in FreeBSD.

     The implementation of the system call  is  straightfor-
ward:   a data structure is allocated and populated with the
arguments provided.  The data structure is attached  to  the
current process' struct proc, its reference count set to one
and a call to the chroot(2) syscall implementation completes
the task.

     Hooks  in  the  code  implementing process creation and
destruction maintains the reference count on the data struc-
ture  and  free it when the last reference is lost.  Any new









                             -9-


process created by a process in a jail will inherit a refer-
ence  to the jail, which effectively puts the new process in
the same jail.

     There is no way to modify  the  contents  of  the  data
structure describing the jail after its creation, and no way
to attach a process to an existing jail if it was  not  cre-
ated from the inside that jail.

66..22..  FFoorrttiiffiiccaattiioonn ooff tthhee cchhrroooott((22)) ffaacciilliittyy ffoorr ffiilleessyysstteemm
nnaammee ssccooppiinngg..

     A  number  of  ways  to  escape  the  confines   of   a
chroot(2)-created  subscope of the filesystem view have been
identified over the years.  chroot(2) was never intended  to
be  security mechanism as such, but even then the ftp daemon
largely depended on the security provided  by  chroot(2)  to
provide the ``anonymous ftp'' access method.

     Three  classes  of  escape  routes  existed:  recursive
chroot(2) escapes, ``..'' based escapes and fchdir(2)  based
escapes.   All  of  these  exploited the fact that chroot(2)
didn't try sufficiently hard to enforce the new root  direc-
tory.

     New code were added to detect and thwart these escapes,
amongst other things by tracking the directory of the  first
level  of  chroot(2)  experienced  by a process and refusing
backwards traversal across this directory, as well as  addi-
tional  code  to  refuse  chroot(2) if file-descriptors were
open referencing directories.

66..33..  RReessttrriiccttiioonn ooff pprroocceessss vviissiibbiilliittyy aanndd iinntteerraaccttiioonn..

     A macro was already  in  available  in  the  kernel  to
determine if one process could affect another process.  This
macro did the rather complex checking of uid and gid values.
It  was felt that the complexity of the macro were approach-
ing the lower edge of IOCCC entrance criteria,  and  it  was
therefore  converted  to  a  proper  function  named p_tres-
pass(p1, p2) which does all the previous  checks  and  addi-
tionally checks the jail aspect of the access.  The check is
implemented such that access fails if the origin process  is
jailed but the target process is not in the same jail.

     Process  visibility  is provided through two mechanisms
in FreeBSD, the procfs file system and  a  sub-tree  of  the
sysctl tree.  Both of these were modified to report only the
processes in the same jail to a jailed process.

66..44..  RReessttrriiccttiioonn ttoo oonnee IIPP nnuummbbeerr..

     Restricting TCP and UDP access to just  one  IP  number
was   done   almost  entirely  in  the  code  which  manages









                            -10-


``protocol control blocks''.  When a jailed process binds to
a  socket, the IP number provided by the process will not be
used, instead the pre-configured IP number of  the  jail  is
used.

     BSD  based TCP/IP network stacks sport a special inter-
face, the loop-back interface, which has  the  ``magic''  IP
number  127.0.0.1.   This is often used by processes to con-
tact servers on the local machine, and consequently  special
handling  for jails were needed.  To handle this case it was
necessary to also intercept and modify the behaviour of con-
nection  establishment,  and when the 127.0.0.1 address were
seen from a jailed process, substitute the jails  configured
IP number.

     Finally  the  APIs through which the network configura-
tion and connection state may be queried  were  modified  to
report only information relevant to the configured IP number
of a jailed process.

66..55..  AAddddiinngg jjaaiill aawwaarreenneessss ttoo sseelleecctteedd ddeevviiccee ddrriivveerrss..

     A couple of device drivers needed to  be  taught  about
jails,  the  ``pty''  driver is one of them.  The pty driver
provides ``virtual terminals'' to services like telnet, ssh,
rlogin  and  X11  terminal window programs.  Therefore jails
need access to the pty driver, and code had to be  added  to
enforce that a particular virtual terminal were not accessed
from more than one jail at the same time.

66..66..  GGeenneerraall rreessttrriiccttiioonn ooff ssuuppeerr--uusseerrss ppoowweerrss  ffoorr  jjaaiilleedd
ssuuppeerr--uusseerrss..

     This item proved to be the simplest but most tedious to
implement.  Tedious because a manual review  of  all  places
where  the kernel allowed the super user special powers were
called for, simple because very few places were required  to
let  a jailed root through.  Of the approximately 260 checks
in the FreeBSD 4.0 kernel, only about 35 will let  a  jailed
root through.

     Since  the  default  is for jailed roots to not receive
privilege, new code or drivers in  the  FreeBSD  kernel  are
automatically  jail-aware:  they  will  refuse  jailed roots
privilege.  The other part of this protection comes from the
fact  that a jailed root cannot create new device nodes with
the mknod(2) systemcall, so unless the machine administrator
creates  device  nodes  for  a  particular device inside the
jails filesystem tree, the driver in effect does  not  exist
in the jail.

     As  a  side-effect  of  this work the suser(9) API were
cleaned up and extended to  cater  for  not  only  the  jail
facility,  but  also  to  make  room for future partitioning









                            -11-


facilities.

66..77..  IImmpplleemmeennttaattiioonn ssttaattiissttiiccss

     The change of the  suser(9)  API  modified  approx  350
source lines distributed over approx. 100 source files.  The
vast majority of these changes were generated  automatically
with a script.

     The  implementation  of  the jail facility added approx
200 lines of code in  total,  distributed  over  approx.  50
files.  and about 200 lines in two new kernel files.

77..  MMaannaaggiinngg JJaaiillss aanndd tthhee JJaaiill FFiillee SSyysstteemm EEnnvviirroonnmmeenntt

77..11..  CCrreeaattiinngg aa JJaaiill EEnnvviirroonnmmeenntt

     While  the  jail(2)  call  could be used in a number of
ways, the expected configuration creates a complete  FreeBSD
installation  for  each  jail.   This includes copies of all
relevant system binaries,  data  files,  and  its  own  /etc
directory.   Such a configuration maximises the independence
of various jails, and reduces the  chances  of  interference
between  jails  being possible, especially when it is desir-
able to provide root access within a jail to a less  trusted
user.

     On  a  box making use of the jail facility, we refer to
two types of environment: the host environment, and the jail
environment.   The  host  environment  is the real operating
system environment, which is used to  configure  interfaces,
and  start  up  the  jails.  There are then one or more jail
environments, effectively virtual  FreeBSD  machines.   When
configuring  Jail for use, it is necessary to configure both
the host and jail environments to prevent overlap.

     As jailed virtual machines are generally bound to an IP
address  configured  using  the  normal  IP alias mechanism,
those jail IP addresses are also accessible to host environ-
ment applications to use.  If the accessibility of some host
applications in the jail environment is not desirable, it is
necessary  to configure those applications to only listen on
appropriate addresses.

     In most of the production environments  where  jail  is
currently  in  use,  one IP address is allocated to the host
environment, and then a number are allocated to jail  boxes,
with  each  jail  box receiving a unique IP.  In this situa-
tion, it is sufficient to configure the networking  applica-
tions on the host to listen only on the host IP.  Generally,
this consists of specifying the appropriate IP address to be
used  by  inetd and SSH, and disabling applications that are
not capable of limiting their address scope, such  as  send-
mail,  the  port  mapper,  and  syslogd.   Other third party









                            -12-


applications that have been installed on the host must  also
be  configured  in  this  manner, or users connecting to the
jailbox will discover the host environment  service,  unless
the  jailbox  has specifically bound a service to that port.
In some situations, this can actually be the  desirable  be-
haviour.

     The  jail  environments must also be custom-configured.
This consists of building and installing a miniature version
of the FreeBSD file system tree off of a subdirectory in the
host environment, usually /usr/jail, or /data/jail,  with  a
subdirectory  per jail.  Appropriate instructions for gener-
ating this tree are included in the jail(8)  man  page,  but
generally  this  process  may be automated using the FreeBSD
build environment.

     One notable difference from the default FreeBSD install
is  that  only  a limited set of device nodes should be cre-
ated.

     To improve storage efficiency, a  fair  number  of  the
binaries  in the system tree may be deleted, as they are not
relevant in a jail environment.  This includes  the  kernel,
boot loader, and related files, as well as hardware and net-
work configuration tools.

     After the creation of the jail tree, the easiest way to
configure  it  is  to start up the jail in single-user mode.
The sysinstall admin tool may be used to help with the task,
although  it is not installed by default as part of the sys-
tem tree.  These tools should be run in  the  jail  environ-
ment,  or they will affect the host environment's configura-
tion.

     # mkdir /data/jail/192.168.11.100/stand
     # cp /stand/sysinstall /data/jail/192.168.11.100/stand
     # jail /data/jail/192.168.11.100 testhostname 192.168.11.100 \
          /bin/sh


     After running the jail command, the shell is now within
the  jail environment, and all further commands will be lim-
ited to the scope of the jail until the shell exits.  If the
network  alias  has  not  yet been configured, then the jail
will be unable to access the network.

     The startup configuration of the jail  environment  may
be  configured  so  as  to quell warnings from services that
cannot run in the jail.  Also, any per-system  configuration
required  for  a  normal FreeBSD system is also required for
each jailbox.  Typically, this includes:

     +o Create empty /etc/fstab










                            -13-


     +o Disable portmapper

     +o Run newaliases

     +o Disabling interface configuration

     +o Configure the resolver

     +o Set root password

     +o Set timezone

     +o Add any local accounts

     +o Install any packets

77..22..  SSttaarrttiinngg JJaaiillss

     Jails are typically started by executing their  /etc/rc
script  in  much  the same manner a shell was started in the
previous section.  Before starting the  jail,  any  relevant
networking  configuration  should  also be performed.  Typi-
cally, this involves adding an additional IP address to  the
appropriate  network  interface,  setting network properties
for the IP address using IP filtering, forwarding, and band-
width  shaping,  and  mounting a process file system for the
jail, if the ability to debug processes from within the jail
is desired.

     # ifconfig ed0 inet add 192.168.11.100 netmask 255.255.255.255
     # mount -t procfs proc /data/jail/192.168.11.100/proc
     # jail /data/jail/192.168.11.100 testhostname 192.168.11.100 \
          /bin/sh /etc/rc


     A  few warnings are generated for sysctl's that are not
permitted to be set within the jail, but the end result is a
set  of  processes in an isolated process environment, bound
to a single IP address.  Normal procedures for  accessing  a
FreeBSD  machine  apply:  telneting  in  through the network
reveals a telnet prompt, login, and shell.

     % ps ax
       PID  TT  STAT      TIME COMMAND
       228  ??  SsJ    0:18.73 syslogd
       247  ??  IsJ    0:00.05 inetd -wW
       249  ??  IsJ    0:28.43 cron
       252  ??  SsJ    0:30.46 sendmail: accepting connections on port 25
       291  ??  IsJ    0:38.53 /usr/local/sbin/sshd
     93694  ??  SJ     0:01.01 sshd: rwatson@ttyp0 (sshd)
     93695  p0  SsJ    0:00.06 -csh (csh)
     93700  p0  R+J    0:00.00 ps ax











                            -14-


     It is  immediately  obvious  that  the  environment  is
within  a  jailbox: there is no init process, no kernel dae-
mons, and a J flag is present beside all processes  indicat-
ing the presence of a jail.

     As with any FreeBSD system, accounts may be created and
deleted, mail is delivered, logs are generated, packages may
be  added,  and  the system may be hacked into if configured
incorrectly, or running a buggy version of a piece of  soft-
ware.   However,  all  of  this  happens strictly within the
scope of the jail.

77..33..  JJaaiill MMaannaaggeemmeenntt

     Jail management is an interesting  prospect,  as  there
are  two  perspectives  from which a jail environment may be
administered: from within the jail, and from the host  envi-
ronment.   From  within  the  jail,  as described above, the
process  is  remarkably  similar  to  any  regular   FreeBSD
install,  although  certain  actions are prohibited, such as
mounting file systems, modifying system  kernel  properties,
etc.  The only area that really differs are that of shutting
the system down: the processes within the jail  may  deliver
signals  between  them, allowing all processes to be killed,
but bringing the system back up requires  intervention  from
outside of the jailbox.

     From outside of the jail, there are a range of capabil-
ities, as well as limitations.  The jail environment is,  in
effect, a subset of the host environment: the jail file sys-
tem appears as part of the host  file  system,  and  may  be
directly  modified  by  processes  in  the host environment.
Processes within the jail appear in the process  listing  of
the  host,  and  may likewise be signalled or debugged.  The
host process file system makes  the  hostname  of  the  jail
environment  accessible  in  /proc/procnum/status,  allowing
utilities in the host environment to manage processes  based
on  jailname.   However,  the  default  configuration allows
privileged processes within jails to set the hostname of the
jail, which makes the status file less useful from a manage-
ment perspective if the contents of the jail are  malicious.
To  prevent  a  jail  from changing its hostname, the "secu-
rity.jail.set_hostname_allowed" sysctl may be set to 0 prior
to starting any jails.

     One  aspect  immediately  observable  in an environment
with multiple jails is that uids and gids are local to  each
jail  environment:  the uid associated with a process in one
jail may be for a different user than in another jail.  This
collision  of  identifiers is only visible in the host envi-
ronment, as normally processes from one jail are never visi-
ble  in  an  environment with another scope for user/uid and
group/gid mapping.  Managers in the host environment  should
understand these scoping issues, or confusion and unintended









                            -15-


consequences may result.

     Jailed processes are subject to the normal restrictions
present  for  any  processes, including resource limits, and
limits placed by the network code, including firewall rules.
By  specifying  firewall rules for the IP address bound to a
jail, it is possible to  place  connectivity  and  bandwidth
limitations  on  individual jails, restricting services that
may be consumed or offered.

     Management of jails is an area that  will  see  further
improvement  in  future  versions of FreeBSD.  Some of these
potential improvements are discussed later in this paper.

88..  FFuuttuurree DDiirreeccttiioonnss

     The jail facility has already been deployed in numerous
capacities and a few opportunities for improvement have man-
ifested themselves.

88..11..  IImmpprroovveedd VViirrttuuaalliissaattiioonn

     As it stands, the jail code provides a strict subset of
system resources to the jail environment, based on access to
processes, files, network  resources,  and  privileged  ser-
vices.   Virtualisation,  or  making  the  jail environments
appear to be fully functional FreeBSD systems, allows  maxi-
mum  application  support  and  the  ability to offer a wide
range of services within a jail environment.  However, there
are  a number of limitations on the degree of virtualisation
in the current code, and  removing  these  limitations  will
enhance the ability to offer services in a jail environment.
Two areas that deserve greater attention are the virtualisa-
tion  of  network  resources,  and  management of scheduling
resources.

     Currently, a single IP address may be allocated to each
jail, and all communication from the jail is limited to that
IP  address.   In  particular,  these  addresses  are   IPv4
addresses.  There has been substantial interest in improving
interface virtualisation, allowing one or more addresses  to
be  assigned  to  an interface, and removing the requirement
that the address be an IPv4 address,  allowing  the  use  of
IPv6.   Also, access to raw sockets is currently prohibited,
as the current implementation of raw sockets  allows  access
to  raw IP packets associated with all interfaces.  Limiting
the scope of the raw socket would allow its safe use  within
a  jail,  re-enabling  support  for  ping, and other network
debugging and evaluation tools.

     Another area of great interest to the current consumers
of  the  jail code is the ability to limit the impact of one
jail  on  the  CPU  resources  available  for  other  jails.
Specifically,  this would require that the jail of a process









                            -16-


play a rule in its scheduling parameters.  Prior work in the
area  of  lottery scheduling, currently available as patches
on FreeBSD 2.2.x, might be leveraged to allow some degree of
partitioning  between  jail  environments  [LOTTERY1]  [LOT-
TERY2].  However, as the  current  scheduling  mechanism  is
targeted  at  time  sharing,  and FreeBSD does not currently
support real time preemption of processes  in  kernel,  com-
plete partitioning is not possible within the current frame-
work.

88..22..  IImmpprroovveedd MMaannaaggeemmeenntt

     Management of jail environments is  currently  somewhat
ad  hoc--creating  and  starting  jails is a well-documented
procedure, but day-to-day management of jails,  as  well  as
special  case procedures such as shutdown, are not well ana-
lysed and documented.  The current kernel process management
infrastructure  does not have the ability to manage pools of
processes in a jail-centric way.  For example, it is  possi-
ble  to, within a jail, deliver a signal to all processes in
a jail, but it is not possibly to atomically target all pro-
cesses  within a jail from outside of the jail.  If the jail
code is to effectively limit the behaviour of  a  jail,  the
ability  to  shut  it down cleanly is paramount.  Similarly,
shutting down a jail cleanly from within is  also  not  well
defined,  the  traditional  shutdown  utilities  having been
written with a host environment in mind.   This  suggests  a
number  of improvements, both in the kernel and in the user-
land utility set.

     First, the ability to address kernel-centric management
mechanisms  at  jails  is  important.  One way in which this
might be done is to assign a unique jail id,  not  unlike  a
process  id  or  process group id, at jail creation time.  A
new jailkill() syscall would permit the direction of signals
to  specific jailids, allowing for the effective termination
of all processes in the jail.  A unique  jailid  could  also
supplant  the  hostname as the unique identifier for a jail,
allowing the hostname to be changed by the processes in  the
jail without interfering with jail management.

     More  carefully  defining  the user-land semantics of a
jail during startup and shutdown  is  also  important.   The
traditional FreeBSD environment makes use of an init process
to bring the system up  during  the  boot  process,  and  to
assist  in  shutdown.  A similar technique might be used for
jail, in effect a jailinit, formulated to handle  the  clean
startup  and  shutdown,  including calling out to jail-local
/etc/rc.shutdown, and other useful  shutdown  functions.   A
jailinit  would also present a central location for deliver-
ing management requests to within a jail from the host envi-
ronment,  allowing the host environment to request the shut-
down of the jail cleanly, before  resorting  to  terminating
processes,  in  the  same  style  as  the  host  environment









                            -17-


shutting down before killing all processes and  halting  the
kernel.

     Improvements  in the host environment would also assist
in improving jail management, possibly  including  automated
runtime  jail  management  tools,  tools to more easily con-
struct the per-jail file system area, and include jail shut-
down as part of normal system shutdown.

     These  improvements in the jail framework would improve
both raw functionality and usability from a management  per-
spective.   The jail code has raised significant interest in
the FreeBSD community, and it is hoped  that  this  type  of
improved   functionality   will  be  available  in  upcoming
releases of FreeBSD.

99..  CCoonncclluussiioonn

     The jail facility provides FreeBSD with a  conceptually
simple security partitioning mechanism, allowing the delega-
tion of administrative rights within virtual machine  parti-
tions.

     The  implementation relies on restricting access within
the jail environment to a well-defined subset of the overall
host   environment.    This  includes  limiting  interaction
between processes, and  to  files,  network  resources,  and
privileged  operations.   Administrative overhead is reduced
through avoiding fine-grained access control mechanisms, and
maintaining  a  consistent  administrative  interface across
partitions and the host environment.

     The jail facility has already seen  widespread  deploy-
ment in particular as a vehicle for delivering "virtual pri-
vate server" services.

     The jail code is included in the base system as part of
FreeBSD 4.0-RELEASE, and fully documented in the jail(2) and
jail(8) man-pages.
























                            -18-


NNootteess && RReeffeerreenncceess

[BIBA]
     K. J. Biba, Integrity Considerations  for  Secure  Com-
     puter Systems, USAF Electronic Systems Division, 1977

[CHROOT]
     Dr.  Marshall  Kirk  Mckusick,  private  communication:
     ``According to the SCCS logs, the chroot call was added
     by  Bill  Joy on March 18, 1982 approximately 1.5 years
     before 4.2BSD was released.  That was  well  before  we
     had ftp servers of any sort (ftp did not show up in the
     source tree until January 1983).  My best guess  as  to
     its  purpose  was  to  allow  Bill  to  chroot into the
     /4.2BSD build directory and build a system  using  only
     the  files,  include files, etc contained in that tree.
     That was the only use of chroot that  I  remember  from
     the early days.''

[LOTTERY1]
     David   Petrou  and  John  Milford.  Proportional-Share
     Scheduling: Implementation and Evaluation in a  Widely-
     Deployed Operating System, December 1997.
     http://www.cs.cmu.edu/~dpetrou/papers/freebsd_lottery_writeup98.ps
     http://www.cs.cmu.edu/~dpetrou/code/freebsd_lottery_code.tar.gz

[LOTTERY2]
     Carl  A.  Waldspurger  and  William  E.  Weihl. Lottery
     Scheduling: Flexible Proportional-Share  Resource  Man-
     agement,  Proceedings of the First Symposium on Operat-
     ing Systems Design and Implementation (OSDI '94), pages
     1-11, Monterey, California, November 1994.
     http://www.research.digital.com/SRC/personal/caw/papers.html

[POSIX1e]
     Draft  Standard  for Information Technology -- Portable
     Operating System Interface (POSIX) --  Part  1:  System
     Application  Program Interface (API) -- Amendment: Pro-
     tection, Audit and Control Interfaces [C Language] IEEE
     Std 1003.1e Draft 17 Editor Casey Schaufler

[ROOT]
     Historically other names have been used at times, Zilog
     for instance called the super-user account ``zeus''.

[UAS]
     One such niche product is the ``UAS'' system  to  main-
     tain and audit RACF configurations on MVS systems.
     http://www.entactinfo.com/products/uas/

[UF] Quote from the User-Friendly cartoon by Illiad.
     http://www.userfriendly.org/cartoons/archives/98nov/19981111.html








