Installing FreeBSD 8.x via serial console and PXE

These are the things you are absolutely going to need, no questions asked. If you don't have any of these, you should stop here.

Note: This procedure works for i386 and amd64. There's no need to change any of the below pathnames for amd64. It also doesn't matter if you build the pxeboot(8) binary on an amd64 box while the machine that plans on using it is i386; the bootstraps are purely i386.

Chapter 2 Common Paths/Terms Used

Below are a list of terms, paths, directories, and filenames which are commonly used throughout this document.

Term	Description
`192.168.1.88`	The IP of the DHCP, TFTP, and NFS server
`192.168.1.100`	The IP of the box which we're doing the PXE boot on/from
`nfsserver.home.lan`	FQDN of 192.168.1.88
`newbox.home.lan`	FQDN of 192.168.1.100
`/usr/local/freebsd8`	Directory containing contents of the FreeBSD 8.x "disc1" CD; you will be modifying some files in this directory, so it cannot be a mounted CD (e.g. read-only)
`boot0` `boot2` `loader`	All of these are the different stages of the FreeBSD bootstraps. All are used when physical media is used for installation (e.g. floppies, CDROM, hard disk, USB thumb drive, etc.), but aren't used when PXE booting. loader(8) is technically used, but it's actually called pxeboot(8). See below for details on that
`pxeboot` `pxeldr`	This is the sole binary that gets fetched via TFTP after the Intel PXE boot ROM gets an IP via DHCP. The binary itself is actually a modified version of loader(8), and includes some extra features (which we'll use)

Chapter 3 Installation

3.1 Configuring DHCP

I'm familiar with ISC's DHCP server, available as net/isc-dhcp41-server. The important pieces you need to add to your /usr/local/etc/dhcpd.conf are here:

Since not a single other document on the Web has described what the important options do, I'll describe their purpose:

This line isn't necessary if the NFS server and the DHCP server are on the same machine. Including it doesn't hurt.

When a DHCP request is made from the Intel PXE boot ROM on the NIC, there are additional parameters you can tell it besides just an IP number. This argument tells the PXE boot ROM what filename to do a TFTP request for now that it has an IP address. It's the first piece of code which gets run after the initial DHCP IP negotiation is completed.

This is for FreeBSD. It tells pxeboot(8) where the root filesystem should be mounted from, NFS-wise. Thus, the kernel will do the equivalent of the following, in an attempt to get a working root filesystem, and then start init(8):

At this point, you should enable the ISC DHCP server in /etc/rc.conf, and start the daemon using the appropriate rc.d script in /usr/local/etc/rc.d.

3.2 Configuring TFTP

Uncomment the following line in /etc/inetd.conf, and modify it so that the -s flag points to /usr/local (that's where tftpd(8) will chroot(2) before serving content):

If you already had inetd running prior to modifying /etc/inetd.conf, all you need to do is send a SIGHUP to it:

3.3 Configuring NFS

Now we need to add an NFS export. Edit /etc/exports and add the following lines to it:

This example assumes your network is 192.168.1.0/24; if it's something else, you'll need to make appropriate changes.

Next, don't forget to make the exported directory. Don't worry about the ownership or permissions, we'll fix those later:

If these were running beforehand and you simply edited /etc/exports, all you need to do is send a SIGHUP to mountd(8): to get it to re-read the exports:

3.4 Extracting the FreeBSD CD contents for use via NFS

We're going to need to copy all the contents off the FreeBSD 8.x "disc1" CD to /usr/local/freebsd8 before continuing. In this example, the NFS server has the ISO file for 8.0-STABLE "disc1" on it, and we're going to extract the contents.

FreeBSD tar is linked to a library called libarchive, which understands the ISO9660 format and Joliet extensions. Simply put, this means you can extract the contents of an .iso file using tar:

3.5 Rebuilding pxeboot for exclusive serial access

FreeBSD's bootloader/bootstraps, by default, are intended to be used and viewed via a VGA console. When it comes to utilising a serial port, things get a little hairy:

First, serial access is limited to 9600bps tops, and requires that the bootloader be rebuilt to support higher speeds. Sure, you can override this using loader.conf, but that's for loader(8) and not for boot(2) or earlier stages.

Most people work around this by creating a file in their root filesystem called /boot.config that contains the argument -S115200 as well as -Dh (see the FreeBSD Handbook for details). This allows for serial output during the boot2 stage, as well as subsequent stages such as loader.

However, when PXE booting, there's only one piece of the bootstrap used: pxeboot(8). This means the only solution is to rebuild the boot blocks with a serial port speed that has the speed you want (in this case, 115200bps). But first, let's clear something up: pxeboot(8) is actually nothing more than loader(8) with some extra code (see src/sys/boot/i386/pxeldr.S) in it. loader(8) uses a library called libi386, which is a library used by the bootstraps only (it's it's not something you'll find in /usr/lib or /usr/libexec). libi386 is what actually controls what the serial port speed is set to, thus, anything using libi386 should have support for higher serial console speeds. The BOOT_COMCONSOLE_SPEED and BOOT_COMCONSOLE_PORT variables are used in libi386.

Secondly, pxeboot(8) reads loader.conf "too late" during the boot procedure, so if you have issues that require commands to be typed into the loader ("OK") prompt, you're out of luck. Rebuilding pxeboot(8) to use the serial port exclusively solves this.

Note: Do not type make install after this step! We don't want to modify the bootstraps on the NFS server itself, we just want a 115200bps-capable pxeboot(8) binary that uses the serial port exclusively!

The removal of all contents within /usr/obj may seem crazy, but it's actually required for some of the below steps to work properly. If /usr/obj is populated from an old buildworld or buildkernel, the make will end up storing files there rather than in the current working directory.

The first parameter, BOOT_PXELDR_ALWAYS_SERIAL=1, explicitly causes all output to go to the serial port instead of the VGA console. This means that you'll get serial output early on (before loader.conf is read), which can really come in handy if things go awry. This option doesn't appear to be documented anywhere besides in the source code.

The second parameter, BOOT_COMCONSOLE_SPEED=115200, causes the serial port to be initialised at 115200bps. If you honestly prefer to use 9600bps, you don't have to specify this at all. This option is documented. For those somewhat familiar with serial consoles on FreeBSD: by specifying this parameter you won't need comconsole_speed in loader.conf.

Finally, there's another undocumented option specific to pxeboot(8) (and isn't part of libi386): BOOT_PXELDR_PROBE_KEYBOARD=1. This causes pxeboot(8) to to look for an attached PS/2 or AT keyboard, and if there is one, use the VGA console and local keyboard as the console device. Otherwise, it defaults to serial console.

Now that we have a 115200bps-capable pxeboot(8) binary, let's use it instead of the one originally on the CD image:

At this point, a PXE booting client should now have the capability to set the serial port speed to 115200bps, and should provide output early on during the booting process.

Finally, don't forget to clean up your /sys/boot directory; we don't want cruft laying around in there:

3.6 Configuring loader.conf

Now we have to update /usr/local/freebsd8/boot/loader.conf to tell it to utilise serial console, and more importantly, point to where the root filesystem (for booting) will be. The lines in bold are the ones you want to add; the others shown should already be there (don't mess with them!):

vfs.root.mountfrom tells the kernel that after it's done booting, to try to load the root filesystem from ufs:/dev/md0 before starting init(8). It's important to note that the device parameter is /dev/md0 and not /dev/md0c like on FreeBSD 7.x. This is normal; there have been GEOM and libdisk improvements which no longer require the "c" slice to exist (you'll find this to be true when creating slices during sysinstall(8) too).

If you specify the wrong device, the root filesystem won't mount. If the root filesystem mounting procedure fails, FreeBSD will continue on and attempt to mount the root filesystem via NFS, which is not what we want. If you want to know more about this part of the booting procedure, you should look at the related Forth files, as well as loader.rc in the same directory.

The mfs_root loading mechanism is programmed to support gzip decompression automatically, so it will also look for the /boot/mfsroot.gz file. In FreeBSD 8.x this actually works, while in FreeBSD 7.x and earlier the gzip decompression would randomly fail.

3.7 Fixing file permissions to work with TFTP and NFS

We now need to fix the permissions on all the files we've extracted, as well as the permissions on the pxeboot(8) binary we copied. This is important for two reasons:

3.8 Attempting your first PXE boot

Alright, it's time to give it a try and see if it breaks! Be patient, since most of the TFTP loading is slow (it's UDP-based, and uses small UDP packets). If all goes well, you should see something like this appear:

Hit Enter here, and let things boot normally. If all goes well, you should begin to see standard kernel output. Notible lines are shown here:

Pick a terminal type, and the rest should be obvious: the standard sysinstall(8) selection menu, and all that jazz.

3.9 Choosing an installation medium

When prompted what installation medium you want to install from, you can pick Install over NFS and enter the name of the share path manually, e.g.:

Or, if you want to install from a public FreeBSD server via HTTP or FTP or some other method, you can choose that. It's up to you.

From there, the installation should go smoothly. But there are some pending things you should remember to configure after the installation but before rebooting...

Chapter 4 Creating /boot.config for high-speed serial

This item is incredibly important. Yes, I realise I haven't written this section of the document yet, but it's still important! Failure to set this up will result in FreeBSD's serial ports operating at a maximum speed of 9600bps. Remember the rigmarole we went through in Chapter 3.5? You get to deal with that again, but this time for the standard non-PXE bootstraps.

I often forget this step when deploying a new server. The workaround is to set your terminal/serial connection's speed to 9600bps after rebooting the new system post-installation, then create /boot.config after-the-fact. This is hoky, and may result in some garbage on your screen briefly, but it works and is a one-time deal. The command echo "-S115200 -Dh" > /boot.config is all that's needed, followed by a reboot.
This nonsense could be avoided if sysinstall(8) offered the ability to edit an arbitrary file on the filesystem post-install but pre-reboot; you know, like what's offered when it comes to editing /etc/ttys in Chapter 6?

Chapter 5 Important Networking options

When configuring your network interface, remember to set a DNS server! Otherwise, things like sshd(8) (see below) won't start.

Chapter 6 Post-installation Configuration options

Things I strongly recommend setting when given the chance to "set any last configuration options" prior to the machine rebooting:

The last item is incredibly important. Without it, your machine will boot as normal, but you'll never see a login: prompt on the serial port. You'll very likely have to go to the physical console and reboot the machine manually to fix this mistake.