Gentoo Diskless Client

From NBSWiki
Jump to: navigation, search

Gentoo Diskless HOWTO for building an SSI image. Used in conjunction with the and Installing Gentoo as a Beowulf Head node, you might just end up with a Beowulf cluster in your hands! This document only gets you to the point of having a system image which is bootable via NFS and gives you a functional console. Adding apps is up to you. If you want to make an X Terminal SSI, check out the references, I haven't had the time to MueCow Gentoo yet.


Reference articles

I try to keep these in order of importance/relevance. More references are inserted in the relevant sections.
HOWTO Diskless X terminal with read-only root
This link is of main interest (Wilhelm Meier's web site on the subject) Especially the "Preparation of the client-root" section
Gentoo Diskless Clients Installation, Configuration by Wilhelm Meier, Markus Müller, Zweibrücken
Project MueKow
Ubuntu's ThinClientIntegration
Linux Boot

Present limitation

  • This procedure is to give you a base system
  • INIT scripts which should and shouldn't run on boot time still need to be identified (ie: do we really need to run checkfs when booting an SSI system).
    • Update, use baselayout 1.12.*, it's much nicer on these processes Kyron 15:22, 1 June 2006 (EDT)
  • I am presently concentrating on building this environment on locally compatible ARCHes (ie: building the root for a PIII on a P4, Athlon-XP on amd64 Opteron...). Even so, you will have to use the following command to chroot into your node's tree (if your on an amd64 and the nodes are older generation AMD procs (x86)):
linux32 chroot . /bin/bash instead of chroot . /bin/bash
  • My true interest is to use this approach for building a "dynamic" cluster. So I won't spend too much time on the SSI as a workstation aspect although I will work on "MueKow"ing Gentoo at one point ;P
  • The approach explored here is a custom kernel for custom hardware. This implies that I won't get into the complicated, yet flexible, approach which requires and INItial Ram Disk (initrd).
  • We should use dev-util/cpuinfo-collection to identify and select the proper arch-optimised SSI

Getting to a basic Gentoo system

Presently, the approach is to CHROOT into the environment. Conversations on the #gentoo-embedded IRC channel from points to using a simple script modified for each environment. There are drawbacks to using such a script and this is not the approach tested/used here. Nonetheless, here is an example of such a script:

ROOT=/tftproot/gentooP3 # Where you want your filesystem to be
CFLAGS="$CFLAGS -O2 -mcpu=pentium3 -pipe -fomit-frame-pointer"
LDFLAGS="$LDFLAGS -L${ROOT}/lib -L${ROOT}/usr/lib"
#Set the building environment if different...
USE="-* netboot sse sse2 blas mpi zlib nls userlocales pbs minimal $USE"
emerge -av $*  

Stage 3 Install

Skip to the stage 3 section of the Gentoo Installation Handbook. We essentially follow the same procedure as a regular Gentoo install. We will use /tftproot/$ARCH_OF_NODE/ as the install dir in our case (where $ARCH_OF_NODE is the actual arch specific for your node, we can have more than one of course).

Code: Quick copy/paste of a typical STAGE 3 root preparation
export ARCH_OF_NODE=AthlonXP
tar -xpjf stage3-*.tar.bz2 -C $GENTOO_INSTALLDIR

#we want to keep the portage trees separated, distfiles can be shared though...
tar -xpjf portage-latest.tar.bz2 -C $GENTOO_INSTALLDIR/usr/ 
cp /etc/resolv.conf $GENTOO_INSTALLDIR/etc/
mount -t proc none $GENTOO_INSTALLDIR/proc/
mount -o bind /sys $GENTOO_INSTALLDIR/sys/

###I have better things to do than re-download packages ;)###
mkdir $GENTOO_INSTALLDIR/usr/portage/distfiles/
mount --bind /usr/portage/distfiles/ $GENTOO_INSTALLDIR/usr/portage/distfiles/

### Don't forget to edit the $GENTOO_INSTALLDIR/etc/make.conf

### We want the same locales on the nodes as the server:
cat /etc/ > $GENTOO_INSTALLDIR/etc/

### Chroot into the environment..
chroot $GENTOO_INSTALLDIR /bin/bash
ln -sf /usr/share/zoneinfo/America/Montreal /etc/localtime
source /etc/profile && env-update
emerge sync

Client kernel configuration

Apart from personnalised hardware support which you must select for your own hardware, the following options must be compiled into your kernel. Don't forget that you will want your network device driver(s) to be BUILT_IN (a * and not an M for module, more than one can be selected).

Linux Kernel Configuration:

Device Drivers --->

   Networking support --->
       Networking options --->
           [*]   IP : kernel level autoconfiguration
           [*]   IP : DHCP support

File systems --->

   Network File System --->
       [*] Root file system on NFS

Specific required packages

Obviously, versions and Masking status may differ.


FORGET the one in portage (as of writing it was to 1.1.4 and it is quite unstable). You are better off with CVS build available at their homepage (version 20060423-1600) coupled with a gentoo-srouces (2.6.16-gentoo-r7) kernel. Once you have downloaded the package, I recommend you untarr it into the chroot environment and build the module from this environment. Note that you cannot just change into the unionfs source directory and type make unless your chroot environment is runnign the exact same kernel version as the system (server) your are building the image from. At the top of the Makefile you will find:

# this should point to where your kernel headers are
KVERS=$(shell uname -r)

You will want to change the KVERS=$(shell uname -r) line with the kernel version you are using for your nodes. Here is an example:


You will also need to add the following entry in the same Makefile:


Once you have set these two variables in th Makefile, you should be able to compile the unionfs module with:


...and installing the module with:

make install

Since we use modprobe in our script to load this module at startup, depmod has to be run manually. Again, if the chroot's kernel is not exactly the same version as the server system your have to tweak the tool's argument with (for example):

depmod -v 2.6.16-gentoo-r7

And, since I really love em, here is a "one-liner" that automagically does all this:

 make clean && make -j3 && make install && depmod -v $(readlink /usr/src/linux | sed -e's:linux-::') -a

sys-fs/unionfs currently does not support kernels >2.6.18, as such you need to either patch the kernel directly using unionfs-2.0, or use sys-fs/aufs (available on gentoo-sunrise). Personally I have had better success with aufs than with unionfs-2.0-u1 during my tests (freenode:_AxS_, 2007-06-26)


You will want at least this version of baselayout as it has better support for unionfs-based systems.


We will necessarily need these since we will be using NFS mounts.

     Latest version available: 1.0.6-r6
     Latest version installed: 1.0.6-r6
     Size of downloaded files: 259 kB
     Description: NFS client and server daemons
     License:     GPL-2

Client side files to add/modify

REMEMBER, all of this is done in the client's CHROOT environment!

Modify /etc/make.conf

Set the CFLAGS and USE flags as you see fit for your environment and hardware (you should have already done this if you started with a Stage1 system). Nothing is preventing you from building more than one SSI ;) Here is an example for stripped down nodes (Beowulf cluster oriented).

File: /etc/make.conf
USE="-* sse sse2 blas mpi zlib nls userlocales pbs minimal unicode"
CFLAGS="-O2 -mcpu=pentium3 -pipe -fomit-frame-pointer"


You can download this version of the script here.
This script is the one called instead of the usual init script (which is called at the end). This script sets up unionfs and sets per-node variables such as hostname. It's a regular BASH script, to you can do as you please with it ;).
Note:Append init=/boot/ softlevel=unionfs to your kernel command line and place this script in your node /boot directory if you haven't already done so.

Don't forget to make executable and to add the mount directories

chmod a+x /boot/
mkdir -p /mnt/unionfs/var
mkdir -p /mnt/unionfs/etc

MYHOST=$(/sbin/dhcpcd -H; /bin/hostname) # By default, we'll use the DHCP assigned hostname


	if [ -z ${MYHOST} ]; then
		echo "DHCP didn't tell me my name. Generating my own hostname..."
	MYHOST="${NODE_NAME}`$IFCONFIG eth0 | awk '/inet addr/ {print $2}' | tr -t . ' ' | awk '{print $4}'`"
		echo DHCP told me my hostname is ${MYHOST}...
		echo "Setting domainname to DHCP's settings"
		/sbin/dhcpcd -D

	echo "STATELESS: Setting Hostname to $MYHOST"
	echo "HOSTNAME=\"$MYHOST\"" > /etc/conf.d/hostname
	/bin/hostname "$MYHOST"

getparams() {
	local cmdline=$(dmesg | grep '^Kernel command line' | sed 's/^Kernel command line://g')
	for pp in $cmdline; do
		echo $pp | grep '^softlevel=' >/dev/null 2>&1
		if [ $? -eq 0 ]; then
			echo $pp | sed 's/softlevel=//g'
			return 0
	echo ""
	return 1

isset() {
	for p in $(getparams | tr ',' ' '); do
		if [ "$p" == "$1" ]; then
			return 0
	return 1

aunionfs() {
	isset unionfs
	if [ $? -eq 0 ]; then
		echo "STATELESS: Loading module unionfs ..."
		$MODPROBE unionfs
		while [ "$1" != "" ]; do
			echo "STATELESS: Mounting tmpfs $1 ..."
			mount -n -t tmpfs -o defaults none /mnt/unionfs/$1
			echo "STATELESS: Mounting $1 unionfs ..."
			mount -n -t unionfs -o dirs=/mnt/unionfs/$1=rw:/$1=ro none /$1
		echo "STATELESS: Not using unionfs as requested ..."

aunionfs etc var
exec /sbin/init


Since we are stateless, we can speed up things a little bit. Note that this forces the node to ALWAYS shutdown (even if you press CTRL-ALT-DEL)!

File: /etc/conf.d/local.stop
# /etc/conf.d/local.stop

# This is a good place to unload any misc.
# programs you started above.
# For example, if you are using OSS and have
# "/usr/local/bin/soundon" above, put
# "/usr/local/bin/soundoff" here.

/sbin/poweroff -f


This is obviously implementation dependant but here is an example. Note that the dump/pass parameters here can have some importance.

File: /etc/fstab (without comments)
# <fs>                  <mountpoint> <type>    <opts>                          <dump/pass>      /           nfs     ro,defaults,hard,intr,rsize=81    0 1                   /home       nfs     rw,defaults,hard,intr,rsize=8192  0 2

# NOTE: The next line is critical for boot!
proc                    /proc           proc            defaults        0 0
shm                     /dev/shm        tmpfs           nodev,nosuid,noexec     0 0

Important server-side files


This file is crucial. It's here that you define many parameters that could change the loaded SSI as well as the kernel being used.
Refer to How do I Configure SYSLINUX?

File: /tftpboot/pxelinux.cfg/default
prompt 1
default ltsp
timeout 50
say Press F1 for boot profiles, default is ltsp in 5 seconds...
F1 BootProfiles

label ltsp
  kernel lts/2.6.9-ltsp-3/bzImage-2.6.9-ltsp-3
  append init=/linuxrc rw root=/dev/ram0 initrd=lts/2.6.9-ltsp-3/initrd-2.6.9-ltsp-3.gz

label Gentoo
  kernel AthlonXP/boot/vmlinuz
  append root=/dev/nfs nfsroot=,hard,intr,rw udev gentoo=nodevfs ip=dhcp init=/boot/ softlevel=unionfs

label Windows
  localboot 0


This file is not essential but is quite useful in a multi-boot environment.

File: /tftpboot/BootProfiles
Type ltsp for regular LTSP bootup (default after 5 seconds)
Type Gentoo for the AthlonXP environment (GENTOO SSI boot dev env)
Type Windows to play worms :)


And finally, the NFS exports:

File: /etc/exports
/home         ,async,no_root_squash)

runlevel scripts

This is not mandatory, we could just use the default runlevel. But here, in the kernel command line, we specify the "softlevel" to be unifonfs. This implies that you create the folder /etc/runlevels/unionfs and you add to this runlevel all apps you want fired up under these conditions. For example:

mkdir -p /etc/runlevels/unionfs
rc-update add local unionfs

We'll be using NFS mounts:

rc-update add portmap unionfs
rc-update add nfs unionfs
rc-update add netmount unionfs

Some other useful services

for I in acpid gmond sshd; do rc-update add $I unionfs; done
Personal tools