PXE Setup with LibVirt and KVM

I’m a heavy user of libVirt on a daily basis. Setting up a VM to test a feature for a service is something I do regularly. Currently I have about 3 different projects simultainously running on my Laptop using libvirt, amassing a total of 10 VMs. Amongst those Windows Server 2k8R2 for an AD+Client(s) 2 nginx webservers and 3 bare Centos server for a networking project. Totalling also around 7 virtual networks.

Setting up these machines back in the day was a huge pain when you had to do it manually. Alas I started investigating into setting up PXE on my machine. Granted, Open Source provided me with a slew of different applications and services I could use to set this up. But most of them didn’t integrate with libVirt. Most frontends to libVirt do not acknowledge the fact that it is actually capable. The best Red Hats own virt-manager could come up with back in the day was bogusly informing me that my setup wasn’t even capable of PXE. Here is how you can do it yourself:

First drop to a commandline and change to the root user and go to your libvirt-configuration:


andreas@kemp: ~ $ su
Passwort: 
root@kemp:~# cd /etc/libvirt/

There you’ll find a directory qemu. If you look around there are XML files defining your hosts:


root@kemp:.../libvirt/qemu# ls -l 
insgesamt 44
-rw------- 1 root root 2002 Nov  4 00:47 ciscoclient2.xml
-rw------- 1 root root 2002 Nov  4 00:47 ciscoclient3.xml
-rw------- 1 root root 2000 Nov  4 00:47 ciscoclient.xml
drwxr-xr-x 3 root root 4096 Nov  3 23:40 networks
-rw------- 1 root root 1991 Okt 29 16:46 nginx2.xml
-rw------- 1 root root 2590 Okt 29 16:46 nginx.xml
-rw------- 1 root root 2108 Okt 15 09:58 WindowsClient.xml
-rw------- 1 root root 2131 Okt 24 02:55 WindowsDomainClient.xml
-rw------- 1 root root 2604 Okt 24 00:03 WindowsServer2008R2-clone.xml
-rw------- 1 root root 2841 Okt 24 12:44 WindowsServer2008R2.xml
root@kemp:.../libvirt/qemu#

And a subdirectory called networks defining your local networks:


root@kemp:.../libvirt/qemu# ls networks/ -l
insgesamt 32
drwxr-xr-x 2 root root 4096 Nov  3 23:34 autostart
-rw------- 1 root root  467 Nov  3 23:34 Cisco01.xml
-rw------- 1 root root  415 Nov  3 23:38 Cisco02.xml
-rw------- 1 root root  415 Nov  3 23:40 Cisco03.xml
-rw-r--r-- 1 root root  281 Okt 23 22:19 default.xml
-rw------- 1 root root  471 Okt 15 00:43 desktops.xml
-rw------- 1 root root  289 Okt 14 12:57 server.xml
-rw------- 1 root root  505 Okt 24 02:04 WindowsServer.xml
root@kemp:.../libvirt/qemu#

Now normally after you’ve created a new VM with the wizard in Virt-Manager you’d have it being dropped into the network “default” where it was in NATed and had gotten an IP via DHCP. The DHCP portion is done using dnsmasq. A small tool that – besides handing out IP adresses – can run a tftp service and provide as a mechanism in our case for automating the installation of HOSTs. For more information check the project homepage: http://www.thekelleys.org.uk/dnsmasq/doc.html

Usually the XML to your network definition looks like this:


root@kemp:~# cat /etc/libvirt/qemu/networks/server.xml 
root@kemp:.../network/iproute2# cat /etc/libvirt/qemu/networks/server.xml 
<network>
  <name>server</name>
  <forward mode='nat'/>
  <bridge name='virbr1' stp='on' delay='0' />
  <mac address='52:54:00:60:E8:AF'/>
  <ip address='10.0.100.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='10.0.100.100' end='10.0.100.200' />
    </dhcp>
  </ip>
</network>
root@kemp:.../network/iproute2#
root@kemp:~#

Which means you have a network of 10.0.100.1/24 to which you can apply hosts which either get their IP address using DHCP or get an IP manually.

Now if we look at the documentation for network definition there is far more we can do with it than Virt-Manager wants us to believe.Such as tftp and bootp, which we can supply with a tftp directory and a bootimage(pxelinux.0 usually) respectively.

So after a while my default network looked like this:


root@kemp:~# cat /etc/libvirt/qemu/networks/default.xml 
<network>
  <name>default</name>
  <bridge name="virbr0" />
  <forward/>
  <ip address="10.0.0.1" netmask="255.255.255.0">
    <tftp root="/srv/tftp"/> 
    <dhcp>
      <range start="10.0.0.2" end="10.0.0.50" />
      <bootp file="pxelinux.0"/>           
    </dhcp>
  </ip>
</network>
root@kemp:~# 

Since that part is taken care of now we can start setting up our tftp directory (/srv/tftp) and our webserver which will host an installer-image of whatever distribution we crave to setup.
First we need the initial boot-image which will allow choosing between options. This is a binary called pxelinux.0 and comes with pxelinux. In Debian you can find it in the syslinux-common package at /usr/lib/syslinux/pxelinux.0. Copy it to the root of your tftp server (/srv/tftp in my case). PXELINUX will not know what to do all by itself it needs a hint which MAC-Adresses it should hańdle how and what the initial Bootoptions are. For this we create the directory pxelinux.cfg and create the file default. In my case it looks like this:


root@kemp:~# cat /srv/tftp/pxelinux.cfg/default 
efault 
default grml32
implicit     0
display      boot.msg
prompt       1
timeout      100
ontimeout    localboot
F1      boot.msg

LABEL centos6
        MENU LABEL CENTOS6 64 bit (Minimal Installation)
        kernel centos6/vmlinuz
        append initrd=centos6/initrd.img ks=http://10.0.0.1/centos.ks

LABEL debian
        MENU LABEL DEBIAN AMD64 (Debian Testing)
        kernel debian/linux
        append auto=true  auto url=http://10.0.0.1/server-base.cfg  priority=critical DEBIAN_FRONTEND=noninteractive install debconf/priority=medium debian-installer/allow_unauthenticated=true vga=788 initrd=debian/initrd.gz -- quiet

LABEL localboot
        MENU LABEL LOCALBOOT
        localboot 0
root@kemp:~# 

With this config I can choose between a Centos 6 and a Debian Testing installation. Both are automated setups using Kickstart and Preseed respectively. The kernel and ramdisk for both installations are hosted on the tftp server. Therefore the directory structure for tftp looks like this:


root@kemp:~# find /srv/tftp/
/srv/tftp/
/srv/tftp/pxelinux.0
/srv/tftp/boot.msg
/srv/tftp/pxelinux.cfg
/srv/tftp/pxelinux.cfg/default
/srv/tftp/chain.c32
/srv/tftp/debian
/srv/tftp/debian/initrd.gz
/srv/tftp/debian/linux
/srv/tftp/centos6
/srv/tftp/centos6/initrd.img
/srv/tftp/centos6/vmlinuz
root@kemp:~# 

Besides the initial ramdisk and kernel for booting the images and the previously explained pxelinux.0 and default this also holds the file chain.32 which is a chainloader and explained in depth on the pxelinux project wiki .
The file boot.msg is a means to present the user with an initial message after pxelinux.0 has been loaded and it is time to choose which operating system to start.

As you can see we also need a webserver for hosting the configuratio for the installers. I host these in an nginx instance listening on port 80 and IP 10.0.0.1 which is the IP of the Host this guest is installed on. The nginx config looks like this:


root@kemp:~# cat /etc/nginx/sites-enabled/default 
server { 
        listen 10.0.0.1:80;
        root /srv/www/;
        location / {
        }
}
root@kemp:~# 

The corresponding directory (/srv/www) holds the config files that are fetched when the image boots. The directory structure looks like this:


root@kemp:~# find /srv/www/
...
/srv/www/config
/srv/www/config/centos.tar
/srv/www/centos.ks
/srv/www/server-base.cfg
...
root@kemp:~#

Where centos.ks is the configuration file for the centos host and server-base.cfg defines the configuration of the debian host. These files look like this:


root@kemp:~# grep -v '\(^#\|^$\)' /srv/www/server-base.cfg 
d-i     debian-installer/locale         string de_DE
d-i	console-keymaps-at/keymap	select de

d-i	mirror/country			string de_DE
d-i	mirror/http/hostname		string ftp.debian.org
d-i	mirror/http/directory		string /debian
base-config	apt-setup/hostname	string ftp.debian.org
base-config	apt-setup/directory	string /debian

d-i	mirror/suite			string testing
#d-i	mirror/http/proxy		string
d-i	partman-auto/disk		string /dev/vda
d-i     partman-auto/choose_recipe select atomic
d-i	partman-auto/method		string regular
d-i	partman/confirm_write_new_label boolean true
d-i	partman-lvm/confirm		boolean false
d-i	partman-lvm/device_remove_lvm	boolean false
d-i	partman-md/device_remove_md	boolean false
d-i	partman/choose_partition	select finish
d-i	partman/confirm			boolean true
d-i	grub-installer/only_debian	boolean true
d-i	finish-install/reboot_in_progress	note

d-i partman-md/device_remove_md	boolean true
d-i partman-partitioning/confirm_write_new_label	boolean true
d-i partman/confirm_nooverwrite	boolean true
d-i partman-lvm/confirm_nooverwrite	boolean true
d-i popularity-contest/participate boolean false

base-config	base-config/intro	note 
base-config	base-config/login	note 
base-config	tzconfig/gmt		boolean true
passwd		passwd/make-user	boolean false
# encryp with: printf "some password" | mkpasswd -s -m md5
passwd		passwd/root-password-crypted password  
base-config	apt-setup/uri_type	select http
base-config	apt-setup/country	select enter information manually
base-config	apt-setup/another	boolean false
base-config	apt-setup/security-updates	boolean true
tasksel		tasksel/first    multiselect standard,ssh-server

console-data console-data/keymap/policy select Don't touch keymap

and this


install
# Mirror URL
url --url http://mirror.netcologne.de/centos/6/os/x86_64/
lang en_US.UTF-8
keyboard de-latin1-nodeadkeys
network --device eth0 --bootproto dhcp
firewall --enabled --ssh
firstboot --disable
authconfig --enableshadow --enablemd5
selinux --enforcing
timezone --utc Europe/Berlin
bootloader --location=mbr --append="console=tty0 console=ttyS0,115200n8r"
key --skip
logging --host=You syslog server
skipx
text
reboot
services --disabled ip6tables
services --enabled ntpd
clearpart --initlabel --all
autopart

# Packages selection.
%packages --nobase
kernel
yum
openssh-server
openssh-clients                                                                                                                                                                                                                                
dhclient                                                                                                                                                                                                                                       
audit                                                                                                                                                                                                                                          
man                                                                                                                                                                                                                                            
logrotate                                                                                                                                                                                                                                      
tmpwatch                                                                                                                                                                                                                                       
vixie-cron                                                                                                                                                                                                                                     
crontabs                                                                                                                                                                                                                                       
system-config-network-tui                                                                                                                                                                                                                      
system-config-firewall-tui                                                                                                                                                                                                                     
ntp                                                                                                                                                                                                                                            
wget                                                                                                                                                                                                                                           
# Remove some stuff we do not need.                                                                                                                                                                                                            
-gnu-efi                                                                                                                                                                                                                                       
-Deployment_Guide-en-US                                                                                                                                                                                                                        
-redhat-release-notes                                                                                                                                                                                                                          
-cryptsetup-luks                                                                                                                                                                                                                               
-hal                                                                                                                                                                                                                                           
-pm-utils                                                                                                                                                                                                                                      
-dbus                                                                                                                                                                                                                                          
-dbus-glib                                                                                                                                                                                                                                     
-kudzu                                                                                                                                                                                                                                         
-ecryptfs-utils                                                                                                                                                                                                                                
                                                                                                                                                                                                                                               
# Run a post script to clean up a bit                                                                                                                                                                                                          
%post                                                                                                                                                                                                                                          
chvt 3                                                                                                                                                                                                                                         
(                                                                                                                                                                                                                                              
echo "Disabling IPv6"                                                                                                                                                                                                                          
sed -i -e 's/\(NETWORKING_IPV6=\).*/\1no/' /etc/sysconfig/network                                                                                                                                                                              

cat <> /etc/modprobe.conf
# disable IPv6
alias net-pf-10 off
EOF
echo "Adding vimrc"
echo -e "syntax on\nhighlight Comment ctermfg=Cyan\nset ts=4" > /root/.vimrc
echo -e '#higher conntrack\noptions ip_conntrack hashsize=32768\n' >> /etc/modprobe.conf
rpm -U http://ftp-stud.hs-esslingen.de/pub/epel/6/i386/epel-release-6-7.noarch.rpm
yum -y install yum-priorities centos-release-cr
wget -q -O /tmp/config.tar http://10.0.0.1/config/centos.tar
yum -y install exim
yum -y remove postfix
chkconfig exim on
tar -C /etc -xf /tmp/config.tar
ulimit -n 8192
yum -y update
chkconfig ntp on
) 2>&1 | tee /root/ks-post.log
chvt 1

After this is done you restart the network with virsh like this:


root@kemp:~# virsh
Welcome to virsh, the virtualization interactive terminal.
virsh # net-destroy default
Network default destroyed

virsh # net-start default
Network default started

virsh # net-list
Name                 State      Autostart
-----------------------------------------
default              active     yes       

virsh # exit
root@kemp:~#

Once you now create a new virtual machine using virt-manager you will be greeted with a prompt like this:

PXE Greeter - Choose your Operating System for installation
Choose your Distribution

Now choose your distribution for the installation and let it run. After the reboot it should present you with a tty login.

You can also find the complete configuration at my Github repository. Feel free to fork and extend it as you wish. I’d also feel delighted if anybody would extend this config to other operating systems and setups.

Happy Hacking!

About these ads

3 comments

  1. Shamim Akhtar · · Reply

    HI,

    This is what i was looking for… will try now..

  2. Shamim Akhtar · · Reply

    Hi,

    my VM getting ip address with kvm network but unable to locate tftp-server file as getting error no file or root path specified.

    KIndly help me to resolve this isssue

    service tftp
    {
    socket_type = dgram
    protocol = udp
    wait = yes
    user = root
    server = /usr/sbin/in.tftpd
    server_args = -s /tftpboot
    disable = no
    per_source = 11
    cps = 100 2
    flags = IPv4
    }
    [root@localhost ~]#

    vim /etc/libvirt/qemu/networks/default.xml

    default
    7acf2586-2e9f-4012-8f0f-a951392faf6a

    [root@localhost ~]# ls /tftpboot/
    images menu.c32 pxelinux.0 pxelinux.cfg

    1. As I said in the post:

      Now normally after you’ve created a new VM with the wizard in Virt-Manager you’d have it being dropped into the network “default” where it was in NATed and had gotten an IP via DHCP. The DHCP portion is done using dnsmasq. A small tool that – besides handing out IP adresses – can run a tftp service and provide as a mechanism in our case for automating the installation of HOSTs. For more information check the project homepage:

      That means you don’t need an extra server for tftp. And here is the libvirt config as written in the following paragraph:

      Usually the XML to your network definition looks like this:


      root@kemp:~# cat /etc/libvirt/qemu/networks/default.xml
      <network>
      <name>default</name>
      <bridge name="virbr0" />
      <forward/>
      <ip address="10.0.0.1" netmask="255.255.255.0">
      <tftp root="/srv/tftp"/>
      <dhcp>
      <range start="10.0.0.2" end="10.0.0.50" />
      <bootp file="pxelinux.0"/>
      </dhcp>
      </ip>
      </network>
      root@kemp:~#

      Just use the one configured server in libvirt using dnsmasq and leave the external tftp-server out.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: