Xen PXE Boot Howto

v1.5 - 2010-04-21 little update

v1.4 - 2009-11-24 update for Oracle VM 2.2

v1.3 - 2008-09-22 update for Oracle VM 2.1.2

v1.2 - 2008-02-28 add trouble shootting

v1.1 - 2008-01-17 little errata

v1.0 - 2007-09-30 initial version


Table of Contents

1. Introduction
2. Setup PXE boot environment
3. Testing
4. Troubleshooting
5. Reference

1. Introduction

In this article, we'll show you the prcesses to setup PXE boot environment for Xen host (hypervisor + dom0) and Xen guests: both PV (Para-virtualized) guest and HVM (Hardware-assisted Virtual Machine).

Testing environment:

  • DHCP and TFTP server - Oracle Enterprise Linux 5 x86.

  • Xen server - An VT enabled host with Oracle VM Server 2.2 installed.

  • PV/HVM guest - Oracle Enterprise Linux 5.4 x86 installation.

2. Setup PXE boot environment

Follow these steps to get it working.

2.1. Setup DHCP server

  1. Create DHCP server configure file /etc/dhcpd.conf with contents:

    ddns-update-style none;
    allow booting; # support PXE booting
    allow bootp; # respond to bootp queries
    
    subnet 192.168.1.0 netmask 255.255.255.0 {
        option routers             192.168.1.166;
        option subnet-mask         255.255.255.0;
        option nis-domain          "example.com";
        option domain-name         "example.com";
        option domain-name-servers 192.168.1.166;
        option time-offset         -18000; # Eastern Standard Time
    
        default-lease-time 120;
        max-lease-time 120;
    
        next-server 192.168.1.166;
        filename "pxelinux.0";
    
        pool {
            range dynamic-bootp 192.168.1.199 192.168.1.254;
        }
    
        group {
            host test_pvm {
                hardware ethernet 00:16:3E:72:79:E3;
                fixed-address 192.168.1.199;
            }
            host test_hvm {
                hardware ethernet 00:16:3E:72:79:E4;
                fixed-address 192.168.1.177;
            }
            host dell_optiplex_745 {
                hardware ethernet 00:19:B9:41:2C:B9;
                fixed-address 192.168.1.188;
            }
        }
    }
  2. Restart DHCP server:

    # service dhcpd restart

2.2. Setup TFTP server

  1. TFTP server is managed by xinetd server and is off by default. To enable TFTP service, edit /etc/xinetd.d/tftp and change the following line:

    disable = yes

    to:

    disable = no
  2. Restart xinetd server:

    # service xinetd restart

2.3. Prepare files

  1. Prepare pxeboot files:

    # cp /usr/lib/syslinux/pxelinux.0 /tftpboot/
    # cp /usr/lib/syslinux/mboot.c32 /tftpboot/

    Note

    "mboot.c32" is added in syslinux-3.08.

  2. For EL5 x86 HVM guest pxeboot installation:

    # mount -o loop Enterprise-R5-U4-Server-x86_64-dvd.iso /mnt/loop
    # mkdir /tftpboot/el5u4_x86_pxeboot_install/
    # cp -a /mnt/loop/images/pxeboot/vmlinuz /tftpboot/el5u4_x86_pxeboot_install/
    # cp -a /mnt/loop/images/pxeboot/initrd.img /tftpboot/el5u4_x86_pxeboot_install/
  3. For EL5 x86 PV guest pxeboot installation:

    # mkdir /tftpboot/el5u4_x86_xen_install/
    # cp -a /mnt/loop/images/xen/vmlinuz /tftpboot/el5u4_x86_xen_install/
    # cp -a /mnt/loop/images/xen/initrd.img /tftpboot/el5u4_x86_xen_install/
    # umount /mnt/loop
  4. For Xen server PXE boot, copy files under /boot of Xen server to TFTP server. On TFTP server, do the followings:

    # mkdir -p /tftpboot/ovm/2.2/
    # scp xenserver:/boot/xen-64bit-3.4.0.gz /tftpboot/ovm/2.2/
    # scp xenserver:/boot/vmlinuz-2.6.18-128.2.1.4.9.el5xen /tftpboot/ovm/2.2/
    # scp xenserver:/boot/initrd-2.6.18-128.2.1.4.9.el5xen.img /tftpboot/ovm/2.2/
  5. Create configure file /tftpboot/pxelinux.cfg/01-00-16-3e-72-79-e3 for PV guest PXE installation:

    default linux
    prompt 1
    timeout 120
    label linux
      kernel el5u4_x86_xen_install/vmlinuz
      append initrd=el5u4_x86_xen_install/initrd.img

    Note

    For detail about pxelinux configure file search path, please refer to pxelinux desgin documents: /usr/share/doc/syslinux-3.11/pxelinux.doc.

  6. Create configure file /tftpboot/pxelinux.cfg/01-00-16-3e-72-79-e4 fro HVM guest PXE installation:

    default linux
    prompt 1
    timeout 120
    label linux
      kernel el5u4_x86_pxeboot_install/vmlinuz
      append initrd=el5u4_x86_pxeboot_install/initrd.img
  7. Create configure file /tftpboot/pxelinux.cfg/01-00-19-b9-41-2c-b9 for Xen server PXE boot:

    default ovm
    prompt 1
    timeout 120
    label ovm
      kernel mboot.c32
      append ovm/2.2/xen-64bit-3.4.0.gz noreboot --- ovm/2.2/vmlinuz-2.6.18-128.2.1.4.9.el5xen ro root=LABEL=/ --- ovm/2.2/initrd-2.6.18-128.2.1.4.9.el5xen.img

    The parameters are copied from /boot/grub/grub.conf of Xen server.

  8. Make sure all files and directories have the right permission:

    # find /tftpboot/ -type f -exec chmod 644 {} \;
    # find /tftpboot/ -type d -exec chmod 755 {} \;

3. Testing

3.1. Test Xen server PXE boot

  1. Reboot Xen server and select to boot from network.

    Note

    Some host using hotkey "F12" to get into the boot methods selection screen, while others need to enter system setup to change boot order.

  2. You'll see Xen server host will get IP 192.168.1.188 and download files according configure file /tftpboot/pxelinux.cfg/01-00-19-b9-41-2c-b9. Then it will boot the system just the same as grub.

3.2. Test HVM guest installation

  1. Create VM configure file hvm.cfg with contents:

    name = 'xen_el5u4_x86_hvm'
    builder = 'hvm'
    memory = 512
    disk = [ 'file:/share/vm/pxeboot/system.img,hda,w', ]
    vif = [ 'type=ioemu,mac=00:16:3E:72:79:E4,bridge=xenbr0', ]
    kernel = '/usr/lib/xen/boot/hvmloader'
    device_model = '/usr/lib/xen/bin/qemu-dm'
    vnc = 1
    vncunused = 1
    vnclisten = '0.0.0.0'
    apic = 1
    acpi = 1
    pae = 1
    boot = 'n'
    vcpus = 1
    serial = 'pty'
    on_reboot = 'restart'
    on_crash = 'restart'

    Note

    boot = 'n' means boot from network.

  2. Start the virtual machine:

    # xm create hvm.cfg
  3. Get the VNC port of the VM by:

    # xenstore-read /local/domain/`xm domid xen_el5u4_x86_hvm`/console/vnc-port
    5900

    Connect to VM's graphical console by:

    $ vncviewer xenserver:5900
  4. You'll see the EL5 installation screen come up just the same as normal installation from CD.

3.3. Test PV guest installation

  1. Install pypxeboot first (see reference).

  2. Create VM configure file pvm.cfg with contents:

    name = 'xen_el5u4_x86_pvm'
    memory = 512
    disk = [ 'file:/share/vm/pxeboot/system.img,xvda,w', ]
    vif = [ 'mac=00:16:3e:72:79:e3,bridge=xenbr0', ]
    vfb = [ 'type=vnc,vncunused=1,vnclisten=0.0.0.0', ]
    #bootloader = '/usr/bin/pygrub'
    bootloader = '/usr/bin/pypxeboot'
    bootargs = vif[0]
    vcpus = 1
    on_reboot = 'restart'
    on_crash = 'restart'
  3. Start the virtual machine:

    # xm create -c pvm.cfg
    Using config file "./pvm.cfg".
    pypxeboot: requesting info for MAC address 00:16:3e:72:79:e3
    pypxeboot: getting cfg for IP 192.168.1.199 (C0A801C7) from server 192.168.1.166
    pypxeboot: downloading initrd using cmd: tftp 192.168.1.166 -c get /el5u4_x86_xen_install/initrd.img /var/lib/xen/initrd.TcwYCz
    pypxeboot: downloading kernel using cmd: tftp 192.168.1.166 -c get /el5u4_x86_xen_install/vmlinuz /var/lib/xen/kernel.eW9kO9
    Started domain xen_el5u4_x86_pvm (id=3)
    Linux version 2.6.18-164.el5xen (mockbuild@ca-build10.us.oracle.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Thu Sep 3 02:41:56 EDT 2009
    BIOS-provided physical RAM map:
     Xen: 0000000000000000 - 0000000020800000 (usable)
    0MB HIGHMEM available.
    520MB LOWMEM available.
    ...
  4. Get the VNC port of the VM by:

    # xenstore-read /local/domain/`xm domid xen_el5u4_x86_pvm`/console/vnc-port
    5901

    Connect to VM's graphical console by:

    $ vncviewer xenserver:5901
  5. You'll see the EL5 installation screen come up just the same as normal installation from CD.

4. Troubleshooting

4.1. HVM guest boot failed with "No IP address"?
4.2. PV guest cannot boot with "Error: Boot loader didn't return any data!"?
4.3. How does pypxeboot find the configuration file?

4.1.

HVM guest boot failed with "No IP address"?

As of version 3.0.3 of the ISC DHCP server, the value of “ next-server” does not have a reasonable default any more (it used to contain the IP address of the DHCP server, which would act as TFTP server then). You will have to set it to the IP address of the TFTP server (in most cases, the same machine as the DHCP server):

next-server 192.168.0.1;

in your dhcpd.conf, somewhere in the global section near the top, will do the trick.

Please refer to: Etherboot troubleshooting.

4.2.

PV guest cannot boot with "Error: Boot loader didn't return any data!"?

You can use the following steps to debug this issue:

  1. You should first make sure the PXE netboot environment is working. You can confirm it by pxe booting a HVM guest.

  2. Then make sure there is only one item in the configure file. Pypxeboot doesn't parse multi-boot items well at this time.

  3. Make sure the TFTP server is accessable in dom0. You can confirm it by downloading files from the TFTP server using tftp.

  4. You should not setup DHCP server on the same dom0.

  5. pypxeboot leverage udhcpc to get the IP of a given MAC address. Currently there is a limitation: pypxeboot will always call udhcpc with

    --interface=eth0

    and this behaviour is not configurable. So you should make sure the DHCP server is running on the eth0 connected LAN. Or you can write a wrapper.

    First rename /sbin/udhcpc:

    mv /sbin/udhcpc /sbin/udhcpc.orig

    Then create a shell script with contents:

    #!/bin/sh
    /sbin/udhcpc.orig --interface=eth1 $@

    And don't forget to add execution permission:

    # chmod +x /sbin/udhcpc

  6. If eth0 (or eth1 after the above step) is a in a bridge, that will work. If not, you should make sure eth0 in promiscuous mode:

     # ifconfig eth0 promisc

4.3.

How does pypxeboot find the configuration file?

It is nearly full compatible to PXELINUX.

5. Reference