scnr.net

It's not possible? Says who?

Installing XenServer 5.5 on a remote machine (gPXE + HTTP)

with 7 comments

Situation

Imagine you have a remote machine several kilometers/miles away. It is running Linux and you have root access via ssh over the internet. But you can not operate the BIOS screen or any regular installers (because you have no access to keyboard-in or VGA-out). Furthermore, the machine is not configured for booting from network. The currently running Linux installation is no longer needed and you have to install XenServer.

There is a section “PXE installation” in the official manual, but that requires setting up or reconfiguring a DHCP server and a TFTP server. But what if there is no DHCP server in that network or if you have no access to it? Let’s do this with a fixed IP and just an HTTP server.

Precautions

Imagine something goes wrong: Especially if you have to drive all the way to the server or call and pay someone to reboot it, you should test everything on another machine you have physical access to.

Battle Plan: automatic installation

  1. Store the installation data on a HTTP server
  2. Provide the installer with an answer file, so it does not require user input.
  3. Modify the installation image to fix some issues regarding the use of a static IP.
  4. Setup the network bootloader
  5. Reboot the remote machine / start installation

1. Installation data -> HTTP server

Download XenServer-5.5.0-Update2-install-cd.iso and XenServer-5.5.0-Update2-linux-cd.iso. From the images extract the following files and directories to your web server (I’ll use http://cargo-bay.scnr/pxe/xen for my examples):

(ISO 1)/boot/isolinux/mboot.c32  -> (WEBROOT)/pxe/xen/mboot.c32
(ISO 1)/boot/vmlinuz             -> (WEBROOT)/pxe/xen/vmlinuz
(ISO 1)/boot/xen.gz              -> (WEBROOT)/pxe/xen/xen.gz
(ISO 1)/install.img              -> (WEBROOT)/pxe/xen/install.img
(ISO 1)/packages.main/           -> (WEBROOT)/pxe/xen/packages.main/
(ISO 2)/packages.linux/          -> (WEBROOT)/pxe/xen/packages.linux/

2. Answer file

Create an answer file as described in the official documentation, so the installation will automatically start after a successful network boot. Here’s an example:

<?xml version="1.0"?>
    <installation>
        <primary-disk>sda</primary-disk>
        <keymap>us</keymap>
        <root-password>mypass</root-password>
        <source type="url">http://cargo-bay.scnr/pxe/xen/</source>
        <admin-interface name="eth0" proto="static">
            <ip>192.168.0.130</ip>
            <subnet-mask>255.255.255.0</subnet-mask>
            <gateway>192.168.0.1</gateway>
        </admin-interface>
        <nameserver>192.168.0.1</nameserver>
        <hostname>gamma.scnr</hostname>
        <timezone>Europe/Berlin</timezone>
    </installation>

Save it as “(WEBROOT)/pxe/xen/answers.xml”.

3. Installation image modifications

The XenServer installer expects a DHCP server when using an answer file, so we need some additional modifications. For the following steps, I assume you work directly on the webserver, with “(WEBROOT)/pxe/xen/” as current working directory.

The installer needs network settings and has to resolve the IP address of the webserver. Create an “etc/hosts” like this:

127.0.0.1   localhost
192.168.0.1 cargo-bay.scnr

And create an “etc/network/interfaces” with similar content:

auto lo eth0
iface lo inet loopback
iface eth0 inet static
        address 192.168.0.130
        netmask 255.255.255.0
        gateway 192.168.0.1

Somehow “auto lo eth0” doesn’t bring up eth0, and we want to prevent the installer from resetting these settings again. Extract the installer script and modify it:

gunzip -c install.img | cpio -ivd opt/xensource/installer/init
sed -i opt/xensource/installer/init -e '/def main/{G;s/$/    os.system("ifup eth0")/;}'
sed -i opt/xensource/installer/init -e '/init_network =/ s/True/False/'

(There are 4 spaces between “/” and “os.system”. You can open the file and search for “def main” to verify that the following line is correctly indented.)

Then add the new and modified files to a new installer image:

gunzip -c install.img > install-custom.img
find answers.xml etc opt | cpio -ovA -H newc -F install-custom.img
gzip install-custom.img
mv install-custom.img{.gz,}

4. Network bootloader

As I said earlier, the remote machine is not configured for network booting and runs Linux. Instead of trying to use a ROM on the network card, we will use the installed bootloader (Grub in my example). Instead of letting it boot the Linux kernel, let it load gPXE, a feature rich network bootloader.

And it’s very easy to use, thanks to ROM-o-matic.net, where you can generate a custom boot image in just a few clicks. To generate a boot image for loading with grub, select the current generator version on the website, ignore all the options and click the “customize” button. Chose output format “Linux kernel (SYSLINUX/GRUB/LILO) loadable image (.lkrn)”. Make sure the other options fit your needs, but usually the default choices should be fine. Finally go to the “Embedded script” text box and enter a boot script like this:

#!gpxe
ifopen net0
set net0/ip 192.168.0.130
set net0/netmask 255.255.255.0
set net0/gateway 192.168.0.1
set net0/dns 192.168.0.1
set 209:string pxelinux.cfg/default
set 210:string http://cargo-bay.scnr/pxe/xen/
chain http://cargo-bay.scnr/pxe/pxelinux.0
boot

(If you want to know more about these options or booting over HTTP, read this article from the gPXE/etherboot wiki.)

Then click “Get Image”, save it and copy it to the remote machine as “/boot/gpxe-custom.lkrn”. Now tell the bootloader of the remote machine to boot this file instead of the linux kernel. In Ubuntu with Grub2 for example, make sure “default=saved” will be set and add the following lines to “/etc/grub.d/40_custom”:

menuentry "gPXE" {
    linux16 /boot/gpxe-custom.lkrn
    savedefault 0
}

Then run “sudo update-grub”, and “sudo grub-reboot gPXE” (this will boot gPXE once, and switch back to the previous default, so if gPXE fails the next boot will be the old system again). But do not run “sudo reboot” yet.

You’ll also need a recent version of pxelinux.0 (>=3.70 for HTTP support). You can find one at http://www.kernel.org/pub/linux/utils/boot/syslinux/. Take “gpxe/gpxelinux.0” from the archive and save it as “(WEBROOT)/pxe/pxelinux.0” on the web server.

Finally create a config file for pxelinux (similar to the one from the offical XenServer documentation) and store it as “(WEBROOT)/pxe/xen/pxelinux.cfg/default”:

default xenserver-auto
label xenserver-auto
    kernel mboot.c32
    append xen.gz dom0_mem=752M com1=115200,8n1 console=com1,tty \
           --- vmlinuz console=ttyS0,115200n8 console=tty0 \
           answerfile=file:///answers.xml install --- install-custom.img

Do not split the lines with a backslash like I did here – write the append directive in one line.

5. Starting the installation

Everything should be set up now, but let me remind you, test it on a local machine before trying it on an important remote machine.

All that is left to do now is to run “sudo reboot” on the remote machine. If everything works out, the old linux system will shutdown and reboot, grub will load gPXE, which will load PXELinux, which should start the installer. To watch progress, you might want to watch the log and traffic of the webserver and/or ping the remote machine. When the installation finishes and the final reboot is done, you should be able to connect with Citrix XenCenter.

Edit: If the installer fails with an error message after the network boot, verify that all modified files made it to the initrd-archive (if it happens on your test setup, switch to console 2 with Alt-F2 and use cat, grep, etc.). Furthermore: A repository error could indicate a network (configuration) problem (check with ifconfig if your settings were applied).

By the way, you should change the root passwort right after the installation, because the installer got it over unencrypted HTTP as part of the initrd “install-custom.img”.

Alternative Battle Plan: interactive installation

If, for some reason, you want to do an interactive installation instead of using an answer file, extract “/etc/inittab” from “install.img”. Create a custom shell script that sets up networking (“ifup eth0”) and launches an OpenSSH server. In “etc/inittab” replace “/opt/xensource/installer/preinit” with your script name. You also need to add host keys and modify “/etc/passwd” so you can log in. Rebuild install-custom.img and remove the “answerfile”-option from “(WEBROOT)/pxe/xen/pxelinux.cfg/default”.

The remote machine should now boot the installer image, but launch the OpenSSH server instead of the installer. Connect and launch “/opt/xensource/installer/preinit” manually to run the installer over SSH. But be careful not to reset your network settings during the installation. Again: Test it on a local machine first.

Written by johnLate

April 24th, 2010 at 5:30 pm

Posted in Uncategorized

7 Responses to 'Installing XenServer 5.5 on a remote machine (gPXE + HTTP)'

Subscribe to comments with RSS or TrackBack to 'Installing XenServer 5.5 on a remote machine (gPXE + HTTP)'.

  1. Hey! Thank you so much for this info!!!

    Thanks to you I’m installing Xen Server 5.5 into a 1and1 server 🙂

    One problem I found was the boot hanging when loading gpxelinux.0 from syslinux, tried using pxelinux.0 instead and it worked like a charm!.

    Thanks again!

    Guzman

    1 May 10 at 23:02

  2. Well, I was wrong, I did not suceed. I got it to start the installation but when it fails to load the repository from the remote http server.

    Just fails saying a repository is not there when monitoring show it’s not even trying.

    I know it’s not related to your instructions because the same happens even if I load the install.img without changing it and try to configure it by hand through serial port….

    Now trying different versions of syslinux gpxe/gpxelinux.0 to check if the problem is not caused by me using syslinux 3.86 pxelinux.0 instead of gpxelinux.0. gpxelinux.0 from syslinux 3.86 and 3.80 both hangs the gpxe image after getting the file from the server.

    Wish me luck

    Guzman

    2 May 10 at 08:41

  3. At the end it didn’t work out. Going now for Xen 4 over Lenny.

    The problem I had was with the loading of networking inside Xen installation.

    However I was unable to even start the installation with the custom-image doing what you said. It seems the problem was the way cpio update the files: it doesn’t. It appends the new one keeping also the old one (!), tried doing the same with pax also failing. Tried help on #xenxen also failed. So I’m going for Xen 4 over Lenny.

    Guzman

    3 May 10 at 01:50

  4. @Guzman:

    Yes, the repository error is usually caused by bad network settings.

    About the cpio update: It’s intended to just append the modified files to the original image. When I tested it and booted with such a initrd-archive, I could only access the newer/appended version of a file, even if two versions were in the archive.

    You could try to extract the whole original image, make your modifications and then create a new image instead of appending to the original one.

    Hint for debugging: if on your testing machine the installer fails with an error message after the network boot, switch to the 2nd console and verify the contents of all modified files… that way you quickly find out whether you booted with the correct initrd-image, whether all files made it to the image and so on…

    johnLate

    3 May 10 at 04:35

  5. nice work!!!!
    but it’s won’t work on my mashines….
    firstly its pretty hard to compile the network-kernel, cause this requeres know-how
    my bootloader wont load dis f***ing kernel, or it loads but unable to connect to webspace (other server)

    however… [why is the webroot path: ../pxe/xen/.. ???
    we have not anymore pxe requried, or?]

    @answer file ur gateway ip similar to nameserver, okay…
    but why is the ip similar to your IP @ /etc/hosts ???

    Acid_Flash

    8 Sep 10 at 14:42

  6. It’s pretty hard to compile the network kernel? I’m not sure what you are talking about. If you mean gPXE, it’s a matter of reading the documentation and using the online generator. I wouldn’t try to download the source and to build it by myself.

    bootloader/webpath: Well, the packages.* directories will be accessed from the Xen Installer; but the kernel, the initrd, etc. will be accessed from gPXE, copied into RAM and /then/ the installer-kernel will be started. But it doesn’t really matter how you structure your web directory as long as you adapt the configuration files accordingly.

    etc/hosts: It just contains an entry for the server hosting the files. You can of course add an entry for the machine itself.

    johnLate

    8 Sep 10 at 17:01

  7. I don’t usually post comments on blogs, but this was incredibly helpful. I managed to use this tutorial to install XenServer 6.0.2 remotely with gPXE. Like the poster above me, I have a server at 1and1 internet. They support DHCP so the static IP addressing is not required, but they use a special MAC filtering and your IP configuration is something like:

    IP: 108.1.2.3
    Mask: 255.255.255.255
    Gateway: 10.255.255.1
    DNS: 10.255.255.3

    Like the poster above, my boot would hang indefinitely. But it was because it was timing out trying to pull down the answer file. 1and1 also supports a 80×25 text VGA serial console accessible via SSH which comes in handy here.

    Because we have DHCP, but a weird network configuration, it takes a little tweaking the above to get a network connection. Here’s what I had to change:

    I downloaded the 6.0.2 ISO and loop mounted it to ~WEBROOT/xen/ (mount -o loop /tmp/file.iso ~WEBROOT/xen/)
    I copied mboot.c32, vmlinuz, xen.gz and install.img to ~WEBROOT/pxe/
    My answer file looked like:

    sda
    us
    mypass
    http://host.name/xen/

    host.name
    America/Chicago

    For the network settings in the install image, I used the same static configuration, plugging my values in instead. I also made the same changes to the init script, except I added a few os.system() calls:

    os.system(“modprobe e1000e”)
    os.system(“ifup eth0”)
    os.system(“sleep 30”)
    os.system(“route add 10.255.255.1/32 dev eth0”)
    os.system(“route add default gw 10.255.255.1”)

    Your system might use a different module. I found mine by using the minimal install already on the machine and “dmesg|grep eth0”. The route lines are the real magic. Without them added manually, DHCP nor static IP’ing was applying them properly on their own. Creating a static route to the gateway IP first (the first route add command) is required apparently.

    I also took the time to extract opt/xensource/installer/constants.py and change GPT_SUPPORT to false and increase root_size. Disabling GPT lets me use the software RAID-1 1and1 provides me.

    For gPXE, DHCP with the weird network config worked fine with version 1.0.1, so I used the script:

    #!gpxe
    echo Performing DHCP on first network interface
    dhcp net0
    set 209:string pxelinux.cfg/default
    set 210:string http://host.name/pxe/
    chain ${210:string}pxelinux.0
    boot

    When creating the gPXE image, I used version 1.0.1 and also enabled CONSOLE_SERIAL and set the speed to 57600 and disabled the wifi authentication options at the bottom (left the other defaults).

    For pxelinux.0, I used gpxe/gpxelinux.0 from 4.05 of syslinux. 3.70 complained about not finding the device. I put pxelinux.0 and pxelinux.cfg/default along with install-custom.img within my ~WEBROOT/pxe/ directory.

    My pxelinux.cfg/default looked like:

    default xenserver-auto
    label xenserver-auto
    kernel mboot.c32
    append xen.gz dom0_max_vcpus=1 dom0_mem=752M com1=57600,8n1 console=com1,vga — vmlinuz xencons=hvc console=hvc0 console=tty0 answerfile=file:///answers.xml install — install-custom.img

    From here, you can use the SSH-based serial console to watch the boot and install. You can also remove the answers bit and perform a manual install. With a little ingenuity, you could PXE install just about anything with a serial text interface or automated installer.

    I hope my additions are helpful to someone who might run into the same problems I had. But kudos to John for the tutorial. I never would have figured out gPXE otherwise.

    Scott

    3 May 12 at 15:34

Leave a Reply