Installing Xen on Gentoo



I have a PC with an AMD X2 4200 CPU which is already running Gentoo-Linux with a 2.6.17 kernel. I decided to see if with that box I am able to let WindowsXP run on top of Xen. If this works, I'll be able to use the Windows programs that don't work with the wine emulator under Linux, and run all "dangerous" programs isolated in a separate environment. I will use the same installation that runs the normal 2.6.17 kernel and install additionally a xen-sources-2.6.16 kernel and its modules.

Index:

  1. Preparing the dom0 kernel
  2. Booting the dom0 kernel
  3. Preparing the domU environment
  4. Trying out the virtualization
  5. Details...
  6. Screenshots
  7. Problems
  8. Reference docs

Preparing the dom0 kernel

  • First of all I checked, just to be sure, that the CPU has virtualization built-in: 
    grep pae /proc/cpuinfo
grep svm /proc/cpuinfo
I think (hope), those are the right flags to search for. Use vmx instead of svm if you're using an Intel processor.
  • Afterwards I updated portage, so that i would have an up-to-date list of packages:
    emerge --sync
  • The Gentoo profile that I am using is:
    [1] default-linux/amd64/2006.1 *
  • I did not modify my CFLAGS to insert the flag -mno-tls-direct-seg-refs, as the CPU is 64bit (described by the guide). My CFLAGS still look like this:
    CFLAGS="-march=athlon64 -O2 -pipe"
  • Checked that I have MAKEOPTS set to 3 (set it ot 3, if you have a dual core CPU):
    # grep MAKEOPTS /etc/make.conf
    MAKEOPTS="-j3"
  • After adding the nptlonly and sdl USE flags to /etc/make.conf I recompiled glibc...
    emerge glibc
..which took 45 minutes.
  • I emerged the usermode-utilities, and inserted into /etc/portage/package.keywords the strings: 
    app-emulation/xen
app-emulation/xen-tools
sys-kernel/xen-sources
sys-devel/dev86
  • IMPORTANT! After getting the kernel of xen-sources, don't delete the old one nor the kernel in the boot directory nor the configuration from grub.conf. I mention the reason in the section about booting a guest OS.
  • I emerged the Xen kernel and the tools:
    emerge -pv xen xen-tools xen-sources
  • The emerge of iproute2 crashed with the following error message: 
    make[1]: Leaving directory `/var/tmp/portage/sys-apps/iproute2-2.6.19.20061214/work/iproute2-2.6.19-061214/tc'
make: *** [all] Error 2

!!! ERROR: sys-apps/iproute2-2.6.19.20061214 failed.
Call stack:
  ebuild.sh, line 1615:   Called dyn_compile
  ebuild.sh, line 972:   Called qa_call 'src_compile'
  ebuild.sh, line 44:   Called src_compile
  iproute2-2.6.19.20061214.ebuild, line 69:   Called die
  • I'll try first to update the system, before searching a solution for the above error:
    emerge --update --deep --newuse world
  • It worked! iproute2 compiled without problems.
  • I didn't use the buildpkg or quickpkg functionality nor I rebuilt glibc as when I updated it, the nptlonly flag was already active.
  • I already had the sdl flag in make.conf active before emerging libsdl, and I emerged xen and xen-tools after emerging libsdl, so, nothing to do here (didn't activate the pae flag, as I'm already running on a 64 bit CPU).
  • Just remembered that I emerged Xen => had a look at the files in /boot/ and found the various xen*.gz files.
  • I didn't remove /etc/init.d/net.eth0 and I didn't add /etc/init.d/xend from the default runlevel, as I am still testing the whole thing - don't want Xen to be started automatically, and want the PC to be accessed remotely.
    rc-update del net.eth0 default
  • I disabled the automatic loading of net.eth0 in /etc/conf.d/rc:
    RC_PLUG_SERVICES="!net.*"
  • At this point the guide says:
    The domain 0 kernel should contain drivers for Xen backend devices, and all of your usual hardware. That is, the dom0 configuration should enable all the options for backend drivers and disable all options for building in the frontend drivers.
Don't really understand it - I will meditate for one more day... .
  • So, I decided to follow first of all the statement The domain 0 kernel should contain drivers for Xen backend devices, and all of your usual hardware. I therefore setup the Xen0-kernel with the usual options I have in the classical kernel (at the beginning almost EVERYTHING was selected as a module - I needed 40 minutes to uncheck those entries...), plus the options mentioned in the Wiki for the Xen0-kernel plus the stuff I wasn't sure about. Now, under XEN, the configuration looks like this: 
    [*] Privileged Guest (domain 0)
<*> Backend driver support
<*>   Block-device backend driver
< >   Block-device tap backend driver
<*>   Network-device backend driver
[ ]     Pipelined transmitter (DANGEROUS)
<*>     Network-device loopback driver
<*>   PCI-device backend driver
        PCI Backend Mode (Virtual PCI)
[ ]     PCI Backend Debugging
< >   TPM-device backend driver
< > Block-device frontend driver
< > Network-device frontend driver
< > Framebuffer-device frontend driver
[*] Scrub memory before freeing it to Xen
[ ] Disable serial port drivers
<*> Export Xen attributes in sysfs
    Xen version compatibility (3.0.2 and later)
Building the kernel:
make && make modules_install
  • Error! While building the kernel I got the error messages... 
      CC [M]  net/ipv4/netfilter/ipt_REJECT.o
net/ipv4/netfilter/ipt_REJECT.c: In function 'send_reset':
net/ipv4/netfilter/ipt_REJECT.c:162: error: 'struct skb_shared_info' has no member named 'tso_size'
net/ipv4/netfilter/ipt_REJECT.c:163: error: 'struct skb_shared_info' has no member named 'tso_segs'
...and found in the Gentoo Bugzilla the solution... 
In line 162 and 163 of file net/ipv4/netfilter/ipt_REJECT.c
change:
skb_shinfo(nskb)->tso_size = 0;
skb_shinfo(nskb)->tso_segs = 0;

to:
skb_shinfo(nskb)->gso_size = 0;
skb_shinfo(nskb)->gso_segs = 0;
...and reran the compilation.
  • Error! Again!! This time I got... 
      CC [M]  net/sctp/input.o
net/sctp/input.c: In function 'sctp_rcv':
net/sctp/input.c:137: error: too many arguments to function 'skb_linearize'
..and I therefore deactivated in the kernel configuration the option Networking => Networking support => Networking options => SCTP Configuration => The SCTP Protocol.
  • It worked! The kernel and its modules are ready.
  • I copied the brand new kernel to the boot partition:
    cp -i vmlinuz /boot/vmlinuz-2.6.16.49-xen0
  • I added into Grub the following lines: 
    title=Xen 3.0 Linux 2.6.16.49
root   (hd0,0)
#I am using Xen 3, so I don't need to specify the amount of memory that has to be allocated for the domain 0.
#kernel /xen.gz dom0_mem=98M
kernel /xen.gz
module /vmlinuz-2.6.16.49-xen0 noapic apm=off acpi=off vga=785 root=/dev/sda3
  • I added loop and tun to /etc/modules.autoload.d/kernel-2.6

Booting the dom0 kernel

  • I rebooted successfully the machine - the domain0 is now at least able to boot. But the network is down :o(
  • I am not using the onboard network card as it is not supported by the 2.6.16 kernel but instead I'm using a PCI Realtek 8169 card. I therefore changed in the config file /etc/xen/xend-config.sxp the line...
    (network-script network-bridge)
...to...
(network-script 'network-bridge netdev=eth1')
Now with ifconfig I see the virtual devices and not even a ping is working, but at least I can startup the network with...
/etc/init.d/net.eth1 start
Don't know if this is correct for the configuration of a domain0, but at least the remote ssh access is working. Actually, I reactivate the Gentoo network loading in /etc/conf.d/rc:
RC_PLUG_SERVICES="net.*"
  • Now I will check if the virtualization was detected:
    # xm dmesg | grep SVM 
    (XEN) AMD SVM Extension is enabled for cpu 0.
(XEN) AMD SVM Extension is enabled for cpu 1.
(replace SVM with VMXON if you have an Intel processor)
  • xentop seems to be happy as well.
  • I emerged nvidia-drivers. Don't do it, or your PC will hang during the emerge (when the system will try to modprobe the driver) and during all successive reboots :o( . Here they say that Xen does not support the nvidia drivers yet.

Preparing the domU environment

(A domU kernel is not needed in order to run Windows, but it is still worth reading this section, not only if you don't intend to install any other OS)

  • I changed in /etc/X11/xorg.conf the parameter...
    Driver "nvidia"
...to...
Driver "nv"
...so that the open source nVidia drivers are loaded and reemerged...
emerge xorg-server
  • Don't forget to put into /etc/make.conf the lines...
    INPUT_DEVICES="keyboard mouse"
...if you're upgrading to Xorg 7.2, otherwise your mouse pointer/cursor will be hidden. Needed 1.5 hours to find this out. If after recompiling xorg-server it still doesn't work, emerge xcursorgen and xcursor-themes.
  • Started up X - worked.
  • Surprise! The PC is hanging! I can still move the mouse, write something in the shell, but all activities involving the access to the harddisk hang. Every x minutes it does something for 1 second, (so some of the commands I typed in get executed) but otherwise it looks like it is more or less hanging.
  • I tried out a lot of stuff, and in the end I think that it hangs a few seconds after I access the network to download files or connecting through SSH. I therefore took out the network card (Netgear Gigabit with RealTek 8169 chip) and replaced it with another one (D-Link Megabit with RealTek RTL8139). In the kernel I selected to compile the driver as module and selected the option "Use PIO instead of MMIO").
  • Compiled, installed. I tried to download a few files using FTP and it looks like it wasn't hanging anymore :o). Interesting to see that I had to link the new network card to eth3 and not anymore to eth0. Strange.
    ln -s /etc/init.d/net /etc/init.d/net.eth3
  • I copied with cp -R the whole directory /usr/src/linux-2.6.16.49-xen to a new one called /usr/src/linux-2.6.16.49-xen-udom and separately as well the configuration file:
    cp /usr/src/linux-2.6.16.49-xen/.config /usr/src/linux-2.6.16.49-xen-udom/
  • Went to /usr/src/linux-2.6.16.49-xen-udom/ and set in the kernel configuration under XEN the options as follows: 
    [ ] Privileged Guest (domain 0)
< > Backend driver support
<*> Block-device frontend driver
<*> Network-device frontend driver
<*> Framebuffer-device frontend driver
<*>   Keyboard-device frontend driver
[*] Scrub memory before freeing it to Xen
[ ] Disable serial port drivers
<*> Export Xen attributes in sysfs
     Xen version compatibility (3.0.2 and later)
  • Built the kernel (use only make, and no make modules_install), copied the new domU kernel to the boot directory:
    cp /usr/src/linux-2.6.16.49-xen-udom/vmlinuz /boot/vmlinuz-2.6.16.49-xenU
  • Didn't need it, but I rebooted the machine - just wanted to see if the network was really ok.
  • Everything looks fine - tried to download a few files.
  • Changed in /etc/xen/xen-config.sxp the line...
    (network-script network-bridge)
...to...
(network-script 'network-bridge netdev=eth3')
...and restarted the machine.
  • Tried to download again the same file as before - it was hanging again - segmentation fault. I therefore rolled back the last changes in /etc/xen/xen-config.sxp => rebooted => managed this time to download 60MB of data - still hanging.
  • Had a look at proc/interrupts: 
               CPU0       CPU1       
  1:         72          0        Phys-irq  i8042
  8:          0          0        Phys-irq  rtc
 10:       5511          0        Phys-irq  libata, ehci_hcd:usb1, ohci_hcd:usb2, peth3
 12:         96          0        Phys-irq  i8042
 14:         37          0        Phys-irq  ide0
256:       7983          0     Dynamic-irq  timer0
257:      12230          0     Dynamic-irq  resched0
258:          6          0     Dynamic-irq  callfunc0
259:          0       1620     Dynamic-irq  resched1
260:          0        131     Dynamic-irq  callfunc1
261:          0       2665     Dynamic-irq  timer1
262:        139          0     Dynamic-irq  xenbus
263:          0          0     Dynamic-irq  console
NMI:          0          0 
LOC:          0          0 
ERR:          0
MIS:          0
Looks like the IRQ 10 is shared by libata, USB and the network. This might be the source of the problem. And now?
  • Found this which describes a problem with IRQs and a fix for the Xen-kernel. I checked and the fix is already integrated in my kernel. Therefore, I decided to activate ACPI and deleted the command acpi=off from the boot like of /boot/grub/grub.conf. Rebooting... .
  • Now cat /proc/interrupts looks like this: 
               CPU0       CPU1       
  1:        102          0        Phys-irq  i8042
  8:          0          0        Phys-irq  rtc
  9:          0          0        Phys-irq  acpi
 12:        448          0        Phys-irq  i8042
 14:         37          0        Phys-irq  ide0
 16:       5675          0        Phys-irq  libata
 17:         79          0        Phys-irq  ehci_hcd:usb1
 18:          0          0        Phys-irq  ohci_hcd:usb2
 19:        109          0        Phys-irq  peth3
256:      12850          0     Dynamic-irq  timer0
257:       2054          0     Dynamic-irq  resched0
258:          5          0     Dynamic-irq  callfunc0
259:          0       2363     Dynamic-irq  resched1
260:          0        136     Dynamic-irq  callfunc1
261:          0       9687     Dynamic-irq  timer1
262:        139          0     Dynamic-irq  xenbus
263:          0          0     Dynamic-irq  console
NMI:          0          0 
LOC:          0          0 
ERR:          0
MIS:          0
Now libata, USB and the network all use different IRQs. Hope it will work this time... .
  • It worked. I reinstalled the RealTek RTL8139 Gigabit network card and after activating Use Rx and Tx Polling (NAPI) (EXPERIMENTAL) the transfer rate was good.

Trying out the virtualization

  • I synch'ed the domU kernel settings with the changes I did for the network card in the dom0 kernel and recompiled it (only make without && make modules_install).
  • At this point I saw in the other docs that I needed qemu. Mmmhh... .
  • Now, do the following:
    • Reboot the PC and use this time not the Xen kernel, but the "normal" one. I didn't manage to the package mentioned in the next step while I was using the normal kernel.
    • Check with gcc-config -l that you're using gcc 3.x.x. This because qemu won't compile if you're using a gcc 4.x.x. If yes, select it. If not, emerge it with emerge -v =sys-devel/gcc-3.x.x and select it with gcc-config x86_64-pc-linux-gnu-3.x.x.
    • Last but not least, check that in /usr/src, your linux softlink points to the Xen kernel source files:
      linux -> linux-2.6.16.49-xen
    • Now that I had the correct 3.x.x version of gcc, I tried to emerge qemu again, but while emerging one of the dependencies, qemu-softmmu, I got: 
      Exiting on signal 2
make[1]: *** [vnc.o] Interrupt
make: *** [subdir-i386-softmmu] Interrupt
make[1]: *** [slirp/tcp_input.o] Interrupt
make: *** [subdir-ppc-softmmu] Interrupt
make[1]: *** [fpu/softfloat.o] Interrupt
make: *** [subdir-sparc-softmmu] Interrupt
sandbox:  Signal already caught and busy still cleaning up!

/usr/portage/app-emulation/qemu-softmmu/qemu-softmmu-0.9.0.ebuild: src_compile aborted; exiting.
    At this point, as described by the guides, I changed in /etc/make.conf the line...
    CFLAGS="-march=athlon64 -O2 -pipe"
    ...to...
    CFLAGS="-mtune=athlon64 -O2 -pipe"
    ...and the package compiled with no problems.
  • Afterwards I went on emerging the remaining two packages, qemu-user and qemu. Done.
  • don't forget to set back...
    CFLAGS="-mtune=athlon64 -O2 -pipe"
...to...
CFLAGS="-march=athlon64 -O2 -pipe"
..and as well gcc-config to your original compiler, if you changed it.
  • Mmmhhh, I still cannot find the qemu-dm-file anywhere.
  • Got it! I had to add the ioemu to the USE-flags and reemerge xen-tools. Doesn't look like any other package I have has to be recompiled after adding this USE-flag. So, forget the whole procedure I described above about installing qemu.
  • Now I first create a directory where to host Windows, and copy into it the Windows installation CD - installation should be faster without accessing the CD itself:
    dd if=/dev/cdrom of=/mnt/big/os/windows/wincd.iso
  • And I create as well a file where to host Windows (4096*1024=4GB file)...
    dd if=/dev/zero of=winfs.img bs=1024k count=4096
...and format it (if you don't find the mkdosfs utility, emerge dosfstools) with a Windows 32bit filesystem:
mkdosfs -vF 32 winfs.img
  • I wrote the config file /etc/xen/winXPhome as follows: 
    # Kernel is the HVM-Loader as the guest system is not adapted to Xen.
# Hope 384MB RAM are enough.
kernel  = '/usr/lib64/xen/boot/hvmloader'
builder = 'hvm'
memory  = '384'
device_model='/usr/lib64/xen/bin/qemu-dm'

# Drives
# First the "fake" filesystem where to install Windows simulating a drive 
# in write mode, afterwards the image of the Windows CD rom.
disk    = [ 'file:/mnt/big/os/windows/winfs.img,ioemu:hda,w',
'file:/mnt/big/os/windows/wincd.iso,ioemu:hdc:cdrom,r' ]
#disk    = [ 'phy:/dev/hda,hda,r' ]

# Hostname
name    = 'WinXPhome'

#  Networking - change xenbr1 to what ifconfig returns.
#dhcp = dhcp
vif = ['type=ioemu, bridge=xenbr1']

# Select "boot='d'" if you want to boot from cdrom, or "boot='c'" to boot from harddisk
boot='d'
# Don't restart the domain once it reboots.
on_reboot = 'destroy'
#vncviewer=0
vcpus=1
#ne2000=1

# Settings to have the mouse have the correct 
# coordinates and make it follow the pointer faster
usb=1
usbdevice='tablet'

# This OR the next section has to be enabled
# Settings for vnc disabled.
vnc=0
sdl=1

# Settings for vnc enabled - disable the previous
# section if these ones are enabled.
#vnclisten="0.0.0.0"
#sdl=0
#vnc=1
  • I started up the windows installation with...
    xm create /etc/xen/winXPhome -c
...and it works!!! :o))))). I see the Windows CD booting.
  • As I am lazy, I emerge vnc to get remote desktop access. I tried creating virtual X desktops on the host domain (dom0) with tightvnc, but I lost mouse and keyboard focus after Windows booted - I couldn't do anything and had to shutdown the guestOS with xm shutdown WinXPhome.

Details...

  • Have a look at my config file (the two "usb" entries) if the position of your mouse pointer does not match the cursor drawn in the Windows window.
  • Remember after installing to set boot='c' to avoid the CDROM to boot.
  • If I start Windows by enabling the VNC server, I can connect to it on the host machine with:
    vncviewer 0.0.0.0:1
This because the display :0 is already used on the host machine by X (I enabled it as default in /etc/X11/xorg.conf).
If you don't have any other VNC server running, the display you will have to connect to will be the :0:
vncviewer 0.0.0.0:0
If you want to connect from a different machine, it works as usual:
vncviewer <IP of remote machine>:<display number>
You can set the password needed to connect to Xen-VNC in /etc/xen/xend-config.sxp

Screenshots

Windows window while it is being installed: xen_winXP_install.jpg: 1024x768, 163k (March 14, 2010, at 03:00 PM)
Windows up and running: xen_winXP_running.jpg: 1024x768, 145k (March 14, 2010, at 02:57 PM)
Accessing it from a different machine: xen_vnc_remote_windows.jpg: 1440x900, 140k (March 14, 2010, at 03:02 PM)
Zattoo running in Windows: xen_vnc_remote_zattoo.jpg: 1440x900, 207k (March 14, 2010, at 02:37 PM)

Problems

  • Windows is very quiet. I'm currently not able to get any sound out of it - the soundcard is not recognized. An alternative would be to give to Windows direct PCI access (PCI-Backend driver active in the kernel config), but unluckily this is not yet supported for non-paravirtualized domains (as Windows).
    Update: now, since I set the following options in the DomU configuration file, Windows recognized a soundcard: 
    audio=1
soundhw='all'
  • Accessing Windows using VNC used the wrong keyboard layout (us). This did not happen when using SDL. To fix this, add to the config file of the DomU the entry "keymap='de-ch'" (the keymaps are defined in /usr/share/xen/qemu/keymaps).
  • Have a look if you need PAE. I deactivated it for the DomU with "pae=0".
  • After upgrading from Xen 3.0 to 3.2 my network did not work anymore. Fixed it by changing in the file...
    "/etc/xen/scripts/network-bridge"
...the line 93 from...
"addr_pfx=`ip addr show dev $1 | egrep '^ *inet' | sed -e 's/ *inet //' -e "s/$1//"`"
...to...
"addr_pfx=`ip addr show dev $1 | egrep '^ *inet ' | head -1 | sed -e 's/ *inet //' -e "s/$1//"`".
  • After upgrading from Xen 3.0 to 3.2 using xenbr0 as bridge did not work anymore - the bridge has now become eth0 while the physical network interface is peth0. Change therefore in the Windows config file the line "vif = ['type=ioemu, bridge=xenbr0']" to "vif = [ "type=ioemu, bridge=eth0" ]".
  • If you cannot see the console after typing something like "xm console 123" (no error message pops up - it's like the console is invisible), see this (extract from my entry in the forum):
    Reading a little bit here and there I saw that people mentioned to disable the serial ports in the kernel configuration to make the virtual console of Xen work. Now, I'm using the latest Xen-kernel provided by Gentoo, version 2.6.21 and in there (in the kernel config menu of the guest domain) I went into "XEN", selected the option "Disable serial port drivers" and rebuilt the kernel. After that the console was still not working, so I uncommented a line in my "/etc/inittab" file which now reads as follows:
    [CODE]# SERIAL CONSOLES
s0:12345:respawn:/sbin/agetty 9600 ttyS0 vt100[/CODE]
    Now my console works. If you have an older Xen-kernel, as far as I remember that option is not shown under the menu-entry called "XEN", but you'll probably find an option called something "Serial ports" under "Device drivers => Character devices".

Reference docs

Gentoo main guide to install Xen
Somebody else doing running Windows in Linux
Qemu Gentoo wiki
nVidia support for Xen?