Creating a bridged network for KVM



I just finished fighting trying to create a bridged network to use with KVM for multiple virtual machines (VM). To be honest I don't fully understand the concept but this is how I did it:

Method 1 - Works for root

Method 2 - Works for other users too

Method 1 - Works for root

First of all I had to create a bridge which will be used to create virtual interface - each virtual interface will be used by one VM. For a network interface called "eth1", execute the following commands:

You'll have to work directly on the local machine as if you do this remotely after executing the first command you'll lose the connection to the host.

  • ifconfig eth1 0.0.0.0 promisc up
    Take note of the IP number that "eth1" has before issuing this command (by executing the command "ifconfig"). The command will put your network card ("eth1") into promiscuous mode which will make your network card listen to all traffic it receives, even if it is not intended for it. But don't worry - your switch/router will send to the card only the traffic that the host or the VMs running on the host is intended to receive. Remark: this step is critical especially for wireless network cards - if you wireless network card is not able to switch to promiscuous mode this method won't work.
  • brctl addbr br0
    Creates a bridge called "br0".
  • brctl addif br0 eth1
    Links the bridge your just created with your network card.
  • ifconfig br0 up
    Activates the bridge.
  • brctl stp br0 on
    No clue.
  • ifconfig br0 10.0.0.20 netmask 255.0.0.0 up
    I assign the IP number that "eth1" had before, to the bridge "br0".
  • route add default gw 10.0.0.2 br0
    By assigning to "eth1" the IP 0.0.0.0 the default gateway is lost - I redefine it to go through the bridge "br0".

Now if you run "ifconfig" you should see that the interface "br0" is up and running. You should still be able to connect to/from this machine to anywhere in the world or your internal network. That's it - the bridge is ready. You can put all the above commands in a script and schedule it to be executed at each boot..

As we have a bridge we can tell KVM to use it. In Gentoo (the Linux distribution I have) this is done automatically (KVM calls them automatically) by two scripts which create/destroy a virtual ethernet device. The two scripts, called kvm-ifup and kvm-ifdown are stored in the directory /etc/kvm/:

  • kvm-ifup: 
    if [ -x /sbin/brctl ]; then
  BRCTL="/sbin/brctl"
elif [ -x /usr/sbin/brctl ]; then
  BRCTL="/usr/sbin/brctl"
else
  echo "no bridge utils installed"
  exit 1
fi

if [ -x /sbin/ip ]; then
  switch=$(/sbin/ip route list | awk '/^default / { sub(/.* dev /, ""); print $1 }')
  /sbin/ip link set $1 up
else
  switch=$(/bin/netstat -rn | awk '/^0\.0\.0\.0/ { print $NF }')
  /sbin/ifconfig $1 0.0.0.0 up
fi

${BRCTL} addif ${switch} $1
  • kvm-ifdown: 
    if [ -x /sbin/brctl ]; then
  BRCTL="/sbin/brctl"
elif [ -x /usr/sbin/brctl ]; then
  BRCTL="/usr/sbin/brctl"
else
  echo "no bridge utils installed"
  exit 1
fi

if [ -x /sbin/ip ]; then
  switch=$(/sbin/ip route list | awk '/^default / { sub(/.* dev /, ""); print $1 }')
  ${BRCTL} delif ${switch} $1
  /sbin/ip link set $1 down
else
  switch=$(/bin/netstat -rn | awk '/^0\.0\.0\.0/ { print $NF }')
  ${BRCTL} delif ${switch} $1
  /sbin/ifconfig $1 down
fi

Now the only thing missing is the configuration to start kvm. Here an example which starts up my WindowsXP using a virtual ethernet interface created on-the-fly: 

kvm -hda /virtual_os/windows_xp/fs_image/windowsxp.img \
-cdrom /virtual_os/windows_xp/fs_image/winxp_install_cd.iso \
-boot c -m 512 -soundhw es1370 -usb -net nic,vlan=0,macaddr=CE:AD:BE:EF:16:10 \
-net tap,vlan=0

Note that you'll have to provide a MAC-address ("CE:AD:BE:EF:16:10" in my case) in order to make the whole thing work - use a different MAC for each of the VMs.

Works for other users too

You'll still have to run the same script as above when the PC boots - add it to your runlevel, after your "normal" network becomes available: 

ifconfig eth2 0.0.0.0 promisc up
brctl addbr br0
brctl addif br0 eth2
ifconfig br0 up
ifconfig br0 10.0.0.20 netmask 255.0.0.0 up
route add default gw 10.0.0.2 br0

Additionally create a script that your user will use to run the VM: 

#THINK ABOUT THE FOLLOWING:
#1) Make sure that the user belongs to the group "kvm"

#2) Check with "visudo" ("sudo" package) that the user has the needed authorizations. E.g.:
#Let the user create taps
#user_virtos localhost=/usr/bin/tunctl -b -u user_virtos
#Limit the user to link max 29 taps to br0
#user_virtos localhost=/sbin/brctl addif br0 tap[0-9]
#user_virtos localhost=/sbin/brctl addif br0 tap[0-2][0-9]
#Limit the user to activate max 29 taps
#user_virtos localhost=/sbin/ip link set tap[0-9] up
#user_virtos localhost=/sbin/ip link set tap[0-2][0-9] up
#Limit the user to delete max 29 taps
#user_virtos localhost=/usr/bin/tunctl -d tap[0-9]
#user_virtos localhost=/usr/bin/tunctl -d tap[0-2][0-9]

#3) Check that the file/device where the OS resides gives r/w access to users of group "kvm".

#START OF SCRIPT

INTERFACE=""
BRCTL_OK="";
IFACE_UP="";
IFACE_DESTROYED="";

#Start kvm. !!Note not to use -daemonize otherwise the interface will be destroyed immediately

KVM_CMD="kvm -nographic -vga std \
        -hda /yourdir/osimage.img \
        -boot c -m 512 \
        -net nic,model=virtio,vlan=3,macaddr=DE:AD:BE:EF:17:29 \
        -net tap,vlan=3,script=no,downscript=no,ifname="

#START-Create the tun interface
INTERFACE=`sudo tunctl -b -u user_virtos`
if [ $INTERFACE  ]; then
        echo my_message: Interface $INTERFACE created
        echo ""

        #START-Link to bridge
        sudo brctl addif br0 $INTERFACE && BRCTL_OK="ok"
        if [ $BRCTL_OK  ]; then
                echo my_message: bridge br0 linked to interface $INTERFACE
                echo ""

                #START-Activate interface
                sudo ip link set $INTERFACE up && IFACE_UP="ok"
                if [ $IFACE_UP  ]; then
                        echo my_message: interface $INTERFACE is up
                        echo ""

                        KVM_CMD="$KVM_CMD$INTERFACE"
                        echo my_message: running the following command:
                        echo $KVM_CMD
                        echo ""
                        #Start the VM
                        `$KVM_CMD`
                        echo ""
                else
                        echo MY_ERROR: interface $INTERFACE could not be started
                        echo ""
                fi
                #END-Activate interface

        else
                echo MY_ERROR: bridge br0 could not be linked to interface $INTERFACE
                echo ""
        fi
        #END-Link to bridge

        #START-Destroy the interface
        sudo tunctl -d $INTERFACE && IFACE_DESTROYED="ok"
        if [ $IFACE_DESTROYED  ]; then
                echo my_message: Interface $INTERFACE was destroyed
                echo ""
        else
                echo MY_ERROR: Interface $INTERFACE could not be destroyed
                echo ""
        fi
        #END-Destroy the interface

else
        echo MY_ERROR: Interface could not be created
fi
#END-Create the tun interface

As you see we're not using anymore the two delivered scripts kvm-ifdown and kvm-ifup.

References:

Other informations

  • Don't forget to compile KVM using GCC 3.*!!! (gcc-config -l and gcc-config 1)
  • In Gentoo, if when starting a VM everthing hangs just after the BIOS startup and the last message is "Press F12 for boot menu", it might mean that the package overwrote the modules provided by the kernel. Fix this by adding to "/etc/portage/package.use" the line "app-emulation/kvm -havekernel -modules".
  • If the above procedure does not help, try the following (exactly the opposite - uses the modules from the package and does not look at your kernel): 
    find /lib/ -name kvm*
...delete the entries for "kvm.ko" and "kvm-intel.ko"...
    rm /lib/modules/2.6.25-gentoo-r7/kernel/arch/x86/kvm/kvm-intel.ko
    rm /lib/modules/2.6.25-gentoo-r7/kernel/arch/x86/kvm/kvm.ko
...check that the system didn't find any other modules...
    modprobe -r kvm-intel
    modprobe -r kvm
    modprobe kvm-intel
    WARNING: Could not open '/lib/modules/2.6.25-gentoo-r7
    /kernel/arch/x86/kvm/kvm.ko': No such file or directory
    FATAL: Could not open '/lib/modules/2.6.25-gentoo-r7/kernel
    /arch/x86/kvm/kvm-intel.ko': No such file or directory
...recompile the KVM-package WITH NO FLAGS...
    [ebuild   R   ] app-emulation/kvm-78  USE="alsa esd gnutls modules ncurses 
    pulseaudio sdl -havekernel -test -vde" 0 kB
...copy the modules that were created to the place where
   I deleted the ones that were provided by the kernel...
    cp /lib/modules/2.6.25-gentoo-r7/kvm/kvm-intel.ko 
    /lib/modules/2.6.25-gentoo-r7/kernel/arch/x86/kvm/
    cp /lib/modules/2.6.25-gentoo-r7/kvm/kvm.ko 
    /lib/modules/2.6.25-gentoo-r7/kernel/arch/x86/kvm/
...and at this point by loading the modules...
    modprobe kvm-intel
...you see a confirmation in /var/log/messages saying...
    Nov 16 17:25:31 MYSERVER loaded kvm module (kvm-78)
...and now I hope that everything works.
  • To be able to access the KVM-VM through VNC, start it up from a console with the option...
    -vnc 10.0.0.123:1,password -usb -usbdevice tablet -monitor stdio
...which will make the VNC server listen on the port 5901 of the server 10.0.0.123, will make the mouse work better (-usb -usbdevice tablet) and will output a KVM-console on the console. Afterwards to set the password for VNC, type in the console...
change vnc password
...and set a password (!!!any chars after the first 8 will be ignored!!!)
Keep in mind that VNC in KVM supports only 1 connection at a time and that the only available encodings are RAW and Hextile.