www.fridu.net

While major ISP like Orange, Vodafone, Telfonica, ... have a numbers of independent cabinet to compartmentalize their architecture, this is not an option for small non profit or SOHO organizations that in no way justify such complex architectures and/or neither can afford it.

This note describes "Fridu-in-Xen" virtual ISP architecture, it explains how to simulate a fully compartmentalize mini ISP (Internet Service Provider) running on a unique Linux box and hosted on a cheap remote site (OVH ) leveraging XEN, VPN and QoS. It is build in such a way that anyone with an acceptable level of network and Linux knowledge should be able to replicate the architecture on its own hardware in few hours, then if you like it you may start contributing to the improvement process.

I use this architecture in real for fridu.org to support a number of non profit organizations, obviously it still lack some nice feature like redundancy, load balancing, supervision, ..., it's not that we could not extend "Fridu-in-Xen" to support it, but I have neither the time neither requirements for it. Nevertheless, in its current version it already allows you to provide for a very reduce cost of administration every typical services we expect from a good service provider Portal, Messaging, Voice/IP, ...this with an acceptable level of quality of service including security, backup, QoS, ...

Disclaimer

Anything I wrote here was done outside of my professional work context and  none of my current/past employers/customers have participate or even be consulted for this work. Fridu is 100% part of my free time, and everything including hosting is funded on our pocket money and used to support non commercial friend organisations. While I think I have the technical background to design a smart architecture (cf:my profile). I nevertheless do not garanty that it will work for you, or even that you will agree with me. I still hope it may help some of you and I would be more than happy to incorporate improvement if ever you have some.

Introduction

For professional reason I was been lead architect for few major internet service providers in Europe, while obviously most of us do not requirer this level of architecture complexity, and without taking about majors GSM-operators and/or world wide contend providers that count users in tens or hundred of million, most of us will never have to deal with even a small million of users.

This being said, even if you support only few tens/hundred of users it is very nice to have a compartmentalized architecture, where you can start a new instance of Linux for a test server in few minutes and/or do a disaster recovery without sweating for hours.

Fridu-In-Xen architecture was design, with the same security constrains and administration concern than big ISPs, only replacing physical element with virtual equivalence. The fundamental requirement was to build an architecture that was as close as possible for the administrator of a traditional physical architecture. I still lack some features like fault tolerance or load balancing, we could obviously add it, and I have no doubt that the same paradigm would work, but within Fridu usage context I have not justification for such an effort.

When using Fridu-In-Xen type of architecture? obviously when ever you want :) This being said it was designed after few major crashes we had one box we used for hosting fridu in the past. Each crash generated best case one night of work and worse case a full weekend. We also had trouble to upgrade some services because of the interdependency of all applications running together in a shared cabinet and when ever we had to reboot after a security update it was a nightmare. Last but not least, because Fridu is 100% based on volunteer work, everyone tend to cook its own food before eating it. Unfortunately this lead in, far too many people with far too many access right, and when a problem raise you never know who made it.

Conclusion: if you want a small and simple architecture than allows you to:

• Add/Recreate an instance of Linux is less than 5mn
• Provide untrusted user with root access to their VM without sweating
• Keep a complete control of your infrastructure without dealing with the detail of each VM
• Have access to a simple administrative network, that makes transparent  the fact that you share a physical cabinet with others.
• Have control on QoS (Quality of service) at network, Ram, Disk level
• Handle smartly multiple external NAT IP addresses.
• ...

Then you should probably check Fridu-In-Xen. Alternatively your may want to check Fridu OpenVZ (here)

ISP typical architecture

Most serious Internet service providers are organized in zones, even if they may use different name like: clouds, classes, .... Each zone groups a couple of machines that share both a common level of security and a class of service. In order to guaranty high availability a zone must at least have two physical boxes, but depending on provided services and load a given zone may have ten or more physical/virtual boxes. Within a zone load is balanced, today load-balancing is mostly achieve by hardware intelligent switch/routers like Alteon, Cisco, ... all those modern hardware provide a very secure way of isolating port by VLAN and outside of some very paranoiac users l(bank, government, ...) most people will accept to share the same physical routers even for different security level. Administration is typically done through a completely independent network, that have a multi-legs firewall in order to make sure that even if a hacker take control over one zone, he cannot use the admin network to move from one zone to the other. Following graphic give a typical view of current major ISP architectures, and depending on the number of physical boxes could be extended from few thousand to few million users.

 Fridu Xen Architecture

As explained in the introduction, the goal of Fridu Xen architecture is to provide for minimal fraction of the cost the same level of administration and security facilities, we do have with a conventional ISP architecture. Nevertheless we have to accept some limits, especially hight availability will be reduce has we only have one physical box, obviously we could have two of them and use some software load balancing methods, but as today this is not part of Fridu-in-Xen reference model (may be later :)). Nevertheless looking at today hardware stability, associate to facilities offer by virtualization for backup, restore and relocation of virtual machines this architecture model attached to a good backup strategy and disaster recovery plan, should be more than enough for medium size operation, this especially for non profit organizations that what ever they need do not have funding for conventional ISP type of model. 

Setting up basic Xen infrastructure

This guide is not a Xen quick start and I expect at this level that you already have Xen/domU up and running. If you're making test on a local workstation then installing XEN is quite straight forward with any current major distributions. With OpenSuSE this is as simple as going in YaST and click on Xen tab for Kubuntu an other Debian flavors aptget will do the job for you. Nevertheless If you are in a remote hosting environment, then it might unfortunately be more tricky, I posted a note on how to make this within OVH remote hosting environment that is available from here and that hopefully should be reusable within most of other hosting facilities.

Default Xen standard distribution provide three network scripts, unfortunately none of them is doing the we want we unfortunatly had to build one, that is downloadable from here Fridu-Script. Assuming that you install it at default /etc/xen/scripts location, you then need to update your xend-config file to call it. Note xend normal script only create only one bridge, this one will create as many bridge as needed.

You need one bridge per zone, but it is not mandatory to build them from xend. For testing purpose it is often easier to create/destroy them manually with  Fridu-network.script as shown in next table and check result with ifconfig command.

Note: Xend when stopping does not remove unused bridges, it is not a problem, but may lead to confusion when making test. You should know that you don't have to reboot to remove bridges, Fridu script does not touch your main NIC also as soon as xend is stopped you can safely remove all unnecessary bridges. Note than default Xen network bridge rename and attach you main NIC, making deletion of bridges more sensitive. In order to clean bridge use following commands:

• /etc/init.d/xend stop      ;# make send is stopped
• brctl show  xen-br1       ;# make sure they is no interface attached
• ifconfig xen-br1 down    ;# stop bridge interface name xen-br1
• brctl delbr  xen-br1        ;# delete old bridge

Making the assumption that Xen is now ready and working check with "xm list" that domu-0 is running. If everything is OK you're ready for next phase.

Installing a new VM 

This chapter explains step by step how to implement a Xen-VM under LVM. Obviously Fridu-LVM script will do all of this automatically for you, but if you want/have to be in a position to debug, when things go wrong, then you may want to know what is under the cover, automatic mode is explain later in Quickstart chapter.

While this is not mandatory, at least for production I deeply recommend you to implement each Xen's VM with three LVMs: first one for root, second one for swap+tmpfs and last one for logs. Keep Xen sparse image file for test only, in production sparse images have absolutely only have disadvantages. Some may argue that LVM model is more complex to setup, but as Fridu scripts does the job automatically, who cares.

Building you VM root image 

As the target of this post is not to explain how to run a basic Xen; in order to save time I propose you to download a pre-build Xen root file system, this image is nothing special. It is an out of the box YaST2 "in-directory" OpenSuSE-10.2 install, and contains a basic English/TextMode OpenSuSE-10.2 root. Nevertheless it is ~350M B also depending on your DSL link you may want to rebuilt it yourself, but following explanation make the assumption you have something equivalent (Download Xen-OpenSuSE-VM )

Outside of performance reason, I see significant advantages to LVM over sparse image.

• we can separate tmp,swap,log very easily, which make the image to save much smaller and may save a lot time during a disaster recovery.
• in case of trouble mounting an LVM is much more simple than mounting a sparse image through a loopback device.
• we can leverage LVM extend/reduce capability including online extend for reiserfs (has trouble to believe it, but ext3 does not support this !!!)
• you can do the fsck directly from the main domain in case of trouble.
• ....

In order to create an LV (logical Volume) you need to have an active VG(volume group) on your system, if this is not the case you need to dedicate one or more physical partitions to LVM. This guide is is a go to production strategy and make the assumption that you have a working LVM with 6G free, if this is not the case please build one before moving forward.

Building you VM config file starting your VM 

Your VM is now almost ready to boot, we still to check a couple of things

• VM config file               (sample here )
• VM Xen kernel+initrd    (sample here )

Kernel+Initrd can be place anywhere on your disk, configuration file need tiny adjustment to reflect your configuration, While it is possible to mount image and make a copy before launching the VM, I found out that it is finally more simple to place a copy of your VM kernel+initrd somewhere on disk.

• disk =  should point on your root+swap LVM is you do not have CD image remove hdd
• kernel    = "/mypath/vm/boot/vmlinuz-xen"
• ramdisk = "/myPATH/vm/boot/initrd-xen"

**

You have now a fully working Xen VM but has you have probably notice network fail to connect this is normal has do not have get set up out network infrastructure.

Network Instrastructure

This chapter describe the necessary step to make your VM to receive a valid IP address from DHCP. All described step are obviously done automatically by Fridu script. 

If you have run "/etc/init.d/xend start" after updating your config with Fridu-network script you should have a xen-br0 or what ever you choose has default bridge name. This bridge should be active with a valid local IP adress (ex: 10.10.1.1) . Now what we want if to have our VM to receive automatically a valid IP address within this  bridge IP/netmask range.

I've try many options to provide VM ip address from domU, but outside of hacking startup scrip inside the VM, I did not find any smart mechanism to do the job, and as adding dhcp=dhcp seem not to work (at least on OpenSuSE),  the only option is to provide a valid DHCP config for Xen VM virtual NIC.

Make you VM DHCP aware 

At this level you have to make sure that within /etc/xen/vm/MyVMconfig, you provided a MAC address to your "vif" interface definition. If you forgot to do so, your NIC mac address wiil change at every boot, forcing Linux to rename eth0 to something different. You can stop this by "FORCE_PERSISTENT_NAMES" value inside /etc/sysconfig/network/config on the VM or by preventing udev to run on the VM.

Install a local DHCP server.

As the best option is to use DHCP for our VM network, we need a local DHCP. As we do not need anything big dnsmasq is the perfect product for our needs, it is very small does not require any special configuration. It takes its config values directly from /etc/hosts and /etc/ethers and also act has a local DNS, plus Internet cache. You probably won't even have to recompile it, and while it is not part of default OpenSuSE DVD it is nevertheless very easy to find on  any good rpm repository like rpmfind.net. You can download my sample config for dnsmaq from Here.  

Your VM should have a working internal IP adress, and you should be able to ping from your VM to Dom-0 and vice versa. As you still have no NAT(Network-Address-Translation) and no routing, you are constrain to limit yourself within your box and cannot reach Internet. Nevertheless you should be able to ping any local address  including any other VM, if you have some.

Notes:

** when making test, I had few froze of my VM vif interface, result is that DHCP does not work. If tcpdumping corresponding interface for that given VM on dom-0 we do not see any packet. Ifconfig up/down will not solve the problem, but restarting the same VM with a different vif name should work :( 

Security

The chapter describe Fridu reference architecture security model, unfortunatly the way iptables work make a manual step by step operation guide useless, and I make the assumption that user will generate iptables rules automatically, obvisouly script can dump iptables commands allowing anyone to double check what it going one.

Fridu-in-Xen security model is designed to be very simple to administrate. Firewall iptables are generated automatically through a small parser script and the administrator only have 3 rules to handle. This describe how to implemented security before you reach a given VM. Obviously each virtual machine may later have its own set of firewall rules, but this is out of scope of this guide. You should look Fridu-in-Xen security model as the equivalence of what a network/infrastructure team is providing inside a traditional Telco.

Note: Firewall has been extended to support other virtualization environment like OpenVZ or VirtualBox and has now its own page (here)

QuickStart

Most of us are love to try without any learning curve :) Making the assumption that you have acceptable administation skills and root password of your box, starting Fridu-in-Xen architecture should take you from less than one hour to probably almost one day. When in place, creation of a new VM take excatly one minute, this for a basic but fully YaST compliant OpenSuSE-10.2.

Creating VM automaticlly

Two components will be use for this quickstart Fridu-scripts and a minimal OpenSuSE-102 distribution. Within scrips archive lvm-script is mandatory, it is the one that does automatically every step described in chapter "Installing a VM", second one firewall-script is mandatory only if for Fridu security model. Nevertheless and what ever you will chose to implement later, for a real quickstart I recomment you to use a default configuration in order to maximize "cut and paste". When your first automatically created VM will run, then it will be time for you to taillor this to your own distribution and/or network strategy, starting by trying an Ubuntu in a domu virtual machine. On Dom0 I tested Fridu-Script with OpenSuSE-10.2 Fedora-core5 and Ubuntu-6.1/server and I would expect it to work out of the box on almost any distribution.

Action:

• get OpenSuS-102.tgz somewhere on a disk visible from your Dom0
• untar Fridu-scripts archive and type edit ./install.sh to reflect your configuration and install scripts and samples.
• edit /etc/sysconfig/Fridu-lvm.config to reflect your site preferences.

Assuming that you have a basic XEN configuration in place and a free LVM volume group starting Fridu-lvm.script should you take less than 15mn. Then time to create a new VM with a basic OpenSuSE-102 take on my system 180s, this from typing "return" on script create command to the point where my VM anwser to Dom-O ping.

Start xend with Fridu Network Script

Note: Fridu script for VM creation will work with standard XEN brigde script (/etc/xen/scripts/network-bridge) , nevertheless Fridu security model leveraging zones requirer Fridu-Xen-network.script. The main difference in between both model is that Fridu bridge VMs by zone and later route in between zones, when default Xen script bridge your all your VM to your external NIC interface.

Check you have enough free space on your volume group disk.

You MUST have a working LVM environment to use Fridu VM creation sript, default configuration will create two Logical Volumes per VM with respectively root=5G for swap/tmpfs=1G. Note than while automatic creation of VM only work with LVM, zone security model is independent on disk layering and work perfectly with sparse image file.

Create your site config file

While every single parameter of VM creation configuration can be defined/overloaded at command line, it is usually a good practice to provide some adequate site default values, this especially for almost static element as: kernel location, bridge name, ...

Creating your first VM

If you have a correctly customized config file and stick to its default name. Then in order to create a new virtual machine, you only have to provide "name+ip-address". Obviously you may want to customize more things as: network mac adress, root filesystem, size, ... but for a basic test those two parameters should be more than enough. 

When using verbose=1 as in previous example you get a trace of each step the script is going trough. If needed you can restart the process step by step. Bypassing a step is done by using the option "stepname=no" in command line. By default value is "yes" for every steps except if obvisouly you changed this in your config file. 

• Create VM 's root+tmpfs/swap logical volume group                         (lvmcreate=yes)
• Format root LV with a filesystem (default=ext3)                              (fscreate=yes)
• Untar your root distribution root LV                                                 (installroot=yes)
• Custumize root distribution to run dhcp client, and few other things (tuneroot=yes)
• Update your host /etc/hosts and /etc/ethers automatically               (updateether=yes)
• Build and place vm config file in /etc/xen/vm                                    (createvmconf=yes)

Check you newly generated VM config file

At least at for first VM creation you should check generated config file, and see if is compliant with your site requirements. As everything is done automatically, it should be OK, but shiz.. happen !!! and I did not test on enough different configurations to garanty that it will work everyware and/or under every conditions.

You should at least check what my script does not check:

• Selected bidge exists and is the one you want to use for that given VM
• IP adress you gave for VM is siting within selected bridge network mask
• The name you gave was not already present in /etc/hosts in which case your "vmip" param has been overloaded automatically.
• Kernel+Initrd: script check files existence, but not if it is a valid one.
• Disk should be OK, except if you bypassed lvm/filesytem creation step.

Start your VM

If everything looks OK, you're ready to try your newly built VM, this with:

This is typical Linux boot process, and it should run smoothly at least in console mode, if you have trouble it will probably come from your network config, this especially is you have many zones and mess up your config.

Distribution Dependencies

They're should not be any decencies to your preferred distribution and this script should work with any distribution. Worse case "tuneroot" will fail to understand your VM administration tree and you will have log by "xm console' to activate DHCP on the right virtual network interface.

I noted that Ubuntu is not shipped with domUloader.py as result "xm create" command cannot extract kernel directly from the VM at boot time and you have to store a local copy on VM kernel inside dom0, but generally in case of trouble it is easier to boot with a kernel from dom0. You can pre-extract kernel from virtual machine with Fridu-kernel-extract.script and then rebuilt xen config with Fridu-lvm.script.

How to delete a VM

Nothing more simple, stop your VM and remove attached logical volume

How to Transfert a VM to an other host/environement.

Stop your VM, mount in Dom-0 root logical volume and send your backup.tgz to where ever you have to, at destination rerun Freidu-in-Xen create script on your received root backup.tgz.

Hoop !!! it's not working

They are obvisouly many way to mess up. First I do not pretend that my script is perfect, it is build in bash and is very far from checking everthing it could. Secondly you're not perfect either and you may have done something wrong :)

First class of errors will break script at creation time. Potential errors are almost unlimtted and range from "not enough space" to "already used IP address", "distribution not found", ... In this case you typically restart creation script changing one parameter and bypassing previsouly succeded step. Here after few sample of typical errors.

Second class of errors: create command fly, but "xm create $vmname" fails. In almost every cases this is coming from your newly generated VM config file. You need to check "/etc/xen/vm/$vmame" and try to understand what went wrong. You may have an invalide config for example in your disk definition, this is especially classical when you generate a config on an existing distribution bypassing previous installation steps.

Third class or error: Your VM start, you can connect onto it with "xm console" but it does not do what you want. In most cases problems are coming from network, but you nevertheless can connect from "xm console", in rarer situations kernel will fail to boot.

• Cannot find mandatory device. After a relatively long wait, you arrive on a "mini shell". What's happen is that your kernel failed to mount your root and you're sitting in the initrd ramdisk. This is probably coming from in invalid "root" value in one of your boot attributes, but it may also come come from a wrong initrd. A this level you have almost no commands but you still can leverage "/sys" pseudo filesystem to browse around. For example "ls /sys/block" will allow you to check which disk are visible, then make sure that "root=/dev/xxx" inside vmconfig is pointing to the good one.
• Kernel Panic, a little more complex to handle has you do not have any running system.  In most of the case this is coming from a wrong Initrd, a typical example is trying to mount a reiserfs formated logical group with an inirtd that only contains ext3. The best option is to look messages, and eventually to mount from Dom-O your VM LV and look from inside what could be wrong.
• Your VM start, you can connect from the console, but your network is not working. This is for sure the most typical case. The first thing is to check basic network infrastructure environment: is your DNS running ? does it listen on Dom-0 bridge port? is your VM IP within bridge IP range ?, ... If you do not find any obvious error then the best is to use "tcpdump" in conjuction with "dhcpcd-test" and ping. You connect on the console to force a request and you check from Dom-O packets coming from your VM. When this is done you need then to run tcpdump inside the VM to see if packet are passing from Dom-0 to your VM.

Creating of your Firewall and Security zones

Distribution is shipped with tree samples, what ever is your target, I advice you to do is to draw your target network, and to build rule from your graphic.

Let's start with a simple case. We only have one zone, with one external IP and two Xen virtual machines.  Access to virtual machines is done by remapping ssh port from external to internal, and each VM should have a full access to Internet.

• ssh -p 22 root@MyInternetIP (connect dom0)
• ssh -p 23 root@MyInternetIP (connect Xen-One )
• ssh -p 24 root@MyInternetIP (connect Xen-Two)

 We only one one zone, each zone is defined with

• Name (what ever you want until is it unique)
• Network Interface, this is use to allow incomming traffic
• External IP address, that is used to allow incomming traffic and to NAT outgoing one.
• Bridge, which define the entry point for the given zone
• Zone IP address and the network mark that will contain all VM IPs

We have three applications, each of them is defined with

• Name (what ever you want until is it unique)
• Zone name the application site in,as defined previsoully . Note that "none" is a special zone pointing on "Dom0"
• External port/proto Internet user will see. Note that external interface and IP is inherited from zone definition.
• Internal VM target IP and port in case we remap the application.

An avanced sample, this is more or less the one we have for Fridu in production.

• One Network Interface
• Two Internet IP
• One VPN for admin and Internal services.
• Two zones (one per admin), First one support two VMs while second one only hold one.
• VM-Common services(LDAP, backup, IMAP, ...)
• VM-Fulup  (WebMail, Photos, Video Streaming, ...)
  • VM-Domi (Joomla)
  • VMs can see each other within and outside their containing zone.
  • VMs can talk route to any VPN endpoint and vice versa

In this example we have two zones and, while they are both mapped on the same physical network interface (eth0) they are using different external IP adresses.

zFulup is mapped onto  91.121.65.140

• Accept external traffic from port 25 and map it locally onto vm(Common) port 2525
• Accept external traffic from 80 and map it onto vm(Web-Fulup) port 8080
• Accept unlimitted internal traffic

zDomi is mapped onto 87.98.130.56

• Accept external traffic from 80 and map it onto vm(Web-Fulup) port 8080
• Accept external traffic from22 and map it onto 22(ssh)

Dom0 (none zone) accept port 22(ssh) and openvpn(44096+563)

• no special restriction on zone
• VPN(tun0) can talk to any VMs and vice versa.
• VMs can talk inside and outside their own zone. 

Guru sample, even if you do not need it today it is important to make sure that you architecture can scale to support your furtur requirements. Next case has been tested in my lab and while today I have no requirement and no money for such a configuration, and wanted to make sure it was possible.

• Tree zones two mapped on internet with a dedicated NIC, one isolated from Internet.
• Dom0 support SSH and OpenVPN.
• First zone as tree VM and second two.
• First zone handle mail, web and SIP
• Second zone only web.
• A tuning rule prevent second zone from sending mail.
• Custom rules allow unlitted VPN traffic and inter zone communication.

Firewall rules for this sample are defined here after. You may note that the Internet isolated zone does not have any applications defined. This is logical as applications only define autorization for external traffic, we could nevertheless have a special custom rule to prevent outgoing traffic to the external word as well. Check tune rule for zDominig zone that prevent every VM from this zone to send mail, this both externally and internally.

If you find bugs, or have done improvement please let me know.