Why OpenVZ and not XEN.

After one year of operation with XEN, I chosed to move Fridu from XEN paravirtualization, to OpenVZ container model. Here after some explanations on the why of this change and the description of my new architecture.

Disclaimer

Anything I wrote here was done outside of my professional work context and  none of my current/past employers/customers have participate or even be consulted for this work. Fridu is 100% part of my free time, and everything including hosting is funded on our pocket money and used to support non commercial friend organisations. While I think I have the technical background to design a smart architecture (cf:my profile). I nevertheless do not garanty that it will work for you, or even that you will agree with me. I still hope it may help some of you and I would be more than happy to incorporate improvement if ever you have some.

Demonstration/Video

• English version
• Version Française

This demonstrations is a live screencast done with xvidcap on Linux, it shows how to create a new virtual machine through Proxmox OpenVZ web graphic interface, and then shows how to expose the newly created zone to the outside world with three different mechanisms: vpn, port forwarding and reverse proxy. Demo run for about 25mn, its using video flash and hopefully runs smootly on any plateform.

My ISP new OpenVZ architecture

The new Fridu architecture is very similar to the old Fridu-XEN, it uses the same firewall and provides the same networking port forwarding facilities and leverage OpenVPN for direct access to virtual machines. The only changes are the transfert from Xen to OpenVZ and the introduction of a reverse proxy at hypervisor level.

Xen is rock solid, but ...

I had no issues with XEN functionalities or stability. I never had to reboot any zones and shut down my system with 360days of update time, which mean XEN never went down from the time I switched on my system on, to the time I moved to a new dedicated server with OpenVZ. My Xen config is described in following post (here).

I nevertheless have two main reproach to Xen, the first one is that it locks physical RAM and thus requirer RAM that I do not have on my cheap dedicated server. The second one is that XEN to XEN networking is extremely slow, which is in fact the only MUST solve problem I see to Xen.

OpenVZ limited but light weight and simple.

OpenVZ has one strong limit compare to XEN, it is not a full visualization and  therefore you're limited to Linux only containers. People working with Sun will recognize Solaris zones concept, that was introduced few years ago. Like for Solaris every OpenVZ zones shared the same kernel, which at OVH translate in a Linux-2.6.24.7 kernel. This being said, it is important to understand that Linux distributions are independent of kernel, you can therefore run any Linux distributions you want under a unique kernel. While OVH ships Debian Etch with OpenVZ hyperviseur, you can chose any other distribution for your zones, new version of Fridu mostly operated with Ubuntu, but nothing prevents you from running multiple distributions. OVH ships template for Debian, CentOS, Gentoo and Ubuntu, but if this is not enough you can either create your own template or download one from Internet (OpenVz-WIKI)

OpenVZ includes a set of scripts to create/manage virtual machines, unlike Xen that is shipped naked and where I had to write more or less equivalent scripts by myself (cf: Fridu Xen Quick Start). Furthermore  OVH ships OpenVZ with a web console from Proxmox, not that I'm a big fan of having a GUI, but as you can see on the video, it is great to make sexy demos.This console allows you to create a new virtual instances literally in a mater of seconds :) It allows you to start/stop change ram size, IP adresses, etc. on any instances without forcing you to remember any special commands. While Proxmox console misses few features like an SSH applet, a firewall config, or a java VPN. I must say that I get used to it and create every virtual machine through the web GUI.

OpenVZ is very light weight, not only it shares the same kernel, but also the same filesystem and networking stack. Direct result is that, on a given server you can run more OpenVZ zones than you could run XEN virtual-machines. From a user point of view when a zone is up, wether you run OpenVZ or XEN is fairly transparent, this being said they are nevertheless some fundamental differences:

• OpenVZ ram is taken from hypervisor global pool (RAM+Swap), when Xen takes RAM from hypervisor physical ram and handles its private swap by virtual machines. Direct result is that with OpenVZ you can allocate more RAM that you effectly have, on the other hand if one zone lack RAM your zone will not expend on swap. In fact OpenVZ refuses to allocate RAM to requesting process, which for most of them translates in a core dump :( As swaping is never a good idea, both methods could finaly look more or less equivalent, this being said the small advantage of OpenVZ strategy is that it is easier to share unused RAM in between differents zones, which may have some important for cheap hosting configuration where RAM is limited.
• OpenVZ zones and hyperviseur share the same kernel, as a result some functions fails (ex: changing system time) while this should not be an issue, it breaks more scripts than it should :(
• No boot console is a real bad point for OpenVZ. It makes template creation complex. In fact you have no idea of what happen until you can access to your system, which obvisouly never happen when boot fail :( The fact you can access any given zone logs directly from the hyperviseur through /vz/private/zone-id directory makes debug possible, but far less simple than with a full boot console as provided by Xen. Note that some distributions like Debian will write into your VM /var/log/init.log the trace of your boot sequence, unfortunatly this is not working on ever distribution, Ubuntu being one of them.
• A snoopable network interface, this is a very good point for OpenVZ. While Xen also allows to snoop your network interface with TCPdump or Wireshark, Solaris zones does not support it. Snooping network  interfaces is a key feature for debugging any network infrastructure and almost mandatory to understand what happen when your hosting configuration is not working.
• A share filesystem, the main advantage is that you can access directly to any zone private disk from the hypervisor, the limit is that you cannot leverage a SAN interface directly from the zone as under xen with ISCSI. This being said for cheap hosting OpenVZ option is a better choice.
• Proxmox web GUI, while this is not OpenVZ as it, it is shipped standard and allows you to create/manage your virtual machines in a very nice and intuitive way.Further more Proxmox provide a list of appliances that you can download from ftp://pve.proxmox.com/appliances this allow you to install drupla, joomla as well as many other nice tool in a mater of seconds, nice isn't it ?

Fridu OpenVZ networking config

I reused the exact same firewall I had with XEN, did not even change one line from previous script. Nevertheless this time I used a more simple configuration with only one security zone, and  added of a reverse proxy (pound) for http port redirection on the adequate virtual machine.

OpenVZ hypervisor runs firewall rules for port firtering and forward to any service of a given zone and pound reverse proxy is handling http, redirecting get/post to the adequate web server. Note that if I had more public IP adresse I could get rid of my reverse proxy, but I use a cheap OVH plan (kimsiffi) which only provide two public IPs.

You can find detail options about Fridu Firewall  (here) my current config is somehow simple and explained here after.

Security zones: OpenVZ default config provides one virtual network interface (vnet0) at the hypervisor level. While this does not enforced strong isolation in between zones, you can nevertheless select a given outgoing IP adress from internal submask, which hopefully should be more than enough for most of us.

Here after an extract from my Firewall config, with my two public IP adresses. Both zones share the same internal (vnet0) and external (eth0) network interfaces. The only difference comes from internal network submasks 10.10.101/10.10.102.

Right after zone definition, comes the hypervisor network configuration. The hypervisor is in a special zone named "none". As it has direct access to external network interface (eth0) port forwarding is unnecessary and we only define port filtering is required. In my config, I need to open:

• Port TCP/22 for SSH
• Port TCP/80 and 443 for HTTP/HTTPS
• Port 5900 is used by VNCterm to provide console thought VNC applet.
• Port UDP/44096 andTCP/453 for OpenVPN

The last mandatory part of firewall rules is how to forward port to a designated virtual machine. For example mapping SSH port (tcp/22) to a external port (ex:tcp/2215) to allow the administrator to access to a given machine without passing thought the VPN.  In following configuration extract we see how I redirect mail SMPT/IMAP traffic, as well as HTTP (Apache/Tomcat) or SSH. With following configuration a "ssh -p 2215 root@my-pubic-ip" will connect  to ssh port 22 on virtual machine 10.10.101.5, and an http://my-public-ip:815 will connect on my tomcat instance on 10.10.101.5, etc.

While Fridu Firewall will do the job with only previous rules, you may want to add some extra rules to optimize your configuration.  Three set of rules in my custom optimization section:

• FTP one is only required if you want FTP to work in passive mode, which is often the case.
• Second one is almost mandatory, (tun+/vnet+) allow VPN to reach any zone directly, while (vnet+/vnet+) allows zone to zone connection. If you run a share VPN you MUST have tun+/vnet+, while vnet+/vnet+ might only be needed if one zone runs a service shared by other, example LDAP, DNS, ...
• Last one is an hugely hack that allow my TCP vpn on port 563 to be viewed as sitting on port 443 when targeting my IP-TWO. This allow me to have HTTPS/IP-ONE and VPN/IP-TWO when both share the same port and the same external interface (eth0)

Download Fridu firewall

Fridu firewall has been extended to support not only OpenVZ but also XEN or VirtualBox and has now its dedicated page.

• Firewall documentation (here)
• Download VM-firewall (here)

My Reverse proxy configuration.

One change with my previous architecture is the addon of a reverse proxy, while no mandatory this add a lot of flexibility.  I found pound to be exactly what I needed. While pound is originally more target to be use as a load balancer, fail over and SSL end point, it works very well as a generic proxy.Pound is part of standard distribution and thus a simple "apt-get install pound" is enough to get it run. While documentation is very limitted, the configuration file remains simple enough for this not to be an issue.

• 1st define a listen IP-ADDR/Port for your reverse proxy
• 2nd define your services, with as input your DNS name as known by external internet users,  and as output your OpenVZ internal zone + port.

In following config sample, my pound wait on interface 91.121.173.80 (Fridu public addres) port 80. It then forwards any http request with destination "www.fridu.net" to virtual machine named into the hypervisor /etc/hosts "vz-opensso" on port 8180, and request to zxid.fridu.net to the same virtual machine but on port 81.

Dnsmaq a small domain named server.

Running a local DNS name server is not mandatory it is nevertheless very convenient to have your VMs being able to use local name of others VMs. For such a small usage a full dns like "bind9" is useless and "dnsmasq" is what we need. When running "dnsmasq" on your hypervisor, it will reads your "/etc/hosts"  and serve it to every VMs, any other request are forward to the DNS defined in "/etc/resolv.conf" . As a result "dnsmasq" config is minimal, you only have to define which interface you listen and your default domain extention.

Last but not least you need your VMs to point on your local DNS, which is hopefully your template default.

OpenVPN

While being optional I highly recommend OpenVPN for such an architecture, it is an easy to install product and it significantly simplify administrator and developer task. OpenVPN is available out of the box with any distribution. Fridu configuration is avaliable (here)

Proxmox web frontend console

Proxmox's web frontend is not a native component of OpenVZ, nevertheless it is a very useful companion, and I deeply recommend to who ever is willing to adopt OpenVZ, to concider Proxmox. If you're not convinced check my small video demo.

For most of it its Proxmox usage is strait forward. The only issue I had was with the embedded vnc java applet that provides a vt100 entry point to virtual machines. While on the browser side Proxmod uses a traditional vncviewer on the server side it uses vncterm. I use VNC frequently, but  never seen vncterm before. Vncterm is a contribution from Proxmox to VNC, it is small vt100 emulation that can be render remotely thought a vncviewer, and does not requirer X11 to be installed on the machine.

When clicking in your browser on the "console" tab of your Proxmox GUI, it starts a vncterm process in your hypervisor. Note that because vncterm is started at the hypervisor level, neither you need vncterm to be installed in each virtual machines, neither you need a working network on targeted VM. If you check your huperviser process after click in console you should see.

A big thank to  Dietmar from Proxmox who provided me with a very quick and very efficient support.

Limitations, bugs and tips.

Limitations/ bugs.

I did not found any significant bugs on OpenVZ, nevertheless Ubuntu template as shipped by OVH need following fix, or SSH from within a template will not work. My system is up and running since summer and I never had any trouble. The only few things I do after each zone creation is to add two missing quite useful pseudo devices, update my /etc/hosts and add required packages with apt-get.

• No /dev/tty           -> "vzctl exec MY-VIRTUAL-MACHINE-ID mknod /dev/tty c 5 0"
• No /dev/random -> "vzctl exec MY-VIRTUAL-MACHINE-ID mknod /dev/random c 1 8"
• No /dev/console

Tips:

• When checking processes list on the hypervisor with "ps -ef" you see every processes, including the one running on VMs. As a result a "pkill -9 apache2" at hypervisor level will kill every single apache running on every VMs. Needs to be know
• If a VM restart fails (ex: you messed up network config, break some init script, not enough RAM, ....). While obviously you cannot log on your broken VM !!! You can still try to enter the VM from the hypervisor with "vzctl enter VM-ID" command. Example: your "syslogd" did not start, as this the1st process to be started by init-rc, it locks everything else and especially "sshd" which is usually the second one to be started. As you have no SSHD you cannot connect with SSH, obvious isn't it ? You can then use the "vzctl enter VM-ID" doing a "ps -ef" you should see a pocess like "/etc/init/rc 2" as init is not finished. You can then restart your syslogd with "/etc/init.d/sysklogd" or what ever is necessary to fix your problem.
• if you can reach a VM from the outside, but cannot reach internet from that VM, then you need to check your "CreateZone" rule.  Double check that your network mask is compliant with your VM local IP adresse.  Example (createZone ... EXT=IP1  INT=10.10.1.0 MASK=255.255.255.0) will map any VMs with local IP 10.10.1.xxx in IP1, when (createZone ... EXT=IP2  INT=10.10.1.2 MASK=255.255.255.255) will only map 10.10.1.2 inside IP2. Note network differences with MASK and INT with/without zero at the end. Zone network mask defines outgoing policies, and a wrong definition will not impact incoming requests, but will prevent outgoing requests to work.
• when learning how to handle multiple public IP addresses, to forward a dedicated intenet port  from a specific pubic IP to an internal VM's IP/port, do not start with port "22" but with something less critical like 80 or 25. An error on port "22" may prevent SSH access to your hypervisor, forcing you to reboot your full server to gain back SSH access. The other good practice is to launch a batch that will stop your firewall automatically after 10mn "(sleep 600; Fridu-firewall.script stop)&  Fridu-firewall.script start)" like than even if you kill your SSH access, it is only for 10mn

Contact me.

Please let's your comment, and if you find any bug or improvement please let me know.