Virtualization for (heavy) home use


I have been a big fan of virtualization since the day one; it's fun to try every OS (Operating System) on the planet and with virtualization it is dead simple (HP-UX sadly not though).
The benefits of the virtualization for the corporations are undeniable:
  • savings: consolidation of hardware and licenses too plus savings in support & maintenance
  • easy High Availability (HA) like setups
  • environmentally friendly option (less hardware & energy efficient) 

There are some pitfalls to virtualization but I will not talk about those - please read my blog post about software licensing.

Some background

My virtualization story started with VMware Workstation in 2001. Since then I have used or still use many of the following virtualization software:
  • VMware workstation starting from version 3.1. Nowadays version 9.0, at work for minor tests, studying and development platform (PHP, Python, Bash scripts, PosgreSQL, MySQL, etc)
  • VMware Fusion... kind of help desk to few fruit fanatics
  • VMware Player, brilliant for quick and dirty trials
  • VMware ESXi, at home and work for quick'n'dirty single VM test setups like centralized management tests (SSO, patch management, monitoring, etc)
  • VMware ESX, at work we operate four DC's consisting of 16 hosts (รก 12 cores, 192-296Gb) so ESX is very familiar 
  • (Oracle) VirtualBox, at home and work since it became available in Ubuntu repos... a long time ago
  • Citrix XenServer Free, at home until 2011 but stopped using because of the XenCenter being available only on Windows and the need to acquire serial annually
  • KVM & virt-manager, at home until 2012. OK for quick'n'dirty setups but lacks functionality if serious about virtualization 
  • Hyper-V, at home lab 2011... trashed few of my test VM's and haven't wasted my time ever since. Could try again but please be informed that Hyper-V has nasty habit of on/off supporting features... like NSF support. Unfortunately typical MS behaviour.
  • RHEV/oVirt, I participated RHEV (RedHat Enterprise Virtualization) Deep Dive training 01/2013 and wanted to test it a bit more. IMHO: I really find the web UI illogical mess that would urgently need some fixing
  • plus some EOL products like VMware Server, etc

Current setup

I currently have a lab setup at home with single 8-core ESXi host with 16Gb of RAM and 2 x 1Tb RAID1 disk setup. There is a cold backup ESXi host (4-core) acting as 'a continuity plan'. ISO images are located on NFS share on a NAS box. Actual data is located on NFS share (RAID 1 setup) with a mirror on another NAS.
On primary laptop I use VMware workstation and Oracle VirtualBox on all other laptops and desktops (too many of them to list here)
This setup exists for self learning and testing reasons as well as home automation and media server purposes.


This time I wanted to go the whole nine yards with the home lab virtualization without any compromises - this is what I want:

Mandatory requirements (aka showstoppers)
  1. Free, but of course :-) (If this works then it may be implemented at work with paid support - support the cause, don't just free ride). (Actually plain MS Hyper-V fits this requirement but not the rest... please read on). Bottom line is that I kind of dislike the whole modern licensing concept...
  2. Live motion/migration of VMs within hosts/nodes. Swift live (=online) migration functionality will require shared common storage which is OK. This does not need to be automatic HA (High Availability) setup though.
  3. Cloning, templates, snapshots, (backup).... the usual virtualization goodies
  4. Linux native client for management or fully functional web UI without browser restrictions (pun intended RedHat for RHEV 2.0 IE/ActiveX requirement & Hyper-V)
Fun to have requirements
  1. Android management app would be nice but not a showstopper
  2. Possibility to allocate USB devices to Virtual Machines (Tellstick, USB modem, etc) - this will not work as HA solution but it would be fun to have
  3. No annual serial key update issues (pun intended Citrix XenServer free)
  4. Some way to utilize my Icinga (Nagios fork) monitoring system on the actual platform nodes (CLI, API or SNMP will do)
I want to setup virtualization lab that consists of two identical dual-core desktops with 8Gb of memory and network setup - both hot. The hardware specs are far from bleeding edge but I only intend to run max 6 Linux server (CentOS6.X 64bit x 5 + Ubuntu x 1 + perhaps FreeBSD x 1) installations without graphical UI.
Minor warning: Linux or any *nix (without GUI) are not resource hungry but as the smallest allocation 'particle' is CPU core (aka vCPU) this virtualization farm will be way overallocated. That's fine as some of the the virtual machines will perform task during nighttime and others during daytime - but basically the design is all wrong as single physical node has no way of maintaining full load in case of hardware failure on second node.

Tested software

This is blog post is the 'management' version; I will not go into the boring details about the overhead,  RAM & CPU restrictions, etc. In a small setup like mine those details have no meaning.

I am leaving plain operating system-level virtualization technologies like Solaris & HP-UX containers (including Solaris Zones) and LXC (LinuX Containers) out of this comparison. Also Open Source Xen is left out as I want to have easy management UI from the same vendor (although there are good 3rd party options like Nuxis available). Hyper-V is left out also because of MS only management requirement.

So here are the final contestants:

Proxmox VE 2.3-12

Thanks to my colleague Kalle for telling me about this wonderful but rather unknown software.
This software is Open Source (AGPL) and free to use even on production environments, but you need to pay to get other than community support. The 'unsupported' version is a full version so you get all the goodies.

  • shared NFS/CIFS are super easy to add (scans paths very nicely). Once shares are added the software creates directory strucuture according to the user specs (ISO, VM, backup, template, etc shares)
  • the best (=easiest) installer of them all - installation CD/USB in and press Enter
  • heavenly UI - Can't get any better than this. I can only find one illogical feature which concerns how the shares are defined; you need to remember to define one or more use sceantios for the share (like ISO images, ISO, VM, etc)
  • one web UI to handle all the nodes. No Active-X or any other platform dependencies
  • supports KVM and OpenVZ for Linux - read more here
  • super easy backup functionality (and the best of breed compared to other tested software); select VM's, define timetable & target media
  • console functionality needs Oracle (Sun) Java 6 or 7 version to work correctly, openjdk won't do. I got Epihany web browser to work with these instructions Java_Console_(Ubuntu) and Install-sun-oracle-java-jdk-jre-7-on-fedora. Chrome had some issues on Windows and Linux environments
  • I had serious console resolution/resize/speed issues plus some color depth issues. Spice implementation seems to be on their roadmap so maybe this gets solved soon
  • OpenVZ issues: especially with OpenVZ (Create CT) I ended up with empty zombie VM's several times. The removal of zombie VM's failed with WUI (Web User Interface) and terminal magic on the host/node was required. Also OpenVZ migration from local storage failed and ended up loosing track of the VM(!)
  • network setup differes from other virtualization software and is somewhat challenging for complicated configurations
  • assigning USB devices not as easy as in ESXi
  • full featured virtualization software that fails only on two accounts: poor and restrictive VM console and bugs related to OpenVZ (or at least I had serious issues) 

oVirt (26/3/2013)

oVirt is the bleeding edge version of RHEV (RedHat Enterprise Virtualization) - like what Fedora is to RedHat Enterprise Linux (RHEL).
With the release of 3.x RHEV/oVirt finally got rid of the Windows Internet Explorer requirement due to Active-X dependency in web browser.
This is full featured enterprise virtualization as open source - excellent!

  • royal PITA to install & setup. Node installation is ugly, confusing and failed multiple times for no reason. Engine setup script failed until Firewalld and SElinux were disabled permanately(!). RHEV most likely does not suffer from these shortcomings but then again this test only includes free virtualization software
  • the logic of oVirt (and RHEV) is just impossible. Die hard engineers should really consult UI designers once in a while 
  • management server is something I have issue on my small environment. This is not a Single Point of Failure (SPoF) but I like Proxmox and XenServer solution more where management server is not needed
  • I know I am being nasty... none. Really none if compared to the other virtualization software. RHEV/oVirt offers many bleeding edge features (CPU/RAM/API/SElinux to name few) but there is no need for them in my modest environment. This clearly is a software for large scale virtualization

  • Unnecessarily complicated piece of fabulous software. Can't really recommend for home lab usage. Perhaps next releases will fix what oVirt lacks in the UI logic. Management server requirement is most likely here to stay

VMware ESXi 5.1

This is the feature poor version of the market leader VMware ESX.
One of the biggest issues with VMware has always been the need for Windows specific VMware vSphere client and I was expecting that this was solved with version 5.1 but I was wrong... this is what VMware has to say: The webclient is only available as part of vCenter and not embedded into ESXi directly. So with the free version (or as long as you don't purchase/deploy vCenter), you can't manage your hosts with this webclient.
What a major disappointment!

  • easy installation
  • the most polished UI (statistics, ease of use, logical grouping, etc)
  • assigning USB devices is simple and works very well
  • no (free) centralized UI for separate ESXi servers
  • needs a Windows spesific VMware vSphere Client to manage servers
  • no migrate (can be done manually though), no cloning, etc - only good for quick'n'dirty simple trial setup or for vmware virtual appliance marketplace apps
  • installation of VMware tools is a must for serious virtualization (not mandatory though). VMware tools enhances the console, network etc experience
  • ESXi is too stripped down version and only good for die hard VMware fans. I wonder why VMware does not allow more functionality within a single node/host setup with the new web UI

Citrix XenServer 6.1 (Free edition)

This version is slightly stripped version of the Enterprise version but all-in-all packed with very impressive set of functionality with rock solid delivery (XenServer has never failed me).
The binary is the same for Enterprise and Free versions but the license makes the diffference. Version comparison is here.
You need to retrieve the license file for free usage annually - the license key retrieval is very easy though..

  • really simple setup
  • KISS UI with plentiful of functionality that you learn within minutes
  • the functionality is robust and I have never had any issues with it
  • live migration of VM's within pool is lightning fast when using shared NFS storage
  • well working console and possibility to define and use scaled VNC for Linux VM graphical installations (define graphics utf8 vnc for new VMs)
  • need to fetch activation license file annually
  • (installer is still the old text based and intuitive one (vs the others) but that is only cosmetic issue; people used to Linux text installers will feel at home with this)
  • Windows only XenCenter required to fully administer the farm (there is OpenXenCenter for Linux users with less functionalities - could not get it to work with XenServer 6.1 though and latest release is from 10/2010 and now seems to have vanished from SourceForge. Still available in many Linux repositories for Ubuntu, Fedora and likes). The Windows client also has issues with Wine so no luck.
  • UPDATE: patching process has been improved, but download speeds are really slow. Patching of fresh install takes easily one hour of intensive 'OK' clicking
  • template selection lags behind and some Googling is required when installing latest and greatest linux distributions
  • open source Xen supports direct USB passthrough, XenServer does not
  • rock solid performance and logic. Bug free functionality. Only real issue is the XenCenter which requires Windows OS

We have a winner

So the winning software for my virtualization lab is... drums please.... Proxmox!
It was a really close call between Proxmox and XenServer, but looking back at the scope of this exercise this has to be the right call: XenServer is missing fully functioning native Linux client/web UI and there is the issue of retrieving the annual license file.
Proxmox's network setup is not the best possible (vs XenServer) and there are some minor bugs but otherwise I can't find anything to complain about it.

Highlights why to choose Proxmox:
  • No need for management server (pun inteded RHEV. RHEV-M is not SPoF (Singe Point of Failure) but still annoys). On small environment all non-essential servers are PITA (Pain In The Ass)
  • No annual licences to request and install (Citrix XenServer, get it right finally)
  • Administration UI via browser (VMware... waiting for the web UI to ESXi). You need to have the Oracle JRE installed but that is a small price to pay for native client
  • Free and fully functioning with all major HA features (pun inteded ESXi, it is just too feature poor version)
  • It is Debian based; easy to update and you can modify it in any way you like - consider monitoring and reporting issues
  • API, SSH, rsync, backup, etc as they should be delivered and the possibility to create your own code for the hosts/nodes
  • Brilliant backup feature
... Proxmox's comparison table also makes good reading.

My new environment

My building blocks for this exercise were:
  • Storage; ridiculously cheap Asus M-25 NAS, with 2 x 1,5Tb SATA II disks:
    • ISO storage, snapshots, templates, backup: NFS share on single HD drive (HDD2)
    • VM disk: NFS share on single HD drive (HDD1)
  • Hosts; 2 x Dual-Core Intel machines with 8Gb RAM, 2 x Giga NIC and 500Gb local HD
  • All the usual accessories: KVM switch for keyboard & mouse for nodes, common monitor, 433MHz remote operated power outlet with 550VA UPS setup, Buffalo Nfiniti WLAN gigaswitch with DD-WRT firmware, etc
.. Almost forgot...
One of the 'fun to have' requirements was Android app - there is one rather nice: QuadProx Mobile by Quadrata (free) in Google Play which looks good on smartphone (Samsung Galaxy SII) and tablets (Nexus 7 & 10). With QuadProx you can:
  • see list of nodes and Proxmox versions used
  • view logs and login data
  • see node statistics and virtual machines
  • virtual machine status, configuration, statistics (cpu, memory uptime) and do all the necessary actions (start, shutdown, stop, migrate, view logs) 

(For iPad there is none, for iPhone there is Proxmoxapp (free) which looks rather plain)


This really shows the power of open source; you get fully featured enteprise grade virtualization for free - and if you so desire/can afford you get the support for very reasonable price (support the cause if possible).
This was just a small sample of virtualization software and the offering is evolving rapidly.