monline

I've set up a Debian (5.0.2, Lenny) server using only the stable repository for most stuff and the backports repository for the libvirt related packages (libvirt-bin, libvirt0, python-libvirt and virtinst). All Xen packages are from the stable tree, currently v3.2.1. I've created a few VMs using HVM and Windows 2003 (Enterprise Ed. with SP2) and using image files (/var/lib/xen/images/*.img) as storage devices. At first everything seemed to work Ok. But once I started to use the VMs for disk-intensive applications (like a database server), filesystem corruptions started the appear.

The problem was not trivial, because FS corruption did not occur presistently. Eg. I've started the installer of an application (Siebel Business Applications v8.1.1.0 -to be more specific- that used the already set up database that was running in the VM too) and this installer stopped with an error at a number of different steps (during the various test runs). Sometimes it stopped already at the beginning (creation of a new database with throusands of tables in the DB instance), sometimes it failed only later on ... during loading of the initial data into the database. It was not consistently failing at the same step, but always at a different point.

I also experienced a few BSODs. In two cases the Windows in the VM refused to start (after a reboot) and claimed that some critical file was not found (or the filesystem was corrupt). Eg. once it claimed that "Windows\System32\Drivers\Ntfs.sys" was missing or corrupt. And indeed: I've shut down the VM, mounted the disk image using losetup, kpartx and mount.ntfs-3g and created a checksum of the ntfs.sys file with md5sum. I've done with my template win2003 disk image too (the one that I used as a source for my testing VM) and the checksums were indeed different, thus the file got corrupted.

I've googled a lot, but did not find any recent events/reports of filesystem corruption regarding virtual machines and Xen. However my guts told me that the problem is with heavy I/O in the VM. I've googled for disk stress testing tools for Windows and found Bart's Stuff Test. It turned out that Bart's test could not finish even once (I mean the "Loop" counter) without spotting some filesystem problem.

Using iotop in the dom0 (the Debian host OS) I saw that during the test with Bart's tool two processes generated 99% of the I/O load: one was qemu-dm (obviously ... since QEMU is used for running the VM) and the other was kjournald. Of course, the latter was obvious too since my VM disk images are on an ext3 (journaled) partition in an LVM volume group.

I've googled a bit on this and many posts suggested that VMs should use LVM volumes directly and not image files. I've created a new VM using an LVM volume as storage and started Bart's tool. After 10 loops I've decided that the problem is solved. Apparently QEMU's (at least the version bundled with xen-tools-3.2-1, which is 3.2.1-2) disk emulation has some problem with disk image based storage devices and high I/O.

Another solution might be to upgrade Xen from the testing or unstable repositories. Maybe the problem is already fixed in a newer Xen (and/or QEMU).

Update: the problem turned out to be memory related and at the moment it seems that Xen had nothing to with it. Actually I've tried to use VMware Server with quite similiar results (I mean errors). Luckily (ie. by accident) we had a full crash (including the host OS) at a time, when no virtual machines were running at all ... thus we came to the conclusion that virtualization might have nothing to do with the problems we were experiencing. (To my excuse: we never stress tested the server and virtualization was something we had no extensive experience with ... thus we jumped to the conclusion that the problem was with VT ... and after a long struggle it turned out to be wrong.)

Everything suggests that putting 6 double-sided RAM modules (ie. two of the OCZ 6GB, 1333 MHz, Triple Channel DDR3 Kits (PN: OCZ3P1333LV6GK) that come equipped with some sort of "heatspreaders") into an Asus P6T motherboard without "extra cooling" (at least for the memory) is a bad idea. The system runs stable with any of the two OCZ kits (I mean using only 3 RAM modules), but plugging both kits in the MB at the same time results in about 20 minutes of stress testing in memory glitches (ie. memory testing software starts to signal corruption). And a manual verification (ie. touching the heatspreader of a module) confirms that it's "boiling". At the moment we are running the system with only 6GB and looking into options of adding extra cooling for the memory slots. I already targeted the OCZ XTC Cooler, but unfortunately it's a bit hard to get in Hungary.