Elastix and VMware

Took the plunge today to update my asterisk server. I’ve been using asterisk for about 5 years now and am pretty adept and manipulating its cryptic configuration files but I wanted to move to more of an appliance. I decided to give Elastix a try.

These days I virtualise all my boxes on a VMware Server environment. I got Elastix installed with no problems but then I wanted to get VMware Tools installed. This gives you better network drivers and make sure your clock stays in sync.

Since this requires you to compile some kernel modules you need to have the kernel-devel package installed so you can compile against your current kernel. This would normally be a simple matter of

yum install kernel-devel

However this seemed to do nothing. After a fair bit of investigation I worked out that Elastix ship there own kernel and modules for some asterisk specific hardware like zaptel and rhino. To make sure you don’t use the CentOS kernel they disable that package from that repository.

If you don’t particularly need the Elastix kernel (I don’t since this system will be pure VoIP) you can renable the CentOS modules by editing /etc/yum.repos.d/CentOS-Base.repo and commenting out ball the lines that look like

exclude=kernel*

Update: So it seems that this means that I won’t get the ztdummy module. This module uses the USB chipset to provide timing for some asterisk related things like the multi user conference module. I don’t really use this at the moment it’s not a big deal but I may have to roll my own kernel RPMs later down the track.

VMware Consolidated Backup

The last few months have seen me working at an insane pace at Bulletproof in the lead up to a launch of our latest and greatest product Dedicated Virtual Machine Hosting or DVMH for short. I’ll ramble on a bit more about it after it’s launched but basically it is similar to our existing Managed Dedicated Hosting but running on VMware and with a whole heap of cool features due to the benefits of virtualisation.

Today saw me working with one of these cool features, Consolidated Backup. Basically what this lets you do is have a Windows 2003 server directly plugged into the SAN which can directly see all the VM images sitting in the VMFS LUNs. It then talks to the ESX servers takes a snapshot and makes a copy of it t local disk. Hey presto Disaster Recovery. Well mostly anyway, the restoration aspect isn’t all that crash hot as you’ll see below.

Documentation on performing the backups is a bit scarce. VMware provide some scripts that let you tie it in to some commercial backup products like Legato, Veritas and NetBackup but no real docs on how to do it yourself.

So here are some quick examples. (You can find all these commands in C:Program FilesVMwareVMware Consolidated Backup Framework

Getting a list of VMs on your ESX farm.
[code]
vcbVmName.exe -h VC_HOST -u USERNAME -p PASSWORD -s any:
[/code]

Backing up a VM
[code]
vcbMounter.exe -h VC_HOST -u USERNAME -p PASSWORD -a moref:MOREF -r DESTINATION -t fullvm -m san
[/code]
where MOREF comes from the list you created above and DESTINATION is a local path on your VCB proxy.

You should then strictly unmount it by doing
[code]
vcbMounter.exe -d DESTINATION
[/code]
but I don’t think this does anymore than delete the files, since the snapshot on the ESX server has already been closed.

The above creates something like this
[code]
catalog
MyVM.nvram
MyVM.vmx
scsi0-0-0-MyVM-s001.vmdk
scsi0-0-0-MyVM-s002.vmdk
scsi0-0-0-MyVM-s003.vmdk
scsi0-0-0-MyVM-s004.vmdk
scsi0-0-0-MyVM-s005.vmdk
scsi0-0-0-MyVM.vmdk
unmount.dat
vmware-1.log
vmware-2.log
vmware-3.log
vmware-4.log
vmware-5.log
vmware.log
[/code]

Mounting a VM image locally
[code]
vmmount.exe -d VMDK -cycleId -sysdl LOCATION
[/code]
VMDK needs to be scsi0-0-0-MyVM.vmdk from above.

You can then unmount it by doing
[code]
vmount.exe -u LOCATION
[/code]

This is nice and easy and really useful means you can now easily backup everything to tape.

Recovery is another matter entirely, apparently in the Beta releases vcbRestore was distributed with Consolidated Backup but in the final release it now only exists on the ESX servers. So you need to move your directory above to one of your ESX boxes. You then do

[code]
vcbRestore -h VC_HOST -u USERNAME -p PASSWORD -s DIRECTORY
[/code]

This will totally replace your existing VM, if you wanted a copy then you should copy the catalog file elsewhere, edit it to change the paths and

[code]
vcbRestore -h VC_HOST -u USERNAME -p PASSWORD -s DIRECTORY -a CATALOG
[/code]

There are a couple more features I haven’t mentioned which you can work out for yourself by using -h. eg File level backups for Windows VMs.

Now all of the above is great but VMware have taken things a step further. With the above if your VM is running VMware Tools the equivalent of a sync is done before the snapshot is taken which effectively gives you slightly better than a crash consistent dump. Though you could still lose some DB data.

So VMware have added some functionality to rectify this. Just before the snapshot is made /usr/sbin/pre-freeze-script or C:Windowspre-freeze-script.bat is run and /usr/sbin/post-thaw-script or C:Windowspost-thaw-script.bat are run afterwards. Taking a snapshot only takes a few minutes so you could use these scripts to stop your database for example.

I highly recommend reading the VMware Consolidated Backup manual for all the extra features I haven’t covered.