VMware Data Recovery log analyser
Update: Improved error reporting, so script generates a “Failed” email if the log copy fails.
vSphere 4.1 and ESXi 4.1 are both very bad at letting you monitor what has happened with the VMware Data Recovery backup system. You cannot currently even email the log.
So I have written a log analyser which takes the raw VDR log file (similar to the log file viewable in the vSphere plugin) and works out what has happened since it was last run.
The log you see in the plugin is actually a heavily processed version of the raw log file, which is not remotely readable.
You can download my script for free. It’s a short Perl script that uses “iconv” to do the UTF-16 to UTF-8 conversion needed, and the “sendmail” binary (which is provided by all major email systems) to send the email report.
There is a comment at the top that you should read, and there is a little block of definitions which you should adapt for your setup. But otherwise it should work on any Linux distribution without further modification.
You may use my script for free, but please do not charge anyone any fee, monetary or otherwise, for the ability to use it in any form or derivative.
As an example of the output, here’s a little chunk of the email it sends:
21/12/10 08:22 VM Win2008R2x64 Server Source Ok
21/12/10 08:23 FAIL kanga-az1g10 Execution incomplete Trouble reading files, error -3948 ( vcb api exception)
21/12/10 08:24 VM kanga-hl1g09 Ok
21/12/10 08:31 VM kanga-cl14e10 Ok
21/12/10 08:31 VM kanga-cm7e09 Ok
21/12/10 08:34 VM kanga-bc2e10 Ok
21/12/10 08:56 Operation Verify (Execution unit 1) Ok
21/12/10 08:59 VM kanga-amg Ok
vSphere 4.1 and ESXi 4.1 are both very bad at letting you monitor what has happened with the VMware Data Recovery backup system. You cannot currently even email the log.
So I have written a log analyser which takes the raw VDR log file (similar to the log file viewable in the vSphere plugin) and works out what has happened since it was last run.
The log you see in the plugin is actually a heavily processed version of the raw log file, which is not remotely readable.
You can download my script for free. It’s a short Perl script that uses “iconv” to do the UTF-16 to UTF-8 conversion needed, and the “sendmail” binary (which is provided by all major email systems) to send the email report.
There is a comment at the top that you should read, and there is a little block of definitions which you should adapt for your setup. But otherwise it should work on any Linux distribution without further modification.
You may use my script for free, but please do not charge anyone any fee, monetary or otherwise, for the ability to use it in any form or derivative.
As an example of the output, here’s a little chunk of the email it sends:
21/12/10 08:22 VM Win2008R2x64 Server Source Ok
21/12/10 08:23 FAIL kanga-az1g10 Execution incomplete Trouble reading files, error -3948 ( vcb api exception)
21/12/10 08:24 VM kanga-hl1g09 Ok
21/12/10 08:31 VM kanga-cl14e10 Ok
21/12/10 08:31 VM kanga-cm7e09 Ok
21/12/10 08:34 VM kanga-bc2e10 Ok
21/12/10 08:56 Operation Verify (Execution unit 1) Ok
21/12/10 08:59 VM kanga-amg Ok
Comments
VMware Delete Consolidate Helper Snapshot
Sometimes a VM, particularly a VDR VMware Data Recovery VM can gain a huge snapshot with a name like "consolidate helper". You cannot simply delete this snapshot, VMware will not let you.
Shutdown the guest operating system on the VM, to shut it down.
Once it has completely stopped, add a new snapshot.
Then open the Snapshot Manager (from the right-button menu in the VM index).
Then "Delete All" snapshots.
Don't worry if that times out and produces an error, this can happen but it is just the GUI timing out, it will still delete all the snapshots.
Power on the VM.
Shutdown the guest operating system on the VM, to shut it down.
Once it has completely stopped, add a new snapshot.
Then open the Snapshot Manager (from the right-button menu in the VM index).
Then "Delete All" snapshots.
Don't worry if that times out and produces an error, this can happen but it is just the GUI timing out, it will still delete all the snapshots.
Power on the VM.
VMware Data Recovery (VDR) Setup
This is a description of the setup and use of VMware Data Recovery (known as VDR).
You can only have 1 VDR VM per CPU host.
Each VDR VM can backup 100 VMs at most.
Each VDR VM can have 2 Destinations at most.
In strict terms, a "destination" is a "deduplication store".
Each destination can be 1 Tbyte at most.
Each destination should be a virtual hard disk stored inside the VM.
Do not use "thin" provisioning for destination vmdk disks.
Do not use independent disks for destination vmdk disks.
Set the SCSI id of each destionation vmdk to be (1,0) or (2,0) or (3,0) and so on. The second digit must always be 0.
Before you start you must install the "VMwareDataRecoveryPlugin.msi" program. Download the msi file from datastore "ugstore1-vol1"/ISOs onto your PC, quit the vSphere Client and double-click on the file to install it. Restart the vSphere Client and on the bottom row of "Home" you will now have the VMware Data Recovery solution.
You can then connect to any one of the VDR VMs.
There are currently 2 VDR VMs:
The actual backup and restore operations are pretty intuitive, the on-line help will answer most of the obvious questions.
The backup time window is shifted from the default to not backup during 9am-7pm Mon-Fri, and 11am-9pm Sat-Sun.
The Reports and Configuration tabs will show you what is going on, any warnings and errors, and how much space is available in the destinations deduplication stores.
The VDR system checks the integrity of the destinations daily automatically.
When looking at the list of destinations in the Configuration tab, the line for each destination will list a low figure for free space, but a very different figure for the "Deduplicated Size" at the bottom of the pane.
The VDR system automatically consolidates identical and unused blocks daily automatically.
Creating a New VDR VM
You need a Virtual Appliance (ovf file) from the download of "VMwareDataRecovery-....-i386.iso" which is on ecsvm-admin in Administrator's Downloads directory. Mount this iso as a new drive letter using MagicISO (installed on ecsvm-admin) or Daemon-Tools.
Using vSphere Client, File / Deploy OVF Template... and point it to the file \VMwareDataRecovery-ovf-i386\VMwareDataRecovery_OVF10.ovf.
Set the VM's name to something useful like "VMware Data Recovery UG1", and put the location somewhere sensible.
Set the host to a CPU server that doesn't already have a VMware Data Recovery VM on it, there must only be 1 VM per server.
Set the datastore to the place you want all the backups to go.
Use Thick Provisioned format.
Set the Destination Network appropriately for the IP you have.
Set the Timezone to Europe/London.
Finish.
Edit the settings to add a new Hard Disk, which will become the destination/deduplication store.
Make it thick provisioned, SCaSI id (1:0) or (2:0) and so on (2nd digit must be 0).
In Options / VMware Tools, tick "Synchronize guest time with host".
You can install the VDR VM from the OVF file on the vSphere iso image which you can download from your account on www.vmware.com.
Power on the VDR VM. Remember to place it on the correct host (only 1 VDR VM per host).
Open a console window on it.
Once it has booted, set the network configuration to use a static IP.
The username is "root" and the default password is vmw@re.
As soon as you have configured the networking, login using the console window and change the root password to something secret.
Now use the vSphere Client Home / Solutions and Applications / VMware Data Recovery to set it up.
It will walk you through a wizard.
Format the Physical Disk which is the VMDK you created earlier. It will mount it automatically.
Create a new backup job.
If you are trying to select to backup another VDR VM, open the tree on that VM and only backup the 5GB VMDK and not the whole VM!
Set the times so it is *not* backed up 9am-8pm Mon-Fri and 11am-10pm Sat-Sun.
Set the number of backups to keep to something sensible, which may be less than the supplied defaults. We use 6,4,3,0,0.
Manually run a first backup by right-clicking on the job name and using the popup menu.
You can only have 1 VDR VM per CPU host.
Each VDR VM can backup 100 VMs at most.
Each VDR VM can have 2 Destinations at most.
In strict terms, a "destination" is a "deduplication store".
Each destination can be 1 Tbyte at most.
Each destination should be a virtual hard disk stored inside the VM.
Do not use "thin" provisioning for destination vmdk disks.
Do not use independent disks for destination vmdk disks.
Set the SCSI id of each destionation vmdk to be (1,0) or (2,0) or (3,0) and so on. The second digit must always be 0.
Before you start you must install the "VMwareDataRecoveryPlugin.msi" program. Download the msi file from datastore "ugstore1-vol1"/ISOs onto your PC, quit the vSphere Client and double-click on the file to install it. Restart the vSphere Client and on the bottom row of "Home" you will now have the VMware Data Recovery solution.
You can then connect to any one of the VDR VMs.
There are currently 2 VDR VMs:
- Teaching / Backups / VMware Data Recovery UG1 - DNS name ecsvm-vdr-ug1 - backs up teaching VMs
- Systems / VMware Data Recovery Infra1 - DNS name ecsvm-vdr-inf1 - backs up infrastructure VMs
The actual backup and restore operations are pretty intuitive, the on-line help will answer most of the obvious questions.
The backup time window is shifted from the default to not backup during 9am-7pm Mon-Fri, and 11am-9pm Sat-Sun.
The Reports and Configuration tabs will show you what is going on, any warnings and errors, and how much space is available in the destinations deduplication stores.
The VDR system checks the integrity of the destinations daily automatically.
When looking at the list of destinations in the Configuration tab, the line for each destination will list a low figure for free space, but a very different figure for the "Deduplicated Size" at the bottom of the pane.
The VDR system automatically consolidates identical and unused blocks daily automatically.
Creating a New VDR VM
You need a Virtual Appliance (ovf file) from the download of "VMwareDataRecovery-....-i386.iso" which is on ecsvm-admin in Administrator's Downloads directory. Mount this iso as a new drive letter using MagicISO (installed on ecsvm-admin) or Daemon-Tools.
Using vSphere Client, File / Deploy OVF Template... and point it to the file \VMwareDataRecovery-ovf-i386\VMwareDataRecovery_OVF10.ovf.
Set the VM's name to something useful like "VMware Data Recovery UG1", and put the location somewhere sensible.
Set the host to a CPU server that doesn't already have a VMware Data Recovery VM on it, there must only be 1 VM per server.
Set the datastore to the place you want all the backups to go.
Use Thick Provisioned format.
Set the Destination Network appropriately for the IP you have.
Set the Timezone to Europe/London.
Finish.
Edit the settings to add a new Hard Disk, which will become the destination/deduplication store.
Make it thick provisioned, SCaSI id (1:0) or (2:0) and so on (2nd digit must be 0).
In Options / VMware Tools, tick "Synchronize guest time with host".
You can install the VDR VM from the OVF file on the vSphere iso image which you can download from your account on www.vmware.com.
Power on the VDR VM. Remember to place it on the correct host (only 1 VDR VM per host).
Open a console window on it.
Once it has booted, set the network configuration to use a static IP.
The username is "root" and the default password is vmw@re.
As soon as you have configured the networking, login using the console window and change the root password to something secret.
Now use the vSphere Client Home / Solutions and Applications / VMware Data Recovery to set it up.
It will walk you through a wizard.
Format the Physical Disk which is the VMDK you created earlier. It will mount it automatically.
Create a new backup job.
If you are trying to select to backup another VDR VM, open the tree on that VM and only backup the 5GB VMDK and not the whole VM!
Set the times so it is *not* backed up 9am-8pm Mon-Fri and 11am-10pm Sat-Sun.
Set the number of backups to keep to something sensible, which may be less than the supplied defaults. We use 6,4,3,0,0.
Manually run a first backup by right-clicking on the job name and using the popup menu.