A Week of Upgrades [4/4]: Hyper-V Backup Script

[sc:windows-category ]The first three parts of this series were all about updating the software I run at home, this last part of the series will instead be about a backup script I modified for the VM servers.

So first of all, let me walk you though the manual method I have been using to make backups of the Hyper-V VM’s:

  1. Connect a USB drive to my workstation
  2. Mount a TrueCrypt volume on the USB drive and share it from the workstation
  3. Connect to the Hyper-V server through RDP
  4. Shutdown the VM (either though Hyper-V Manager or for the Linux boxes, through a console logon)
  5. Open an Explorer window and copy the VHD to a temporary directory on the server
  6. Restart the VM
  7. Copy the backup VHD across the network using Explorer
  8. Rinse, wash, repeat for each VM

This is not a fast process and has multiple steps that can take a lot of time to complete (mostly the copies) which makes is prone to taking longer than it should as I get working on something else and forget to check the progress.

After poking around the net a bit I found this article with a script that backs up a VM through PowerShell.  It’s not too bad but it does have a few things that needed changing for my own uses:

First off the script is in two parts, a calling script (backup.ps1) and the core script that actually does the backup (function.ps1).  I modified Backup.PS1 in two ways:

$script_dir = “D:\VMBackup”

$bdate = get-date -format “MMM dd yyyy”

$guest = “VM01”
$dest_server = “\\desktop\k\$guest\$bdate”
. “$script_dir\function.ps1”

$guest = “VM02”
$dest_server = “\\desktop\k\$guest\$bdate”
. “$script_dir\function.ps1”

The first change is to set $bdate using the get-date cmdlet so I don’t have to manually edit the script each time I want to run the script.

The second change is to move setting the $dest_server variables from function.ps1 to backup.ps1 so each VM can be stored in a different directory.

Function.ps1 had some more significant changes:

##
## Create a backup of the VM defined in the variable $guest
##

write-host “Backing up $guest…”
$temp_dir = “d:\VMBackup”
$VM_Service = get-wmiobject -namespace root\virtualization Msvm_VirtualSystemManagementService

$VM = gwmi -namespace root\virtualization -query “select * from msvm_computersystem where elementname=’$guest'”

$VMReturnState = $VM.EnabledState
$VMName = $VM.ElementName

if (($VM.EnabledState -eq 2) -or ($VM.EnabledState -eq 32768) -or ($VM.EnabledState -eq 32770))
{
write-host “Saving the state of $VMName”
$VM.RequestStateChange(32769)
}

while (!($VM.EnabledState -eq 32769) -and !($VM.EnabledState -eq 3))
{
Start-Sleep(1)
$VM = get-wmiobject -namespace root\virtualization -Query “Select * From Msvm_ComputerSystem Where ElementName=’$VMName'”
}

if ([IO.Directory]::Exists(“$temp_dir\$VMName”))
{
[IO.Directory]::Delete(“$temp_dir\$VMName”, $True)
}

write-host “Exporting the VM…”
$status = $VM_Service.ExportVirtualSystem($VM.__PATH, $True, “$temp_dir”)
if ($status.ReturnValue -eq 4096)
{
$job = [Wmi]$status.Job
while (!($job.PercentComplete -eq 100) -and ($job.ErrorCode -eq 0))
{
Start-Sleep(1)
$job = [Wmi]$status.Job

$i = $job.PercentComplete
write-progress -activity “Export in progress” -status “$i% Complete:” -percentcomplete $i
}
}

## Re-start the VM.
write-host “Restarting $guest…”
$VM.RequestStateChange($VMReturnState)

## Remove any old file that exist in the destination.
if ([IO.Directory]::Exists(“$dest_server\$VMName”))
{
write-host “Found existing backup at destination, deleting it…”
[IO.Directory]::Delete(“$dest_server\$VMName”, $True)
}

write-host “Removing backup VHD’s from the export…”
remove-item(“$temp_dir\$VMName\Virtual Hard Disks\* – Backup Disk.vhd”)

write-host “Copying backup to destination…”
Copy-Item “$temp_dir\$VMName” “$dest_server” -recurse

write-host “Deleting local copy…”
[IO.Directory]::Delete(“$temp_dir\$VMName”, $True)

write-host “Backup of $VMName complete!”

First off I replaced the echo commands with write-host, the original intent of this was to add the -no-new-line option in the export loop so instead of printing the percentages one per line I could add them together with some “…” between them.

Of course doing some additional poking around I found the write-progress cmdlet which gives a nice progress bar instead so I replaced it and reduced the sleep time from 5 seconds to 1 second.

In the original article they do mention moving the VM restart to right after the export and I agree this is a good idea, no sense waiting for the network copy to finish.

I added in a few extra status messages just to make sure things were progressing.  The next item I changed was the last if/else statement in the original script.  It was there as a way to handle existing backups, if one existed it renamed the old folder, did the copy across the network and then deleted the old folder.  However at no point did it do any error checking so it would never exit out, effectively making the rename a redundant item.  Instead I simply checked for the existence of an old backup and if so deleted it.  The rest was common after that so I removed the else.

I also renamed the old $dest variable to $temp_dir to make it clear it was only a temporary export location.

Another item I changed was the destination location.  My external USB drive directory structure is setup as \HOSTNAME\DATE\FILES, so I can have multiple backups of the same server.  The original code made that \HOSTNAME\DATE\HOSTNAME\FILES.  I removed the second redundant hostname entry from the script logic.

The final item I added was the “backup” VHD deletion from the export.  On the Windows servers BM’s I have at least two VHD attached, one for the OS and one for Windows Server Backup to use.  This second VHD is always called “SERVER – Backup Disk.vhd” and as I’m pulling a “clean” copy of the VHD I don’t really need the Windows Server Backup volume so I delete it before moving the export to the USB drive.

This script is pretty good and now my process for backing up the VM’s is much simpler:

  • Connect a USB drive to my workstation
  • Mount a TrueCrypt volume on the USB drive and share it from the workstation
  • Connect to the Hyper-V server through RDP
  • Execute d:\VMBackup\backup.ps1

This process is fully automated and I can let it run over night.  Likewise it only keeps the VM’s down for a short while during the export and then brings them back up as soon as possible.

The only issue I have with it is the copy of the export to the USB drive, copy-item provides NO feedback, it’s just a black hole until the copy is complete.  For large files like the VHD’s this really isn’t a very good thing.  In the manual method the copies provided feedback through the standard Windows Explorer copy dialog.

I believe I have a solution to this, however it is only for one file at a time and doesn’t recurse through the export directory so it will take a bit more work to build the recursion in to it.  As I’m now done all of the backups for this round, it will probably wait for a few more weeks until the next time I run the VM backups.

A Week of Upgrades [2/4]: Windows Server 2012 Domain Controller, DFS and SQL Upgrades

[sc:windows-category ]Yesterday I upgraded my Hyper-V servers and now its time to do the domain controllers, DFS servers and SQL server.

As always, backing up the system is a must and as my DC’s are all VM’s, I  simply shutdown the systems and copied the VHD’s to a safe location.

After that the next step was to run ADPREP, as always getting the forest up to date before installing the first 2012 DC was simple enough.  No issues were reported when running ADPREP.

As with the Hyper-V servers, I had to un-install my anti-virus software.

As the DC’s are pretty clean installs I decided to do an in place upgrade to Server 2012 and after running setup a few issues were identified:

  1. Lack of disk space.
  2. Virtual Graphics Driver

The disk space is kind of surprising, admittedly I only gave the virtual systems a 32g system partition, but it did have over 8g of free space.  Setup though wants 13g of space before it continues.  I shutdown the VM and used the disk wizard in Hyper-V to expand the virtual disk to 48g.  Then after rebooting the server, used disk manager to extend the system partition to fill the entire 48g.

To resolve the virtual graphics driver I assumed it would be just an update to the new virtual machine extensions from Hyper-V 2012.  A quick reboot and a rerun of setup though brought me back to needing another reboot.  After that the warning was still present but I proceeded anyway.

The upgrade proceeded pretty much as with the Hyper-V servers, once again sitting on the “Getting Ready…” screen for quite a while but eventually moved on.  As I was doing the upgrade later at night, I simply let it run overnight and in the morning I was greeted with a completed install.

I didn’t receive any issues with the video driver, everything was fine after logging in so it seems safe to ignore the warning.

The second domain controller proceeded in the same way and completed without issues as well.

The final step was to move the forest and domain up to the 2012 functional level, which completed without issue.

Next up are the DFS file servers and the MySQL server.  These are pretty straight forward and are all VM’s.  Other than extending the disk space, proceeded without issue.  After the upgrades to the DFS servers I did have to reboot my desktop as connecting to any of the shares on the servers came up with an access error.  After the workstation reboot all was fine again though.

One item of note is licensing, I installed Windows Server 2012 Data Center edition on the Hyper-V servers and it comes with unlimited virtual instance licensing.  Of course this means you have to install the same edition of Windows on the VM’s as you have on the physical host.  The one issue I can think of with this is that you have to use the same activation key as on the physical host, which means moving servers between host servers requires an update to the activation key on the VM’s.

The only minor glitch on any of these systems after the upgrade came up on two of them.  When installation was complete and I logged on for the first time, instead of the light blue theme, a light brown theme was selected.  There doesn’t seem to be anything in common on the systems (One was a DFS server, the other was the SQL server).

Overall, a very easy upgrade, now on to other things.

A Week of Upgrades [1/4]: Windows Server 2012 Hyper-V Upgrade

[sc:windows-category ]I recently spent some time (the better part of a week) upgrading my servers at home, what follows is a 4 part series (one part each day this week) of articles around those upgrades. The first part here is about upgrading my Hyper-V host servers to Windows Server 2012.

Having upgraded my desktops to Windows 8 it is now time to move to Hyper-V 2012 as well so I can manage the VM’s without having to RDP in to the servers.

Currently I have two Hyper-V servers running, each hosts several VM’s which are a mix of Linux and Windows.  I’m running OpenSUSE for Linux and mostly Windows Server 2008 R2 Standard for the Windows boxes.  The Hyper-V hosts themselves are 2008 R2 Enterprise.

With Windows 2012 some of the licensing for virtual machines has changed so I’m installing 2012 Data Center on the Hyper-V hosts.  And before anyone asks, yes I’m installing the GUI as well.  I understand from a logical perspective as to why Microsoft is moving to a GUI less server environment but for small IT shops it just doesn’t make sense.  I can only hope they realize this and continue to offer the GUI on server option in future versions.

I’ve chosen to do an in place upgrade on the Hyper-V hosts because I keep these boxes very clean.  They only thin on them is Hyper-V, all other services I run inside the VM’s.  This didn’t use to be the case when I was running VMware Server, as I had the host servers running the VM’s as well as being Windows Domain Controllers.  However Hyper-V doesn’t allow that so when I moved to the Hyper-V (which turned out to be a good idea as VMware has since stop development on their Server product) I made a conscious effort to keep the host servers as simple as possible.  This actually saved me once about a year ago when the OS drive on one of the host severs failed and I had to re-install, with so little on the physical box it was only a little over an hour before I had it up and running again.

Of course the first thing to do is to backup all the VM’s I’m running.  I really don’t have a good way to do this, between the mix of OS’s and not enough time to find a solution, I just shut down the OS’s and copy the VHD’s to an external disk.

This isn’t the fastest process and it takes a couple hours of manual work to do, but it does make a nice clean break point to go back to if I need to for some reason.

After searching the Net for anything I should look out for only one thing came up, which was to shutdown the VM’s before the upgrade.

Shutdown was easy and then I copied the 2012 install files to the local disk and ran setup.  Setup looks pretty much the same as always and getting through to the upgrade portion was easy enough.  When it checked for any issues three items came up:

  1. Incompatible Anti-virus
  2. Incompatible Intel Pro-Set
  3. Warning about a physical disk attached to a VM

I kind of expected my anti-virus to be incompatible, it looks like the vendor isn’t updating it for 2012 so I simply uninstalled it.  My gut feel is I won’t install another AV product since I’ve only every had false positives from the server side.  I may do some more research though and see if I can find a replacement.

The Intel Pro-Set software didn’t surprise me but did cause the most problems.  Uninstalling it broke the Hyper-V virtual networking.  First I had to go in to device manager and search for new devices (after install Windows only picked up one of the two network cards) and then re-assign the virtual network to the physical card again.  After that I had to re-enter the IP address for the server as that had been lost in one of the changes.

The warning about the physical disk is a little interesting.  My main file servers (one on each Hyper-V host) have direct attached physical 2tb disks to the VM’s.  I did this for two reasons:

  1. A virtual 2tb vhd seems kind of iffy…
  2. I moved the disks from physical servers to VM’s, so simply attaching them to the VM’s meant I didn’t have to copy all the data around.

The upgrade proceeded with a couple of reboots and a final restart, there were no issues and it only took about an hour to complete.  During one of the reboots Windows did stick on the “Getting Ready…” screen for about 10 minutes, but finally did progress.  Overall it was painless, but during the reboots you never knew what was going on, you just had that blank “Windows” screen with the twirling dots.

I’ll admit its a little weird seeing the Start screen when you logon, but other than that the upgrade worked without issue.

The physical disk did have to be reconnected to the VM before I restarted it, but that was simple enough.  All of the other VM’s came up without issue.

Of course I couldn’t just leave well enough alone and decided to do a firmware upgrade to the servers as well, which promptly eat both the networking and disk configuration 🙁

In Windows the network card had changed id’s, which meant the Hyper-V virtual adapter was no longer able to find what it was supposed to use for network access.  Loading up the Virtual Network configuration allowed me to re-assign the correct adapter but as Windows had assigned a DHCP address to the new adapter, Hyper-V wiped out my previous IP config and used DHCP instead.  Simply re-applying the correct IP address solved this problem.

The disk configuration was a little more strange as the physical disk that I had assigned to one of the VM’s had come back online and been assigned a drive letter on the Hyper-V server.  To be used inside a VM a disk must be offline.  Taking the disk offline and assigning it back to the VM solved this as well.

In an unrelated issue (which caused a bit of extra work) that I’ve had with my two servers is around the KVM I’ve been using.  Seemingly randomly on reboot the keyboard/mouse will stop working.  This has been an annoyance for quite a while but never significant enough of an issue for me to track it down.  However to upgrade to 2012 I do need the KVM to work well and it was being very uncooperative.  I decided to rip out the KVM and direct connect the keyboard and after that I still had issues.  This must be BIOS related and I’m hoping the upgrades will resolve it.

The second unrelated issue I had during the upgrade is that the motherboards I’m using have an option for a hardware remote management on them which I’ve installed.  The first server I connected to with IE 10 from my Windows 8 PC and IE simply refused to connect due to the security certificate.  I poked around a bit but didn’t find a resolution so I simply used Firefox instead.  The second server however was not responding on the management IP.  Reconfiguring the IP addresses didn’t work and I finally resorted to re-flashing the BIOS on the management board.  This resolved the issue and the rest of the Server 2012 proceeded without issue.

Now that both Hyper-V servers are updated I’ll be doing the rest of my servers in the following order:

  • Windows Domain Controllers
  • DFS servers
  • MySQL server (on Windows)
  • VPN servers (run OpenVPN appliances)
  • Mail Gateway (currently on OpenSUSE 12.1, but really need an upgrade to 12.2)
  • Exchange Server (both OS to 2012 and Exchange to 2013)

The Exchange upgrade will be the most challenging, I really should setup two new servers and configure them for redundancy but that seems like a lot of work and of course there’s the SSL certs to mess with 😉

Tomorrow I’ll post the domain controller, DFS and SQL upgrade entry, come back then to find out how it went.

 

 

Windows Phone 8 Media Sync

[sc:mobile-category ]Ok, time to take off the gloves, let’s talk about the biggest hole with Windows Phone 8… syncing music!

First things first, Windows Phone 7 came with the Zune music client which managed all the syncing to the phone.  Of course there was no other way to do it so everyone had to use Zune.

Windows Phone 8 takes a step forward with the fact you can now use pretty any music client to sync your music to the phone, including just good old File Explorer.

The problem is that while that step forward is good, it comes with a massive leap backwards by dropping the Zune client for two new clients.  One for the traditional Windows desktop and one for the new Windows 8 Apps.  Both of these clients are badly broken.

Lets start with the Windows 8 App, you can move music and photos over to the phone, but you cannot create or manage new playlists in it.  Likewise it only talks to the Windows libraries.  On one PC I have it won’t even talk to the phone at all.  It says to reboot the phone to resolve the problem but it doesn’t.  It has no settings and doesn’t pick up the album covers or existing Zune/Media Player playlists.

The Windows Desktop Client is a beta, when it starts up it has to scan through your entire library of music, pictures and videos.  On my system this takes over 20 minutes and it does it each time you load it.  It can at least understand the Zune/Media Player playlists, but trying to sync them runs all the way through to the end when the playlists get synced and then the whole app crashes.

I eventually resorted to using Windows Media Player to sync across my playlists, which works, but WMP isn’t the best app for this but at least it worked.

I have to admit that I actually LIKE the Zune client, it was slick and provided a much nicer interface than WMP.  I hope Microsoft brings it back in some form, perhaps as a replacement to WMP.

It’s so bad that I’m almost at the point that I’m considering writing my own player/sync client.  That will be a last resort though and I’ll live with it like is for the time being.

Microsoft Surface Pre-Order Update

[sc:hardware-category ]In a previous post I detailed the ordeal I went through with my pre-order of the Microsoft Surface RT tablet.  At the end of that post I mentioned I was still waiting for my escalation on the $50 promo code and now I can finally relate the rest of the story.

Just as a quick refresher, I pre-order my Surface on Oct 16, 2012 for delivery by the 26.  It didn’t arrive until Oct 30 and Microsoft was dishing out $50 promo codes to those that didn’t receive their orders on the 26th.  I had two problems with the promo code, first it meant I had to purchase something to get it and second it really wasn’t enough.

On the Monday when I was still waiting for the Surface to arrive I had requested an escalation on the promo code and was told I would expect back an answer from their tier 3 group within 72 hours.  I didn’t receive anything and I called in again on the Thursday to follow-up.  Apparently they had sent out an e-mail to me on Wednesday afternoon but it didn’t get to my inbox.

They sent a follow-up e-mail and after a few more days of back and forth on Monday Nov. 5th they offered me a $200 rebate on the purchase.

I probably could have held out for more, but I have to admit the person from tier 3 support was very good and didn’t try to generate false sympathy or any other tricks to get me to cave to the $50 promo code so I decided it satisfied all the issue I had and accepted it.

I have to say that the pre-order experience was terrible and the support after the missed date was worse.  However at the end of the day I did receive the table a few days late but managed to get an almost %40 discount on it so it all worked out.

Will I be buying a pre-order from the Microsoft Store again?  I don’t think so, as there is a new Microsoft store opening soon close to me.