Building a new VM server Part 1: Intro and Hardware

[sc:hardware-category ]At the moment, my servers at home consist of two VMWare host servers and several VMs on each. The hardware specs are:

  • Server 1: Quad core 3.2 ghz AMD processor with 8 gig of RAM. 3 250g disks (OS and VM partitions) and 1 2tb drive for file storage.
  • Server 2: Dual core 2.2 ghz AMD processor with 8 gig of RAM. 3 250g disks (OS and VM partitions) and 1 2tb drive for file storage.

Both servers run Windows Server 2008 R2 with VMWare Server 2.0.2 for the Hypervisor.

The file storage disk is currently shared and replicated between the two Windows servers using DFS for redundancy.

The current workload for these servers has pretty much stretched them to their limits and it’s time to upgrade to something with a little more oomph.

And so…

The Plan

My mandatory upgrade criteria includes:

  • At least 4 cores
  • 16 gig of RAM
  • Standard ATX form factor
  • AMD based

On the nice to have list includes:

  • Remote KVM management
  • Two processor slots

I usually build my home servers from standard PC motherboards, but currently most of those top out at 16 gig of RAM so I’ve had to expand my search a little bit.

I’ve been using AMD processors for many years now and my preference to keep them now has more to do with ensuring all the VM guests transition smoothly more than anything else.  However this may break this time as it looks like I’ll be moving to the Opteron processor.

VMWare server will likely be making an exit at this time as VMWare seems to have pretty much abandoned updating the product and the limitations on how many processors it can support.  Which leaves me choosing between ESXi (now renamed VMWare vSphere Hypervisor), MS’s Hyper-V or the various Linux based virtualization solutions.

The Motherboard and Hardware

After looking around quite a bit, I’ve settled on the ASUS KCMA-D8 motherboard, some of the reasons are:

  • Standard ATX form factor
  • Standard ATX power supply
  • Support for greater than 16g RAM
  • Remote KVM support with optional module
  • 2 processor support with 4 or 6 cores

Of course finding the MB and buying the MB turns out to be two completely different things.  The vendor I was looking to purchase it from turns out to not have ANY CPU heat sink and fan that supports the Opteron 4100 series processors.  I’ve had to order the heat sink and fan from a vendor I have never used before however it turns out they were a very good choice and I’ve received all the parts without incident.

So the server components are as follows:

  • MB: KCMA-D8
  • KVM Option: ASMB4-iKVM
  • RAM: 4 x KVR1333D3E9S/4GHB (16g total)
  • CPU: 2 x AMD Opteron 4122 2.20 GHz
  • CPU Fan: 2 x Dynatron F555

I picked up a basic mid-tower case and power supply from the local computer shop and I’ll be using disks from the existing server when I finally move everything over to the new system.

Currently my disk configuration on the VM’s is spread out over two disks.  Each Windows VM has at least two virtual disks on separate physical disks to allow for backups to reside on a separate disk in case of hardware failure.

I expect to change this to instead use disk mirroring to support redundancy in case of hardware failure.  My current thought is to use the built in Windows RAID support, but the D8 motherboard does support hardware raid on the SAS ports.  There is conflicting statements in the manual if the SATA ports are supported but it would seem likely they are, however, using the built in Windows raid functionality will allow me to change motherboards without have to rebuild the entire raid disk set.

After assembling the components on the motherboard I hit my first snag with the hardware build.  The KCMA-D8 motherboard is a standard ATX form factor, however it uses pretty much every inch of the ATX spec and the generic mid-tower case need a little TLC with a pair of tin snips to remove some of the 5.25 drive bays.  The mid-tower case won’t work for a long term solution as with the motherboard installed, only two of the five 3.5 drive bays are available.  I’ve ordered a large tower case that claims it’s “silent”, so we’ll see how that works.

After getting everything installed in the case and hooking up a keyboard and mouse, the second hardware issue came up.  The F555 heat sinks have a high speed fan on them, which sounds kind of like a front load washer running at full steam inside of the case, times two for the dual processors Smile.

The MB has several fan settings, from full to whisper mode, however they seem to have no effect on the fan speed, which runs at a constant 5000 rpm.  By accident it turns out I had plugged one of the fans in to a case connector instead of the cpu connector, which ran the fan at ~3000 rpm.  This had the effect of making the system livable, however it was still too loud to be a long term solution.

After checking the temperature of the CPU while running and their thresholds, I pulled the stock fans off of the CPU heat sinks and installed a low speed 80mm case fan on each.  These run virtually silently and after stressing the server for a bit, it looks like I loose about 5-10 degrees of cooling, still well within the thermal limits for the AMD 4100 CPUs.

The only issue I’ve found with the motherboard so far is the extremely basic video card that is on board, it’s pretty much just a frame buffer, no 2d or 3d acceleration at all.  Screen redraws are slow and painful to watch, however dropping in a replacement video card means that the IP KVM no longer works.  Luckily the console isn’t used much so its a trade off I’ll have to live with as the KVM functionality is more important then the video performance on the server.

Next up in part 2… selecting a new hypervisor!

KeePass for Windows and 7Pass for Windows Phone 7

[sc:software-category ]Password management has always been a challenge and over the years the password vault has come a long way.  Currently KeePass is my vault of choice as it is open source, support for multiple platforms and is actively developed.

But I use KeePass to store more than just my computer account information, I store any important passcodes I have in it, including phone backing details, credit card pass codes and pretty much anything else that is sensitive.

However, like all vaults, sometimes you don’t have direct access to it.  This can be because you are off site at a clients, on vacation or at a friends house.  This is when a portable version of KeePass comes in handy and fortunately there is a number of ports for most major mobile OS’s.

When I had my Windows Mobile 6.5 device, I had a port of KeePass installed and used it infrequently but it provided a value service and a handy backup of my database at the same time.  Moving to Wnidows Phne 7  knew I’d be losing this but low and behold when I went to upgrade my local KeePass installation, two separate WP7 apps support KeePass files.

KeePassWP which seems to have stalled and 7Pass.

7Pass has one big limitation, its read only at the moment, but that is not a showstopper for me.

Installing 7Pass from the Marketplace was simple and the “Trial” version is actually fully functional with a simple nag notification on startup.  It otherwise uses the standard interface conventions of WP7 and provides a straight forward interface to the user.

Unlike Windows Mobile, Windows Phone 7 does not support any kind of user access to the file system, so getting a KeePass database on the phone is not quite as straight forward as it might otherwise be.

To get around this limitation, 7Pass support two options:

  1. Web Server Location
  2. DropBox

I don’t use DropBox (no other reason than I’ve never had a need for it) and while I believe KeePass has a secure file format, I see no reason to test that theory as I have a webserver I can use to host the database.

7Pass has little in the way of documentation (it’s a young project,  don’t blame it Winking smile) and nothing on how to configure the web server to support it.  7Pass does support user authentication on the web server and has fields for username, password and domain name.  So, I created a folder on my internet accessable webserver (II7), configured SSL as a requirement, and set authentication to basic, digest and Windows.  Dropped a copy of my KeePass database in the folder and pointed 7Pass to it.

If only it were so easy Smile.  7Pass came back with a file not found error.  Using IE on the phone worked without an issue (though it didn’t know what to do with the file).

Looking through the IIS logs it became apparent that the username/password were not being passed to the webserver from 7Pass, having been whacking at SharePoint and Windows Phone 7 a few weeks ago (which is still not working by the way) I remember a few posts I had found around what authentication modules were supported by Windows Phone 7 and most seemed to indicated Windows Auth was not one of them.

Take a stab in the dark, I left basic and digest on and turned off Windows.  7Pass promptly found the file but complained about it not being a valid KeePass file.

One step forward, one step backward Sad smile.

I found one reference on the 7Pass site about the issue, but no resolution.

Now I’ve had my KeePass database for quite a while, since early version 1, and it’s been upgraded to 2, I figured it might be something in the older version that did the conversion that might be the culprit, so in KeePass I exported the database to a new file and tried that.

Hazzah!  7Pass promptly loaded the file successfully and I was then prompted for my database password.

I don’t know if it’s a bug in 7Pass or a limitation/bug in WP7 but either way it works now.

The Good:

  • Trial version fully functional
  • Clean WP7 interface
  • Support for KeePass databases
  • DropBox support
  • Webserver support

The not so bad/not so good:

  • Cheap to get rid of the nag screen

The Bad:

  • Little documentation around webserver setup
  • No editing at the moment

OpenVPN and Failover Clustering

[sc:linux-category ]Being in the tech field means that quite often I’m working away from my house and security becomes a concern.  For quite a while I simply used Microsoft’s PPTP solution built in to Windows, however PPTP is not exceptional secure and more and more networks are blocking the protocols it uses.

For the last year or so I’ve been running an OpenVPN server to get around the limitations of PPTP and support a more “standard” HTTPS protocol that nobody blocks 🙂

OpenVPN is an open source implementation of a VPN using HTTPS and supports multiple client types, including Linux, Windows and Mac.   There are two version of the software, the true open source implementation which has server support for Windows and *nix.  There is also a commercial version called Access Server which supports multiple Linux versions as well as per-packaged VM appliances.

OpenVPN AS is actually free to use for 1 or 2 simultaneous connections so instead of building yet another VM to install it on I decided to use the VM appliance which made installation a breeze.  I’m running VMWare server  for my hypervisior and OpenVPN came in a zip file that contained the virtual disk and machine definition file, all that was required was to go in to the VMWare admin web site and add the VM to the host.

Resourcing for the VM was quite reasonable, with only 256 meg or RAM required and a few hundred meg of disk space.

Performance is likewise quite good as well, running on a system with multiple other VM’s going I’ve still been able to video stream across the net with it without stutter or pauses.

I have two VM servers that I use to host several different VM’s and over the years I have been creating more and more redundancy between them to ensure that if one of them is offline for some reason (power supplies die, hard drives fail, etc.) I still have my core services up on at least one of them.

When I first installed OpenVPN, no clustering was supported at all, so I installed it one a single VM host and accepted the risk that if that server was down, I wouldn’t have access to it.  I still am running the PPTP server and also have a SSH server available on the second VM server so all was not lost if OpenVPN was not available.

Recently OpenVPN added failover clustering to Access Server and they are planning on supporting load balancing at some point in time as well.  As I was doing some additional work on my VM servers over the last couple of weeks I decided to setup a second OpenVPN server and see how failover mode worked.

The first thing I did was to upgrade my existing OpenVPN installation to the latest release and ensured that it worked, which was a simple package download and install.

And then I made my first mistake.  I broke the cardinal rule of upgrades and didn’t backup my VM before proceeding.  This, as I’ll tell you about shortly, cost quite a bit of trouble Winking smile

After installing the VMDK and configuring the VM host server I booted up the second OpenVPN AS image and configured the basics, to support failover clustering OpenVPN requires three IP addresses; one for each server and a third “virtual” IP address to connect the actual clients to.

On the second VM, I only configured the most basic of of setting to get things up and running.  From  the documentation it indicated that settings like the SSL Cert etc., would be taken from the primary node.

I stayed connected to this “second” VM and proceeded in to the failover cluster setup, which asks some for some basic information, the IP address of the primary and secondary servers and the virtual IP address to use as well as the root password for the servers.  All of this was straight forward and I entered it as requested.  Once setup, there is a test button to ensure everything is correct before you have to save the settings and I got the green light to save away.

And this is where my mistake came back to haunt me.  Still working from the new VM, I applied the configuration settings, expecting the replication to push the configuration from the primary node to the secondary node.  This is NOT what happened Sad smile.

Instead, the configuration from the new VM was pushed over to my original server, wiping out the SSL cert and all other settings I had on it.

The hardest part of reconfiguring the settings was the SSL cert, as I had not backed up the private key (again, my bad…) that is required to configure SSL cert to the server.

However, after a bit of work, and reconfiguration, I managed to get everything back to how it was before my ill fated attempt to configure clustering.  At which point I promptly backed up the VM Smile

Attempt number two went much smoother, working this time from my original OpenVPN AS node, setting up the failover clustering went smoothly.

The final steps were to reconfigure my router to point to the virtual IP address and update my internal DNS entry to point to the new virtual IP address as well.  I also created two new DNS entries, one for each of the cluster nodes, so that if I need to connect to an individual node I can do so easily.

A quick internal test from my notebook to the VPN proved to be functional and some quick experimentation with failing one cluster node or the other proved everything was up and going.  The next day, while offsite, provided conclusive evidence that everything was fully functional.

The Good:

  • OpenVPN is a well supported SSL VPN
  • Free
  • Reliable
  • Clustering was easy to setup
  • Low overhead

The not so bad/not so good:

  • It’s not clear that you need to setup clustering on the node you want to use the configuration from in the GUI
  • Clustering is limited to failover only, no load balancing

The Ugly:

  • Dumb system administrators that don’t backup before major configuration changes Winking smile
  • OpenVPN doesn’t backup it’s own configuration on the nodes when clustering is enabled, just in case…

One last thought I’ll include here, though it doesn’t relate to the failover clustering.  The first version of OpenVPN AS I installed was 1.3.4, which had the Windows VPN client included in it.  At some point (I believe it was 1.4, but I could be wrong) the OpenVPN Windows client was rewritten and looks much nicer, every version beyond 1.3.4 has been extremely unreliable for me, dropping connections all the time, failing to connect and in general, unusable.

As I noted above, I upgraded to 1.7 before configuring clustering and so I decided to once more give the new client a try.  The client has now been split in to two different clients, both of which seem to suffer the same kinds of problems, however not to nearly the degree of the 1.4~1.6 clients.

For me, they still are not as reliable as the 1.3 client and offer me no additional features to me and so I am sticking with the 1.3 client.