Desktop Linux Update Hell

[sc:linux-category ]It’s no secret that I’m primarily a Windows man for my desktops, but I do run Linux for a mail gateway and do have one virtual workstation running Linux.  In particular (though I doubt it has a significant impact on this post) I run OpenSUSE.

In general, once I’ve setup a Linux box I tend to forget about it until I need something from it, but since I setup the workstation I’ve been using it almost on a daily basis.  I had been using a very old version of OpenSUSE for the workstation and  I recently upgraded it to OpenSUSE 12.1.  The previous version was so old that they were no longer releasing security patches or other updates.

However 12.1 is the current release and so patches come down all the time.  And when I say all the time, I mean ALL THE FREAKIN TIME!

So much so that every day when I logon to the system, there’s more updates to install.  OpenSUSE by default doesn’t enable the auto updater (doesn’t even install the management console for it by default) so a user is prompted all the time (unless of course they turn off the update notified Winking smile).  And the auto updater doesn’t seem to work either, I still get prompted for patches almost every day!

This is highly annoying to say the least.  I never realized how good of job Microsoft does of minimizing update hell with their monthly schedule.

And don’t even get me started on GNOME and KDE…

OpenVPN and Failover Clustering – Part 2

[sc:linux-category ]My previous post on OpenVPN and Failover Clustering had a happy ending, unfortunately that’s not the end of the story.

It turns out that the cluster was randomly failing between the two nodes every few minutes and the log files on the servers were filling up with errors.

I’ve been forced to disable the clustering at this point while I investigate what’s going on.  The log files indicate that UCARP (the software that manages the virtual IP address between the two systems) is detecting a crash of the OpenVPN process and failing over to the other system, however the OpenVPN server doesn’t actually seem to be crashing.

A new release (1.7.1) of the OpenVPN Access Server did not help either.

I’ll have to spend some time on this, but there are other more important items on my agenda for the time being so this will take a back seat for a while.

OpenVPN and Failover Clustering

[sc:linux-category ]Being in the tech field means that quite often I’m working away from my house and security becomes a concern.  For quite a while I simply used Microsoft’s PPTP solution built in to Windows, however PPTP is not exceptional secure and more and more networks are blocking the protocols it uses.

For the last year or so I’ve been running an OpenVPN server to get around the limitations of PPTP and support a more “standard” HTTPS protocol that nobody blocks 🙂

OpenVPN is an open source implementation of a VPN using HTTPS and supports multiple client types, including Linux, Windows and Mac.   There are two version of the software, the true open source implementation which has server support for Windows and *nix.  There is also a commercial version called Access Server which supports multiple Linux versions as well as per-packaged VM appliances.

OpenVPN AS is actually free to use for 1 or 2 simultaneous connections so instead of building yet another VM to install it on I decided to use the VM appliance which made installation a breeze.  I’m running VMWare server  for my hypervisior and OpenVPN came in a zip file that contained the virtual disk and machine definition file, all that was required was to go in to the VMWare admin web site and add the VM to the host.

Resourcing for the VM was quite reasonable, with only 256 meg or RAM required and a few hundred meg of disk space.

Performance is likewise quite good as well, running on a system with multiple other VM’s going I’ve still been able to video stream across the net with it without stutter or pauses.

I have two VM servers that I use to host several different VM’s and over the years I have been creating more and more redundancy between them to ensure that if one of them is offline for some reason (power supplies die, hard drives fail, etc.) I still have my core services up on at least one of them.

When I first installed OpenVPN, no clustering was supported at all, so I installed it one a single VM host and accepted the risk that if that server was down, I wouldn’t have access to it.  I still am running the PPTP server and also have a SSH server available on the second VM server so all was not lost if OpenVPN was not available.

Recently OpenVPN added failover clustering to Access Server and they are planning on supporting load balancing at some point in time as well.  As I was doing some additional work on my VM servers over the last couple of weeks I decided to setup a second OpenVPN server and see how failover mode worked.

The first thing I did was to upgrade my existing OpenVPN installation to the latest release and ensured that it worked, which was a simple package download and install.

And then I made my first mistake.  I broke the cardinal rule of upgrades and didn’t backup my VM before proceeding.  This, as I’ll tell you about shortly, cost quite a bit of trouble Winking smile

After installing the VMDK and configuring the VM host server I booted up the second OpenVPN AS image and configured the basics, to support failover clustering OpenVPN requires three IP addresses; one for each server and a third “virtual” IP address to connect the actual clients to.

On the second VM, I only configured the most basic of of setting to get things up and running.  From  the documentation it indicated that settings like the SSL Cert etc., would be taken from the primary node.

I stayed connected to this “second” VM and proceeded in to the failover cluster setup, which asks some for some basic information, the IP address of the primary and secondary servers and the virtual IP address to use as well as the root password for the servers.  All of this was straight forward and I entered it as requested.  Once setup, there is a test button to ensure everything is correct before you have to save the settings and I got the green light to save away.

And this is where my mistake came back to haunt me.  Still working from the new VM, I applied the configuration settings, expecting the replication to push the configuration from the primary node to the secondary node.  This is NOT what happened Sad smile.

Instead, the configuration from the new VM was pushed over to my original server, wiping out the SSL cert and all other settings I had on it.

The hardest part of reconfiguring the settings was the SSL cert, as I had not backed up the private key (again, my bad…) that is required to configure SSL cert to the server.

However, after a bit of work, and reconfiguration, I managed to get everything back to how it was before my ill fated attempt to configure clustering.  At which point I promptly backed up the VM Smile

Attempt number two went much smoother, working this time from my original OpenVPN AS node, setting up the failover clustering went smoothly.

The final steps were to reconfigure my router to point to the virtual IP address and update my internal DNS entry to point to the new virtual IP address as well.  I also created two new DNS entries, one for each of the cluster nodes, so that if I need to connect to an individual node I can do so easily.

A quick internal test from my notebook to the VPN proved to be functional and some quick experimentation with failing one cluster node or the other proved everything was up and going.  The next day, while offsite, provided conclusive evidence that everything was fully functional.

The Good:

  • OpenVPN is a well supported SSL VPN
  • Free
  • Reliable
  • Clustering was easy to setup
  • Low overhead

The not so bad/not so good:

  • It’s not clear that you need to setup clustering on the node you want to use the configuration from in the GUI
  • Clustering is limited to failover only, no load balancing

The Ugly:

  • Dumb system administrators that don’t backup before major configuration changes Winking smile
  • OpenVPN doesn’t backup it’s own configuration on the nodes when clustering is enabled, just in case…

One last thought I’ll include here, though it doesn’t relate to the failover clustering.  The first version of OpenVPN AS I installed was 1.3.4, which had the Windows VPN client included in it.  At some point (I believe it was 1.4, but I could be wrong) the OpenVPN Windows client was rewritten and looks much nicer, every version beyond 1.3.4 has been extremely unreliable for me, dropping connections all the time, failing to connect and in general, unusable.

As I noted above, I upgraded to 1.7 before configuring clustering and so I decided to once more give the new client a try.  The client has now been split in to two different clients, both of which seem to suffer the same kinds of problems, however not to nearly the degree of the 1.4~1.6 clients.

For me, they still are not as reliable as the 1.3 client and offer me no additional features to me and so I am sticking with the 1.3 client.