Backups in ProxMox

linux-category-inverted

In Hyper-V, taking backups was, welll, not very pleasant.  It was fine if you had a dedicated backup solution, but for a small set of VM’s like I have, it require basically taking snapshots of each individual VM and exporting them.

Microsoft didn’t make the process very easy, but with a bit of PowerShell scripting I had a pretty good system in place that kept the VM’s down only for as long as it took to export the disk images.

ProxMox on the other hand has a built in VM backup system that looks like it will automate most of the things I had scripted.

However, the first thing that becomes a problem is that by default, ProxMox stores the backup images on the boot drive, which is only 80g in size.  I installed a 1TB SSD to host my VM’s so that math doesn’t work very well 😉

I had upgraded my main data drives a while ago from some 2TB HD’s to 6TB HD’s and those old drives have just been laying around, so I decided to add them to the host servers as backup space.

My first attempt to do this was to create a new LVM storage group add them to the ProxMox server as a new storage group, but it turns out that doesn’t work as you can’t use an LVM disk for backup.  Instead I had to simply create a ext4 file system on the disks and mount them to the ProxMox host server.  I found this article useful, with the exception that it’s a little old and suggests ext3 still.

Once the drive was mounted, going in to the ProxMox web interface, selecting Datacenter->Storage->Add->Directory brought up the standard dialog to add storage to the system.  One item to note is that since I have two nodes in my cluster, you have to connect to the node that has the local drive on it.

There some standard fields to fill in, like the ID, which I labeled “backups”, note that creating a directory store in ProxMox will assume it is available on all nodes, not just he current one.  Make sure to select the right content type, for me I only selected “VZDump backup files”, but you might want to also support other things like ISO images or container templates.

The other item to note is the “Max Backups” field, this sets the maximum number of backup images that are allowed on the disk for each VM.  Once this number is exceeded the old backup files will be removed automatically (at least if I’m reading the documentation correctly).  For me, I set this to 3.

The final step is then to setup a backup schedule.  To do this, select the host Datacenter->Backups->Add.

Since this is backing up to local storage, the first option “Node” should be set to the physical node your backing VM’s up from.  Then you can simply select your newly added backup storage, what time you want the backups to start and which VM’s to backup.  The rest of the options are straight forward and now you have a weekly backup of all your VM’s.

I also keep a copy of my backups offsite, which is easy to do as you can just sftp a copy of the backups from your /mnt/backups/dump directory and move them to a USB drive or other storage to take with you.

One last point, as mentioned above, my main file server has a 6tb drive, which of course can’t be backed up to a 2tb disk.  I have a separate set of backup disk for this and so I don’t won’t ProxMox to backup that volume.  Fortunately, if you go to [Your VM]->Hardware->[Disk you want to exclude]->Edit, you can select the “No Backup” checkbox and it will not be included in the ProxMox backups.

Moving VM’s to ProxMox

linux-category-inverted

In my previous post I installed ProxMox as my HyperVisor, but I didn’t mention anything about moving VM’s from Hyper-V over to it.  Well, here’s how I went about the process.

First, I tracked down this article, which has a pretty good detailed guide of the process.  Basically it comes down to the following:

  • Export the VM from Hyper-V to get a clean snapshot.
  • Transfer the .vmdk files up to the ProxMox host.
  • Create a new VM in ProxMox with the same CPU/Memory/Disks.
  • Convert the .vmdk files to the KVM format.
  • Start the VM and make any other changes, like IP, DNS, etc. that are required if the OS doesn’t detect the changed hardware properly.

There are a few caveats:

  • While exporting from Hyper-V, the completion percentage is displayed in the “Status” column of the Hyper-V manager.  However that column is not always visible depending on the resolution of the display and how large the window is, so make sure to expand the Hyper-V manager until you can see it.
  • In the article, the .vmdk is converted to a .qcow2 file, however for me the default format of disks is actually raw, so replace “-O qcow2” with “-O raw”.
  • Similarly, the location of the disks in the article is incorrect, it’s actually in /dev/pve, but that’s actually a link back to /dev/dm-? but you can use either.

That makes the conversion command something like:

qemu-img convert -O raw ~/BootDisk.vhd /dev/pve/vm-102-disk-1

Overall the migration process is quite easy, if a little slow between the transfer of the .vmdk files as well as converting them over to the new format.

 

Replacing Hyper-V aka ProxMox

linux-category-inverted

As I have been moving away from Microsoft solutions, one of the outstanding items on my TODO list is replacing Hyper-V as my virtualization system.

Previously I had configured two Windows 2012 servers with Hyper-V and used them as my physical hosts, adding VM’s for each of the various functions I needed.  That included Exchange, Web, VPN, Domain Controllers, etc.

I’ve replaced Exchange with a combination of NextCloud (for contacts and calendar) and DoveCot (for mail), which really limits my requirement for the Domain Controllers or relying on Hyper-V.

So I started to look around for a hyper-visor to replace Hyper-V and after looking at a few, ProxMox, an open source “bare metal” hyper-visor.  I’ve put bare metal in quotes here as it’s really just running Debian underneath a pretty slick web UI.

Installing ProxMox is pretty straight forward, download the ISO, burn it to a CD (or there are USB stick instructions as well) and let it go to town.

Once it is done, you connect to it through the web interface (note that you get a certificate warning as by default it uses a self signed certificate) and you can go to town installing VMs.

As my first VM, I spun up an Ubuntu 16.04 image to act as the replacement file server.  The first challenge was getting the ISO of Ubuntu on to ProxMox, it’s well documented that the web interface has a hard limit for upload size that most ISO’s exceed.  It’s a well documented issue and a quick Internet search brought to light the solution, using SFTP instead.

In my old configuration I had two Windows 2012 file servers, one on each VM host, using DFSR to replicate the data between them and DFS to present them as a single share.  My main data volume is a 6TB drive, which contains just a little over 3TB at the moment (and growing of course).

I’d decided recently to dump the DFS-R replication in preparation for moving to Linux (it was really more trouble than it was worth, the number of times replication broke for any number of reasons was kind of a pain) and so after Ubuntu installed (no issues there), the next issue was attaching the 6TB physical disk to the VM.

The web interface is of little help here, but after some more digging around the web, I found a helpful article on how to achieve it.

 

Once both the VM and additional storage were in place, configuring Samba was pretty straight forward and copying the data across, while time-consuming, was easy enough as well.

My intention, when I’ve moved my other VM host over to ProxMox, is to similarly attach the second 6TB drive to my NextCloud install and sync the data between them, allowing me to expose my file share through the NextCloud interface as an external storage provider as a local directory instead of though an SMB share as I’m doing it now.

So what about ProxMox?

Overall it’s been good so far, there is one other annoyance I will mention.  Each time you logon to the admin interface you get a pop-up about you not having “No Valid Subscription”.

This, in effect, makes ProxMox NagWare instead of Open Source as there is no requirement to get a subscription unless you require support.

Fortunately all of the source code is available and so you can remove the pop-up pretty easily.  I used these instructions on how to do it.  Though the JavaScript file had moved since it was written, but was easy enough to find.

The web interface seems quite good, with console access to the VM’s as well as the ProxMox host, you can do pretty much everything from it.

There are also lots of advanced options, like replication and failover.  I haven’t tried these yet, but as I decommission the last of my Windows Servers, I’ll likely start poking at them some more.

The only other slight annoyance (and it’s not so much ProxMox’s fault) is that when your using the console, sometimes the input focus can switch from the console and before you know what’s happen your browsers has done something like go to the previous page because you hit the backspace key to correct a command.

However, since after initial setup, your unlikely to use the web console very much, it seems like a minor thing overall.

NextCloud

linux-category-inverted

As I mentioned a while ago when I reviewed ZorinOS, I’ve been thinking more and more about moving away from big commercial software.  It started with my move away from Windows Phone (or more accurately, Microsoft abandoning the platform and moving away from their users) and continued with the details of how much data Windows 10 collects.

I had known for a while that I was going to have to eventually do something with my Exchange server, 2010 was old and getting older all the time.  I did have a newer version of Exchange, but for the few mailboxes I host, it always seemed like overkill.

The big issue with moving from Exchange had never been mail per say, IMAP servers and clients are plentiful and fully function, but contacts and calendar had always been another story.

I knew CalDav and CardDav had come along to try and solve the issue so I decided to look around and see what state things were in.  It turns out, much better than I had expected.

There were several good CardDav/CalDav server implementations around but the one that really stood out to me was NextCloud.

Not only is NextCloud a full CardDav/CalDav server, it’s a full file sync (ala OneDrive) server as well.  That ticked off another check box for me as the convince of OneDrive is huge.

Of course installing such a complicated piece of software can be daunting, fortunately, there’s an easy solution, they have a VM image with everything setup for you (they even have a step by step walk through on how to set it up).

Once the VM was up and running, logging in to the web interface is easy enough (after you of course accept the self sign cert of course).

I’ll go in to how to resolve the certificate issue in a future post, but obviously Let’s Encrypt will be involved.

Features

NextCloud has a host of features, both the ones that come standard, and those that are found in the store.

The ones that I have installed are:

  • Files, Contacts and Calendar (obviously)
  • Mail
  • Notes
  • News
  • Phone Sync
  • Bookmarks
  • Tasks
  • External Storage

At first I connected Mail to my existing Exchange server, but in another article I’ll go in to the details of how I setup DoveCot on the NextCloud server and decommissioned my Exchange server completely.

News

This is another feature that really caught my attention, for many years I have been using a perl script to pull RSS feeds from various sources and add them to a folder on my Exchange server via IMAP.  That way, once an article had been read, it synced across all of my mail clients.

Outlook has always worked well with this system, but other clients have always struggled and I suspected to replace Exchange, I’d have to find another way to read my RSS feeds.

News solved the problem.

It is an RSS feed aggregator and has support for updating the read/unread status across clients that support it.

Phone Sync

This was an unexpected surprise, an SMS sync client for Android.  It’s a little buggy, but better than nothing.

Bookmarks

I’ve always hated being tied to one browser, I tend to move across devices and browsers pretty fluidly and so I seldom use the built-in bookmark features of a browser.  Instead I created a bookmark web app several years ago that manages all my bookmarks for me and I use it as my home page.

That app uses a MySQL database to store the bookmarks but NextCloud’s UI is better and so I’ve started to look at porting my app over to NextCloud as there are plugin’s for every major browser available for NextCloud’s bookmark feature.

Tasks

I installed this to sync up my phone’s tasks, I don’t use them often, but it’s kind of nice to have them available online and backed up.

External Storage

This was another big feature that I like quite a lot.  As I’ve mentioned before, I have a large library of software on my Windows file server, but that means I have to be connected to my local network to get access to them.

With NextCloud and the External Storage plugin, I can expose that file share as part of the NextCloud web interface and get to my library from anywhere.

NextCloud Clients

One of the things that really drew me to NextCloud was the availability of clients for just about every OS.  Windows, Linux, Android, iOS and Mac.

This broad support really makes the case for NextCloud as a replacement for OneDrive or any other cloud storage provider.

About that VM…

There is one issue with the supplied VM image that I didn’t touch on earlier, the default image only has a 32g partition.

You can download other images that have 500g or 1tb, but those aren’t very granular choices.

Fortunately all of the VM’s come with LVM partitions so adding additional space to the VM is easy… just kidding, it’s a real pain in the butt if your not familiar with LVM.

Unfortunately finding a good tutorial isn’t easy, but this one was the best I found and solved the problem by adding an additional 80g to my VM.

Final Thoughts

I’ve been very happy with NextCloud, it was an eye-opening find that really solved a lot of issues I knew I was going to have by moving away from Exchange and other services that I had always taken for granted in the Microsoft ecosystem.

Without it my move away from Exchange may not have even been possible and it certainly has made it possible to go beyond what I thought was possible.

At this point, I’m now convinced that moving 100% off of big commercial software and entirely on to open source.

That’s something that just a few months ago I would have said was impossible.

 

 

 

 

 

 

 

 

 

 

Update: Remapping keys in Linux

linux-category-inverted

In my previous pose, Remapping keys in Linux, I provided some instructions on how to make the right control key act like the menu key.  However, one issue was that you have to run the script after each login.

That’s more than just a little bit annoying, so I’ve tracked down how to solve that as well.

From this AskUbuntu post, it turns out that xmodmap is no longer used and instead xkb is what really needs to be updated.  The process is simple enough, edit:

/usr/share/X11/xkb/symbols/pc

find the following block of text:

 key <RTSH> { [ Shift_R ] };
 key <RCTL> { [ Control_R ] };
 key <RWIN> { [ Super_R ] };
 key <MENU> { [ Menu ] };

And change <RCTL> to Menu and <Menu> to Control_R.

Reboot and voila, the right control key now works like a menu button.

Of course, this is system wide so if you have multiple users logging in, they all get this.

One note is that the linked article does mention to delete the compiled keymaps, but there were none on my system so there was nothing to delete.