Business Trip

So, I’m just heading home from what’s really my first official business trip. (There’s not a lot of business travel associated with my job normally.) Pretty cool!

I flew out to Chicago for a Virtualization Summit being put on by PC Mall, one of our newer vendors and the one that helped us put together most of the equipment for our recent server virtualization project. (Quick plug for PC Mall – my rep Jason is super helpful. If you need someone to help you with your IT purchasing, let me know and I’ll get you in touch with him.) The summit was pretty good, and gave me a chance to discuss our project with some other IT managers who were looking to do the exact same thing. It was also a good chance to see where some of the big players in the virtualization space are going with their products.

As an awesome side benefit, I got to hang out with my friend Charley, who I grew up with and now lives in Chicago. It’s always fun to get to hang out with old friends, especially when they’re doing really well. Charley got to attend the summit as well, and we got to talk shop a bit (he’s also an IT Director for a non-profit).

One thing the summit definitely solidified for me is that working for a non-profit is an entirely different ball game than normal enterprise level IT. We still have a lot of the same needs as other companies, but generally the IT funding isn’t there for those needs. I’m actually on board with that, as there are way more important things we can do organizationally with the money we’re blessed with than sink it into IT, but it does mean that we have to get pretty creative sometimes.

Thanks for the great trip and the awesome information PC Mall. Now it’s off to Ontario to get home to the wife and boy!

Server Virtualization

We just finished a project at Forest Home to virtualize (almost) our entire server infrastructure at Forest Home. It went super well! After a lot of deliberation, I decided to go with Hyper-V as our virtualization platform. The newest version of Hyper-V built into Server 2008 R2 seems to have all of the same features built into it that VMWare’s ESX server does, at least from the perspective of things we need. VMWare would probably disagree, but I’ve found Hyper-V to be a great platform. The final nail in the coffin for us was pricing. VMWare doesn’t offer any sort of discount to non-profits, whereas Microsoft gives huge discounts to non-profits. When I talked to a VMWare rep, it would have cost us somewhere in the neighborhood of $3,000 per year per host to license ESX and get the same features that Hyper-V has in this latest version. (I believe that VMWare does have a free version of ESX, but it leaves out some of the more critical migration capabilities that Hyper-V is now able to match). With the non-profit pricing that we get from Microsoft, we spent around $500 per host, and (if we want to maintain the ability to upgrade to the newest versions) we’ll spend around $200 per year (starting the year after we bought the licenses) on Software Assurance. For that amount of money, we got licenses for Windows Server 2008 R2 Datacenter Edition, which is licensed for an unlimited amount of virtual machines. If we had gone with VMWare we would have spent somewhere in the neighborhood of $45,000 on licensing over a five year period. Hyper-V over the same period will cost us about $3,900. For a non-profit, it’s really a no brainer decision. One of these days Microsoft will eat into VMWare market share enough, especially in the non-profit and SMB sectors, that they’ll hopefully have to deal with their pricing, but for now I suspect they hold such a dominant position in the market that they don’t really have to deal with the pricing gap. Here’s to hoping that competition helps out the consumer! Anyway, on to the technology!

For our virtualization initiative, I had a few goals. First and foremost, I wanted to reduce our operating costs. As it stands we’ve got about a four to one consolidation ratio, meaning we’ve got four virtual machines running on one Hyper-V host. The biggest obvious cost savings is in terms of our power draw. We’ve calculated that our IT infrastructure after Hyper-V is using around 40% less electricity than it was before the consolidation. We are also undoubtedly using our air conditioning less (three servers instead of 17 means an awful lot less heat being produced), but unfortunately I don’t really have a good way to estimate our savings there.

Our second goal was to improve reliability with our server platforms. We’ve already got a pretty good track record (last year we were able to provide about 99% uptime), so there weren’t huge gains to be made, but Hyper-V does allow us a greater degree of uptime due to the fact that we went with a three node HA (High Availability) cluster and the Live Migration feature (comparable to VMWare’s VMotion capability). Live Migration allows you to move a running virtual machine from one Hyper-V host to another with no interruption to the user. This means that we are able to put a Hyper-V host into maintenance mode (which will automatically live migrate all virtual machine’s off of that host and onto another one), perform any maintenance tasks that need to be performed, and then take it out of maintenance mode (bringing the virtual machines back). The HA cluster aspect means that if one of our Hyper-V nodes were to crash and burn, the other two would see that and automatically start up whatever virtual machines that were on that host on the other two. This means that even if we have a critical server failure, none of our servers are unavailable for more than a couple of minutes.

Thirdly, I wanted to make it easier to upgrade in the future. Prior to the virtualization initiative, if we wanted to upgrade a server we would have to order a new piece of hardware, install the OS, configure any applications that needed to run on it, and then migrate over any necessary data from the old machine. Worse, if we had a critical server failure we would have to order new hardware (which could take a few days at best to arrive), setup the OS and applications, and then restore the most recent backup of the critical data that we had on that server. At the time we decided to virtualize our servers, at least a third of our servers were approaching an age where we wouldn’t be able to renew the warranty and needed to start thinking about buying new hardware to replace them. With a virtualized platform, not only do we have less physical servers to upgrade, the upgrade becomes much easier. When we need to upgrade our hardware, all we need to setup are the OS, Hyper-V and Clustering support. Then we add the new machine to our cluster, migrate any virtual machines we have on the old hardware to the new hardware, and evict the old hardware from the cluster. Upgrade complete!

Let’s talk a little bit about our setup. We’re running a three node cluster made of Dell PowerEdge 2970s. They’ve each got dual quad core Opteron’s running at 2.4GHz, with 16GB of RAM. The crucial part, and part of why we’re currently a little bit stuck at a four to one ratio, is memory. I’m hoping to upgrade the memory pretty soon, which will allow us to migrate the last two servers that we’ve got over to our cluster. All three of the nodes have dual onboard Broadcom gigabit NICs and an Intel Quad Port server controller, giving us four more gigabit NICs on each machine. The backend storage SAN (needed to do a cluster) is a Dell MD3000i iSCSI SAN. Each of the Broadcom NICs is plugged into an HP ProCurve switch, along with the four iSCSI ports on the MD3000i, giving us redundant, multipath storage. One of the Intel Ethernet ports is used for the Hyper-V host network access, one for Live Migration traffic, and two for virtual machine network traffic.

The last piece of our setup is System Center Virtual Machine Manager, which has made managing our VMs and the physical-to-virtual migration process a breeze.

So far we’ve been really happy with our setup. Hyper-V has all the features we need, and we’ve made a tangible difference in the sustainability of Forest Home’s IT infrastructure. If you have any questions, leave them in the comments. I’d love to talk to you about them!

I’m back!

Well, it’s certainly been a long time since I posted. In the interim, I’ve been pretty busy. Expect a few posts over the next couple of days talking about some of those things. I’m going to warn you ahead of time, at least one of them will be a little long!