Kenya Update

So, I got some more details about how people can help financially with my Kenya trip. You can either write a check, or donate online. WePartner is a fully recognized 501(c)3 organization, so you’ll get a receipt at the end of the year for tax purposes.

If you want to write a check, go ahead and make it out to WePartner, and then you can either hand it to me (if you live around here) or mail it. I’ll gather them all up and then pass them on in one package to WePartner. If you need my address, send me an e-mail, Tweet me (@jerquiaga) or send a Facebook message. :)

If you’d like to donate online, you can go to WePartner’s site and go to their Donate page. Just make sure you put Josh Erquiaga Kenya in the note section at the bottom.

Again, thanks so much to anyone who is thinking about donating to help out. It’s still one of the hardest things to ask for help on, but you can check out my previous post if your curious why I’m asking. Thanks, and I’m looking forward to getting out there and helping out!

Kenya!

Well, it looks like I'm going to Kenya next month. Through a strange series of events and people who happen to know someone who in turn knows someone, I got connected with a man by the name of Fady who runs an organization called WePartner. WePartner (www.wepartner.org) is an organization that tries to help connect people with needs in their communities. As Fady is an IT guy by trade, a lot of the projects he's been involved in have been IT projects. They've also done a lot of work in the Middle East (Fady grew up in Egypt). Recently, someone from the Rift Valley Academy in Kenya contacted him about getting some help with their systems. At the same time, I had mentioned my trip to Malawi to a friend, and he had been talking to Fady about Kenya, and he connected the two of us.

I got a chance to talk with Fady a bit, and surprisingly, it seems that I have just the skills they were still looking for for their team. After that first chat and a conversation with my wife, I decided to join the team!

Both my wife and I have felt like Africa is a place we'd like to be involved with for a long time. Our trip to Malawi cemented that for us. We've also always been more of the service missions type of people than evangelistic missions type of people. It's just the way we're wired. The chance to go and use the skills that I've been developing over the last decade on a missions trip seems to be a dream come true.

The hardest part of planning for any missions trip for me has always been fundraising. That being said, a wise person told me along time ago that fundraising is an opportunity for people who either don't have the ability or desire to go on these trips to be involved. So that's where you come in friends. The trip is going to cost about $2,000. I will be paying a significant part of that, but if you're interested in helping financially, I would love to have you partner with me in this. I'm still talking with WePartner about the details of how people can donate, so I'll follow up with those details shortly. Thanks, and we're off to Kenya May 14th!

A Quick Note About Hyper-V

Just a quick note to anyone out there running Hyper-V in a cluster situation. We seemed to have a strange issue where occasionally when we Live Migrated a Virtual Machine from one host to another, the host that the VM was being migrated from would crash. After banging my head against the wall repeatedly, and then checking and re-checking my cluster configuration, I ran across this TechNet Forum thread: http://bit.ly/vmcrash. Basically, it seems that there is now a known issue with Windows Server 2008 R2 and some newer AMD processors. We’ve got dual quad core Opterons in each of our cluster nodes, and this seems to be exactly the problem we were experiencing. The hotfix is not publicly available yet, but if you call Microsoft product support and mention KB 981618, they should be able to get you the current hotfix. I installed it on all three of our nodes, and it seems to have resolved the issue for us.

Business Trip

So, I’m just heading home from what’s really my first official business trip. (There’s not a lot of business travel associated with my job normally.) Pretty cool!

I flew out to Chicago for a Virtualization Summit being put on by PC Mall, one of our newer vendors and the one that helped us put together most of the equipment for our recent server virtualization project. (Quick plug for PC Mall – my rep Jason is super helpful. If you need someone to help you with your IT purchasing, let me know and I’ll get you in touch with him.) The summit was pretty good, and gave me a chance to discuss our project with some other IT managers who were looking to do the exact same thing. It was also a good chance to see where some of the big players in the virtualization space are going with their products.

As an awesome side benefit, I got to hang out with my friend Charley, who I grew up with and now lives in Chicago. It’s always fun to get to hang out with old friends, especially when they’re doing really well. Charley got to attend the summit as well, and we got to talk shop a bit (he’s also an IT Director for a non-profit).

One thing the summit definitely solidified for me is that working for a non-profit is an entirely different ball game than normal enterprise level IT. We still have a lot of the same needs as other companies, but generally the IT funding isn’t there for those needs. I’m actually on board with that, as there are way more important things we can do organizationally with the money we’re blessed with than sink it into IT, but it does mean that we have to get pretty creative sometimes.

Thanks for the great trip and the awesome information PC Mall. Now it’s off to Ontario to get home to the wife and boy!

Server Virtualization

We just finished a project at Forest Home to virtualize (almost) our entire server infrastructure at Forest Home. It went super well! After a lot of deliberation, I decided to go with Hyper-V as our virtualization platform. The newest version of Hyper-V built into Server 2008 R2 seems to have all of the same features built into it that VMWare’s ESX server does, at least from the perspective of things we need. VMWare would probably disagree, but I’ve found Hyper-V to be a great platform. The final nail in the coffin for us was pricing. VMWare doesn’t offer any sort of discount to non-profits, whereas Microsoft gives huge discounts to non-profits. When I talked to a VMWare rep, it would have cost us somewhere in the neighborhood of $3,000 per year per host to license ESX and get the same features that Hyper-V has in this latest version. (I believe that VMWare does have a free version of ESX, but it leaves out some of the more critical migration capabilities that Hyper-V is now able to match). With the non-profit pricing that we get from Microsoft, we spent around $500 per host, and (if we want to maintain the ability to upgrade to the newest versions) we’ll spend around $200 per year (starting the year after we bought the licenses) on Software Assurance. For that amount of money, we got licenses for Windows Server 2008 R2 Datacenter Edition, which is licensed for an unlimited amount of virtual machines. If we had gone with VMWare we would have spent somewhere in the neighborhood of $45,000 on licensing over a five year period. Hyper-V over the same period will cost us about $3,900. For a non-profit, it’s really a no brainer decision. One of these days Microsoft will eat into VMWare market share enough, especially in the non-profit and SMB sectors, that they’ll hopefully have to deal with their pricing, but for now I suspect they hold such a dominant position in the market that they don’t really have to deal with the pricing gap. Here’s to hoping that competition helps out the consumer! Anyway, on to the technology!

For our virtualization initiative, I had a few goals. First and foremost, I wanted to reduce our operating costs. As it stands we’ve got about a four to one consolidation ratio, meaning we’ve got four virtual machines running on one Hyper-V host. The biggest obvious cost savings is in terms of our power draw. We’ve calculated that our IT infrastructure after Hyper-V is using around 40% less electricity than it was before the consolidation. We are also undoubtedly using our air conditioning less (three servers instead of 17 means an awful lot less heat being produced), but unfortunately I don’t really have a good way to estimate our savings there.

Our second goal was to improve reliability with our server platforms. We’ve already got a pretty good track record (last year we were able to provide about 99% uptime), so there weren’t huge gains to be made, but Hyper-V does allow us a greater degree of uptime due to the fact that we went with a three node HA (High Availability) cluster and the Live Migration feature (comparable to VMWare’s VMotion capability). Live Migration allows you to move a running virtual machine from one Hyper-V host to another with no interruption to the user. This means that we are able to put a Hyper-V host into maintenance mode (which will automatically live migrate all virtual machine’s off of that host and onto another one), perform any maintenance tasks that need to be performed, and then take it out of maintenance mode (bringing the virtual machines back). The HA cluster aspect means that if one of our Hyper-V nodes were to crash and burn, the other two would see that and automatically start up whatever virtual machines that were on that host on the other two. This means that even if we have a critical server failure, none of our servers are unavailable for more than a couple of minutes.

Thirdly, I wanted to make it easier to upgrade in the future. Prior to the virtualization initiative, if we wanted to upgrade a server we would have to order a new piece of hardware, install the OS, configure any applications that needed to run on it, and then migrate over any necessary data from the old machine. Worse, if we had a critical server failure we would have to order new hardware (which could take a few days at best to arrive), setup the OS and applications, and then restore the most recent backup of the critical data that we had on that server. At the time we decided to virtualize our servers, at least a third of our servers were approaching an age where we wouldn’t be able to renew the warranty and needed to start thinking about buying new hardware to replace them. With a virtualized platform, not only do we have less physical servers to upgrade, the upgrade becomes much easier. When we need to upgrade our hardware, all we need to setup are the OS, Hyper-V and Clustering support. Then we add the new machine to our cluster, migrate any virtual machines we have on the old hardware to the new hardware, and evict the old hardware from the cluster. Upgrade complete!

Let’s talk a little bit about our setup. We’re running a three node cluster made of Dell PowerEdge 2970s. They’ve each got dual quad core Opteron’s running at 2.4GHz, with 16GB of RAM. The crucial part, and part of why we’re currently a little bit stuck at a four to one ratio, is memory. I’m hoping to upgrade the memory pretty soon, which will allow us to migrate the last two servers that we’ve got over to our cluster. All three of the nodes have dual onboard Broadcom gigabit NICs and an Intel Quad Port server controller, giving us four more gigabit NICs on each machine. The backend storage SAN (needed to do a cluster) is a Dell MD3000i iSCSI SAN. Each of the Broadcom NICs is plugged into an HP ProCurve switch, along with the four iSCSI ports on the MD3000i, giving us redundant, multipath storage. One of the Intel Ethernet ports is used for the Hyper-V host network access, one for Live Migration traffic, and two for virtual machine network traffic.

The last piece of our setup is System Center Virtual Machine Manager, which has made managing our VMs and the physical-to-virtual migration process a breeze.

So far we’ve been really happy with our setup. Hyper-V has all the features we need, and we’ve made a tangible difference in the sustainability of Forest Home’s IT infrastructure. If you have any questions, leave them in the comments. I’d love to talk to you about them!

I’m back!

Well, it’s certainly been a long time since I posted. In the interim, I’ve been pretty busy. Expect a few posts over the next couple of days talking about some of those things. I’m going to warn you ahead of time, at least one of them will be a little long!

Creating Custom Drive Rails

Working for a non-profit, I’ve gotten accustomed to using donated equipment when we get it. Recently, the credit union that Forest Home does all it’s banking through donated about 15 computers to us. They didn’t have optical drives in them (which makes sense, security wise), so I picked up 15 DVD drives for about $7 each (NOTE: it’s not easy finding old school beige drives anymore – go figure). I went to pop one of the drives in and found myself at a loss. The cases are the kind that use rails on either side of the drive to mount them.  Since the systems didn’t have drives in them, they also didn’t have the rails!

After a little bit of brainstorming with some pretty smart guys in our maintenance department, we can up with a cheap, easy solution. I bought some 1” x 1/8” aluminum bar stock and some heavy duty mounting tape at the local Home Depot. We then cut down the aluminum bar stock into 5.5” pieces (the depth of the tracks the rails were designed to slide into). I marked the drives with a felt pen to make sure I knew where the new rail pieces needed to end up, and then attached them with the mounting tape. In our case, it took a double thickness of mounting tape to get the drive in about the right spot, but once they’re in you can’t tell the difference. I’ve got more pictures of the build at my Flickr site, here. Check them out if you need more detail.  Check out the finished product!