Fake it till ya make it!

So no one who is reading this should be in the dark about some of the interesting things I have been doing as of late.  But if you are here is a quick re-cap.

Language Access Network my employer is undergoing an installation of a first of its kind Video Call Center.  I will more on that to write soon.  As part of this process we had a WHOLE LOT of infrastructure put into place.  For starters we needed a “SAN”, we needed Servers, we needed DC Switching and we needed lots and lots of licensing and that all before the developers and engineers jump in to make the whole thing work.  The cool bits of this first part are what we did for “SAN”, Server and Network.  As you all know I am a past Cisco UCS zealot and I have a NetApp in my basement so you would think that it would be simple math as to what I would have installed.  You would be right.  UCS and NetApp were about $100,000 more than I could scrape out of my budget and still have anything left for other major components.  Before people get bent out of shape about me saying Cisco UCS and NetApp are to expensive, I did not say that.  Honestly I think within existing DC platforms they are both very well priced if you don’t bring next gen platforms into the mix.  In my case the next gen platform is Nutanix.  If you don’t know anything about these guys click the link and check them out.

In a nutshell Nutanix is 4 Blades of Compute and 20TB of Storage in a 2RU chassis with FusionIO, SSD and SAS Drives and no common backplane between the 4 nodes.  Along with my four pod node we added Arista 7124SX as our DC Switching/Fabric.  There are lots of details around this combination like currently Nutanix does not support using the Node for a bare metal server like you can do with UCS or other Blade Enclosures and the storage has limited access to the outside world (it is setup to presented to ESXi Hosts as iSCSI targets and VMs as ViSCSI targets) but so far I love the platform.  It gave me what I needed in the price point I needed and offers huge scale out options considering it is based of the GFS files system that Google uses across their DC’s.

But I’m getting way off track and taking up content for future posts.  This one is about me being dumb.  I am a Network Wonk by trade and even though I originally cut my teeth in large scale PC and Novell Server environments those days are long past.  I have tried to keep up to date and have ran VM Server since it was first released for free as a Linux package in my home lab.  I have also designed networks for both small and large ESX/i deployments.  But other than some limited lab work my production experience has been limited to my single ESXi server I have ran as an app server for the past year hear at Language Access Network to consolidate services off of some crazy old and dying servers.  So needles to say spinning up my first production ESXi Cluster of (4) Servers, Sphere and provisioning the storage was a bit nerve-racking.  Its a HUGE statement for all the vendors that it was actually pretty painless.  Maybe even a bit to much so.

The Arista Switches for my purpose configured just like an IOS device.  The Nutanix gear came pre-loaded with ESXi on each node, storage already sync’d in the distributed files system, the virtual controllers configured  and the Vsphere appliance already onboard ready to be turned on.  All in all it was a Production VMWare environment in 4 Boxes and my Licensing.  Hard to beat that.  Then I lost one of my VMs within the first 8 hours of being live.

So first of I am taking 1000% of the blame for this.  I don’t want any of my friends at Nutanix to think I am going to toss them under the bus because if you are smart you don’t do that to partners.  The long and short of what happened is thanks to how easy things had always been going back even to my ESXi 4 install over a year ago I had lulled myself into a false sense of security and never really started doing backups.  Yeah yeah I know your all saying nasty things about me at the moment.  Guess what I am to.  I write this at 2:31AM and I have a day full of meetings ahead so I deserve the flogging your offering up.  Anyway I pushed my IPAM appliance over to our Nutanix cluster just to see our shiny new box working and by the time I came back from lunch it was MIA and never to be recovered.  We were still in stand-up phase and with some tweaking the remote install team re-zero’d the array and away went my VM.  So the moral to this story is HAVE FREAKING BACKUPS!

So after I finished beating my head on the table for not having them I called my software vendor told them to get key quotes ASAP for Veeam and I downloaded the trail version of Veeam Backup & Replication.  I followed that by putting my originally planned “backup” server out out of the rack and threw Openfiler on it.  Over a week later I’m tired of screwing with Openfiler so I just tossed FreeNAS on it and after some more nashing of teeth in which I had to give up a 2TB drive as an OS drive I have a function 5.2TB of backup for my VMs.  As we speak Veeam B&R is crunching away on my third of four production VMs and then I get to migrate them to their new home on Nutanix.

So I guess I am making it but man I’m rusty and as good as I can talk about it I can promise you most of it has been faking it.  But not for much longer.

One comment

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.