Led Zeppelin said it best I guess.  This past week Ohio along with lots of other states got hit with the remains of hurricane Dean.  So far it has been the most damaging storm for my clients in my short consulting career.  The first call came on Tuesday morning August 21st.  That call was from one of our account managers who indicated a client had sustained catastrophic damage to their 6509 when water rushed into their core network closet.  My first two thoughts were how quickly can we get replacement hardware and how long should it take for me to get them back up and going?

Incredibly the answer to question #1 was less than 24 hours.  I was stunned, when ordering the hardware after 1pm on Tuesday a pallet landed on the clients dock before 12pm on Wednesday.  On more than one occasion I have seen Cisco chassis line cards take more than two months to be available for shipment.  So for the record this is the hardware we are talking about.

• 6509E Chassis
• Supervisor 32 8GE (2)
• Eight Port Gig Ethernet GBIC CARDS (2)
• 48 Port 10/100/1000 RJ45 Line Card with POE daughter card slot
• SX SFP Fiber Modules (16)
• 4000 Watt Power Supply (2)

All I can say is that our account and purchasing managers must have called in some markers or sold their souls to pull this off.  Or there is the option that Cisco harware is not flying off the shelves like it used to.

The answer to question #2 ended up being about two hours.  While our clients network was not particularly complex it was new to me.  The last engineer who worked on their network from our shop was more than three years ago and the configs on record were a bit out of date.  To the clients credit they had taken the opportunity to save a copy of their RSM and Switch engine code after they had added their last VLAN in late 2005.  With a quick talk through of the network and the configs we were able to move forward.  While electricians ran two new 220 circuits to the data closet we found a pair of 15 amp circuits to connect to the 1400 Watt power supplies that I had brought with me from our surplus supplies.  For the record the 6509 chassis bears labels that state that it wont accept power supplies less than 2400 Watts.  However if you throw two 1400 Watt supplies in I know from experience that it will chain them in non-redundant mode and power up a single Sup32, two 8 port GBIC cards, a non POE 48 port RJ45 card as well as the FAN E card which I will have to say is a much better fan system than the 6500’s FAN and FAN-2 card.  With that done we moved the client to their new hardware and had everything set for install by 2:30pm.  At 3:3opm the electricians completed their handy work and were clear of the room by 4:00pm for us to install the hardware.  By 5:00pm the network was running off of their new core chassis.

So overall I will have to say that it was less exciting than I thought it may be.  I guess I expected frantic whiteboarding to define network design and resources that were lost along with the chassis and it’s Sup1A-MFSC2.  But a bit of luck with the saved configs, quick thinking on the clients part that had the network functioning on light hardware before I even arrived and my training and experience to apply the resources at hand to get the new core up and installed in under five hours and it was just another day at work.

Keep in mind that once in awhile we all get wet int the rain.  Make sure you have a plan for how you intend to dry off.