Category: On the Job

Why I do what I do.

Hi my name is Josh O’Brien.  I am the CTO of Language Access Network.  We provide remote video language services to hospitals and clinics across the United States.  Blah Blah Blah….I go through this at least once a day and sometimes as many as ten times a day.  Also every day I hear about all the cool new milestones that Language Access Network is reaching.  Most of these go something like today our interpreters provided 70 more interpretations than our last high water mark a week ago.  My answer has always been that’s nice please let me get back to work making sure everything is working, scalable and supportable.  I know I am a cold bastard, but it has been the reality of what I do.

Don’t get me wrong, I 100 percent believe in our company, what it does and the profound impact we make in the medical market…I really do!  But I am not an interpreter, doctor or customer advocate my job is to make the zeros and ones fly down the correct pipe at the correct time and to so as efficiently as possible.  I just don’t have time for all the slaps on the back an congratulations for doing what I was brought here to do which is to massively grow the company and win.  Even my own staff does not get me, they love reveling in the record breaking and I am thrilled that they are because at the end of that day it is one more amazing testament to how good they are and the hard work they put in.  But I am a geek, not a mercenary in it for the cash but just a flat out geek, nerd, packet pusher whatever you want to call me.  I get excited about the tech.  I have always been that way.  If I won $100,000,000 in cash today I would renounce my salary, hire more staff, reduce my travel load and then out of my pocket buy all sorts of cool gear for Language Access Network and my team.  I just freaking love what I do! Read more

Vendors Beware

Simply put I am not going to target a specific vendor in this post but if you work for an IT vendor (Telco, SP, MSP, Hardware/Software maker, VAR, LAR, distributor, consultant) this list is a warning of the things you can do to really piss me off. It is not just me it is all of us. We have way to much to do on a daily basis to have to babysit the people who we pay to help us. So read, pay attention and don’t do this stuff!

1. Lie – Tell me what you can or can’t do and stick to it. This includes what your products can or can’t do.
2. Have a messy house – If your provisioning team does not talk to your sales team and no one talks to support you’ll piss me off.
3. Wing it – If I wanted to you to wing it on my project I would not have paid your your stupidly high rate. My 7 year old can wing it.
4. When you are getting it wrong ask me to not be me – Piss Off I am the customer I am right because I pay you monies.
5. Cover up your lie/mistake/bug/whatever else your covering up – Be honest with me I may not fire you.
6. Presume you know me by watching my twitter feed – I am more than 128 characters.
7. Blame your contractors – You hired them you fix it. I have to do that when you mess up!
8. Get me in trouble – If your stupidity makes me eat a bullet for you the world will know!
9. Lead with List price – We all know its BS. Give me the best deal you can and we’ll each do better by wasting less time.
10. Take a social or political stand corporately that makes me have to hate you – Really an Internet company siding with SOPA….

I am sure I have like 1000 more and I encourage my readers to jump on board and add them to the comments. In the end we all have jobs to do please don’t make me regret engaging you by making mine harder.

Fake it till ya make it!

So no one who is reading this should be in the dark about some of the interesting things I have been doing as of late.  But if you are here is a quick re-cap.

Language Access Network my employer is undergoing an installation of a first of its kind Video Call Center.  I will more on that to write soon.  As part of this process we had a WHOLE LOT of infrastructure put into place.  For starters we needed a “SAN”, we needed Servers, we needed DC Switching and we needed lots and lots of licensing and that all before the developers and engineers jump in to make the whole thing work.  The cool bits of this first part are what we did for “SAN”, Server and Network.  As you all know I am a past Cisco UCS zealot and I have a NetApp in my basement so you would think that it would be simple math as to what I would have installed.  You would be right.  UCS and NetApp were about $100,000 more than I could scrape out of my budget and still have anything left for other major components.  Before people get bent out of shape about me saying Cisco UCS and NetApp are to expensive, I did not say that.  Honestly I think within existing DC platforms they are both very well priced if you don’t bring next gen platforms into the mix.  In my case the next gen platform is Nutanix.  If you don’t know anything about these guys click the link and check them out.

In a nutshell Nutanix is 4 Blades of Compute and 20TB of Storage in a 2RU chassis with FusionIO, SSD and SAS Drives and no common backplane between the 4 nodes.  Along with my four pod node we added Arista 7124SX as our DC Switching/Fabric.  There are lots of details around this combination like currently Nutanix does not support using the Node for a bare metal server like you can do with UCS or other Blade Enclosures and the storage has limited access to the outside world (it is setup to presented to ESXi Hosts as iSCSI targets and VMs as ViSCSI targets) but so far I love the platform.  It gave me what I needed in the price point I needed and offers huge scale out options considering it is based of the GFS files system that Google uses across their DC’s. Read more

Method and Madness: How to fix what other people broke

I will be the first to admit that I make mistakes.  I make lots of mistakes.  But I learned a long time ago that it is not the mistakes that define us but
what we do when we make them.  To that point it is both fascinating and infuriating when I enter someone’s network land and find that they have gone out of their way to do nothing to solve their own problem or in many cases to even
build a proper functioning network.  So I am going to set out to show the Methods I use to deal with these people and their problems and the Madness that they have brought to the table to cause the issue.

This is going to be a series.  Really my first series of posts, and they will all fall under the heading of Method and Madness with some sort of witty little
tagline.  I will be pulling on a careers worth of situations that were totally preventable and how we resolved the problem or moved past it.

The format for each of this will be as follows:

Situation:  This will be the “lab” Scenario piece of the post in which I outline
the base problem as well as the end goals of the situation.  This section will
include a base diagram for use as reference.

The Method:  This will be the troubleshooting and resolution phase of the post and
will breakdown the ultimate solution or bypass to the problem.

The Madness: This section will be my analysis of how things got to this point in the first
place.  Keep in mind I have no in interest in posting garbage about oh look the senior
network engineer put a wrong static route in at 4am and a junior guy had to find it.
No I am more interested in exposing the issues where a Network Engineer or worse a team could not solve a problem that should be second nature for them to solve.  In doing so I hope others will learn from and not find themselves in these situation.

So with all that said look for them.  I hope to have the first one out by the end of
the week with about 50 Million other projects so bear with me.  I think these will be a great recap for me and both education and humorous for you.  I really encourage your feedback on these posts because I am not always right and there are lots of different paths so resolution of a problem.

Quick and Dirty…Ooohhhh….Yeahhhh

Quick and dirty is how I like it when I have 4000 menial tasks to get done.  So another oldy but goody that I had to dig up today was how to delete a full directory structure and its contents from a Cisco files system.  So here it is enjoy.

From normal enable mode:

delete /recursive /force flash:(enter the root file name)

So delete is the easy one.

/recursive sets the flag to recursively cycle through the whole directory structure you specified.  So you should probably never type

delete /recursive /force flash:  BAD DON’T DO IT!

And finally /force eliminates all the are you sure you don’t want to shoot yourself in the forehead messages.

Again quick and dirty saves time but if your dumb about using it can get you in trouble.

“the best $1.80 I ever spent”

If your not sure what the title quote is from Check out Young Guns 2.  As for this post however keep reading to see if I ever get to a point.  Currently it is 3:37am in the morning.  Again I can’t sleep.  Not sure if I rested to much today after a week of almost not sleep, drank a bit to much caffeine or if I am just currently unable to calm down during this career transition.  No matter what the cause I did what I normally do when I realize I wont sleep before I have to get dressed and head into work  (in this case a my old office, my current clients office and the new office by the end of the day) I took a shower shaved and put on some coffee.  I don’t know about you, but I do some of my best thinking in the shower or when driving on long trips alone.  It was durring the above mentioned shower that this post and the thoughts it contain popped into my racing mind.  Enjoy it could be a fun read.

Most of the feedback I have received concerning my new job and career path has been overwhelmingly positive.  But some has not.  Quite bluntly a few feel I have whored myself out for money.  So let me tell you what I told them…Yes as a matter of fact I did.  I did it for the money.

Read more

Josh’s Rules on Selling Solutions

So here I am at 2:15 AM.  Storming outside and me all hopped up on caffeine.  I just took a break from righting my last post:

Welcome to the HP Dream world where reality does not apply.

During that little breather I though you know what people must think I am a Cisco Zealot.  Well that could not be further from the truth.  I am a self confessed Technology Zealot.  It it is new, shiny, blinks, chirps or at some point in its lifecycle lived in a Data Center I want it.  But alas at least at this point in my life I have to make money.  I do that by working for a Cisco VAR.  We sell 90% plus Cisco.  Unlike past jobs I do not rep Juniper, HP, F5, Foundry…oops I mean Brocade, Arista, 3com, Shoretel, Avaya or anyone else that directly competes with Cisco Networking, Compute or Unified Communications.

That being said I do not think Cisco has the best product in every segment.  But I wont flesh that out on my blog.  If you want that info there is a price.  You are either a customer with a requirement I can’t meet at which point I will be honest with you or you are a professional friend who I feel comfortable discussion the finer and rougher points of our industry with.  What I will say though is I have some Rules for what I will sell and I wont sell.  I am going to lay those out to you and in a few cases why I feel how I feel.  I hope this will provide insight to others who design, sell and deploy solutions in our industry for clients.  At the end of the day our integrity is all we really have, Vendors crash, employers go under and clients come and go.

1.  If I wont run it in my basement I wont install it at a client!

Read more

TACACS+ on Nexus 7000

I have been through a couple of these Nexus deployments now that use a combination of 7Ks, 5Ks, and 2Ks. If you know anything about this platform you know that TACACS and AAA only really apply to the 7K and 5Ks. Here is my working template of what it takes to get these guys talking to and ACS server.

tacacs-server key 0 YOUR.ACS.KEY
tacacs-server host X.X.X.X
tacacs-server host X.X.X.X
tacacs-server host X.X.X.X
aaa group server tacacs+ GROUP.NAME
server X.X.X.X
server X.X.X.X
server X.X.X.X
source-interface YOUR.VLAN or YOUR.VRF or YOUR.ETHERNET

aaa authentication login default group GROUP.NAME
aaa authentication login console group GROUP.NAME
aaa authorization commands default group GROUP.NAME
aaa accounting default group GROUP.NAME
aaa authentication login error-enable Read more

Get Your ACS in Order!

ACS 1113 Appliance Password and IP Change Process:

1.  Insert ACS Recover CD into DVD-Drive
2.  Connect Console Cable (DB9 to DB9) to Laptop and Appliance
3.  Start Terminal Session with Following  (115200, 8, None, 1, NONE)
4.  Connect Monitor and Keyboard to ACS Appliance
5.  Power Cycle ACS Appliance
6.  Use Keyboard and mouse to Select Option 1 for Administrator Password Reset
7.  Remove Recovery CD from Appliance
8.  Press Enter on Keyboard to reboot appliance
9.  Disconnect Keyboard and mouse from Appliance
10. Wait approx 5 minutes for Console session to return.  (Don’t rush it, get a coffee or a snake then come back)
11. At login prompt user the Default = Administrator with no password.
12. You will be prompted to enter a new username.
13. You will be prompted to enter a new password, you will be prompted to enter this twice
14. Login with new Username and Password
15. Connect Ethernet Port 1 (Top Port) on Appliance to laptops ethernet port wait for green link light  (Without this step the appliance will not accept interface changes.)
16. Type “Set IP”  Follow the prompts to enter new IP information and select YES at the end
17. Type “Set domain” Follow the prompts to enter the new DNS prefix select YES at the end
18. Type reboot
19. Wait approx 5 minutes for Console session to return.  (Don’t rush it, get a coffee or a snake then come back)
20. Login with new Username and Password
21. Type Show to validate your config changes
22. Disconnect from laptop
23. Connect to production network
24. Done

DHCP is Doing What?

Recently, we had a branch site complete the renovation of a new wing. In doing so, we had to build a new telecom closet to feed the area. We added another stack of 3750’s, that connected back to the pre-existing closet via 1Gig fiber link (which that closet then connects back to our core via Optiman). We’ve basically added another stack to the same subnet for that site. Nothing out of the ordinary for us networkers.

Then a few weeks after they opened, they started having issues with certain pc’s in that new wing. There were approximately 15 pc’s that would have a daily Duplicate IP Address Conflict error message on them. So, Help Desk sends a ticket to our group. It was a Saturday afternoon, when I called the lady back and she confirms she has had these issues since they opened. A reboot fixes the issue, but only temporary for that shift. I promised this user that I would follow-up until we found the issue. Read more