Where have all the cowboys gone?

No, I don’t mean like the song, I mean, where have all the geeks gone in IT?

I was reading an article / blog / posting about Developers vs. Coders, and boy was that guy right on! His point is that we don’t have Developers anymore, we have coders. Developers know the (dare I say a term used around here a lot) end-to-end picture of the problem whereas coders are people who come to work, code VB, Java, etc all day and go home, never to work on a computer until they return to work. The business I work for is loaded with coders, and very few Developers.

Geeks, Similar to Developers who know their whole environment, Geeks are given a problem in IT, and can usually understand the whole picture (Network, Server, App, O/S, Hardware, etc), and wrap their brains around it to come up with the proper solution.

What happened to all the geeks in IT? Ithink they quit and went to work for Google?

What I find we have are a bunch of IT people that know little or nothing of IT. This has always been an issue in the Microsoft space, with MCSE badges flashed at a whim, but it’s getting worse every day.

For example, lets look at the people who work on Windows XP/NT/200x. I am currently working on an issue with a memory leak in a vendors program. Now memory leaks are a bitch, and tough to find. Try to find a regular IT person who can find the memory leak is close to impossible. Whats Ironic about this particular topic, is in the NT 4.0 Resource Kit Book that MS used to sell, they spent almost two(2) chapters explaining how to use all the tools to find memory leaks, and even gave you a sample program that simulated a leaky program. You think any MCSE has ever read that book? Unlikely.

I know ‘true’ computer types don’t like to see the MCSE badge promoted because they feel it doesn’t stand for anything, and I tend to agree, to a large extent. Most MCSE’s couldn’t troubleshoot their way out of a paper bag without calling Microsoft first.

But this isn’t just true of MS type, some, and very few Linux types have limited knowledge of what they are truly working on. There are some linux types that site outside my office, that similar to coders, will SSH into a box, run a predefined script, copy some files, and that is the extent of their knowledge, yet they are considered IT.

So where have all the Geeks in IT gone?

I

Keep the managers out of the system – Please!

Throughout my enterprise IT career I have had a variety of managers, some bad, some good and some great.

One of the main philosophys I try to make sure my managers live by is “Stay out of the system and I’ll be able to keep it running”, but you always get a lose cannon or two in your career, and I have had two.

When I had just started at a large enterprize media company in the late 90’s, I had a manager who was assigned to our group because basically we didn’t have anyone else, and I certainly wasn’t qualified. So, OK, this guy (we’ll call him Bill) was assigned as our manager of the email group.

We ran a shop of Exchange 5.5 servers, servicing aproximately 4500 clients. One of the things Bill insisted on was having full access to Exchange and the domain the servers were in. I asked why would a manager (who knows nothing of running a system like Exchange) need access to the system, and his reply “Don’t ask, just do it!”. OK Bill, you now have access.

About a week later, one of our systems alerted that it was being rebooted. I called Bill and asked if he knew anything about it:

Me: Bill, we just got an alert that serverX rebooted, you know anything about this?

Bill: Yea, it wasn’t working right, so I rebooted the server

Me: You what? Why would you reboot the server?

Bill: Don’t worry about it.

Me: What do you mean don’t worry about it?

Bill: Just deal with it!

I mentioned this incident to our (then) Operations manager. Suffice it to say, Bill was not longed to be our manager much longer.

A few managers came and went until we merged with another large media company, and I got another new manager (we’ll call him BennyV). He had a few more qualifications than Bill in running Exchange, but my philosophy still applied, why do managers need access to the system. Well BennyV would have nothing of it, again, he (seriously) demanded access to the environment.

Luckly for us, BennyV wasn’t as stupid as Bill and never really got into the system to do things, he just liked the satisfaction of knowing he had the keys to the kingdom. We did however have a fun time one evening with him and our team.

Seems BennyV told management that the Exchange password was not all that secure (it wasn’t but it was the least of our security concerns). So he promissed management we would change it.  Now obviously BennyV (being the guru of Exchange that he was) didn’t realize the work involved in doing this, but none the less, we setup a team task to change the Exchange password on 20+ servers in one of our sites.

During the task, the entire messaging team and BennyV were on the phone. Also, we had a group IM chat going, which did not include BennyV. Here is an exerpt from that chat:

Group member 1: Hey, does anyone know what the new password is going to be

Me: I’m generating that now, stand by

Group member 2: I have an idea

Group memeber 2: K33pB33nYV0Ut

Group member(x): smilies

(Now for those of you who don’t speek l33t (leet speak), that translates to Keep BennyV out.)

When our team member wrote that in the chat, I started laughing so hard I almost cried. All BennyV could say is “Whats so funny? Whats so Funny?”, which of course made me laugh even harder. It set the mood that night, for what could have been a long, and very boring task, but having that fun in the beginning made it all worth while.

OMG… I can’t connect to the Internet, someone help

So it’s 4am the day after Thanksgiving, and I’m sleeping peacefully, when I get a call from our Help Desk.

Seems the users in UK can’t connect to the Internet on their Blackberries?

Now I don’t seem to be unreasonable, but come on, you can’t connect to the Internet on your Blackberry? This seems like trying to ride a bicycle on the freeway, doesn’t it. Now to be fair, it is middle of the UK day.

The worst part about all of this, is the philosophy in this damm company, that every problem gets the same amount of urgency as a every other problem regardless of how rediculous it is (like this one). It’s not until our resources run out, does management tell them to start prioritizing our staff to handle the most important calls. Never mind I need some damm sleep!

What I would rather see is that they let us know it’s not working (i.e. the Blackberry browser) and we take non-priority time to fix it, like when it’s not 4am.

Give me a minute to figure it out…

I have many gripes about this place, but one of them is how IT is treated when things break.

I don’t know of any system that’s perfect (ok, well maybe DOS), but in any case, why does this place expect everything to be perfect? It’s not so much that they expect the systems to be perfect, but rather, Management doesn’t even give us a chance to look around and figure out what the #$#@$ went wrong.

As I usually do, here is a typical example:

0:00 hr System X goes down

0:05 hr Me: Gee, let me look at System X and see if I can get it up again….

0:06 hr IT Management Help Desk joins the call.

0:07 hr IT Management: What’s wrong?

Me: System X is down, i’m trying to see if I can get it back up.

IT Management: Well what happened?

Me: I don’t know, i’m working on getting the system back up right now, please wait a sec

IT Management: Well how can we keep this from happening again?

Me: I don’t know yet, I’m still working on getting the system back up, once I finish that I can try to determine what broke it.

IT Mgmt: So any luck with that, getting it up yet

Me: I’ve only been look at this for less than 5min, please wait.

IT Mgmt: So, what do you think caused the issue, should we call the vendor, contact other groups?

Me: Can you please just give me a 10 minutes or so to look at what happened and then I’ll tell you what I need, thank you.

IT Mgmt: OK, but as soon as you know what went wrong, we should figure out how to prevent this in the future.

Me (to myself) : Gee, really, you mean we don’t want systems just going down randomly for no reason – Jeezz!

Do you see the crazyness here? I don’t have 10 frigin minutes to see what the hell went wrong with something before people want an explination as to why. Also, if the problem persists for more than 30min, we are supposed to contact the CTO of the Company (a $15B, 25k user company), for ANY incident over 30min, can you imagine!

Blackberry 4.x Design 101

So, when our corporate parent was given the go ahead to deploy our new E2K3 mail system, our business unit (ok me) insisted they deply Blackberry 4.0 (at the time, 4.1 is out now).

We had several discussions on why we wanted 4.0 (funny that I’d have to explain 4.0 reasons to a company with over 15k devices, but oh well). One of the discussions went something like this:

Business IT: So, when you deploy BES 4.0 we want it hosted on SQL server, not MSDE.

Corporate IT: No, that’s not how we do it.

Business IT: Why?

Corporate IT: Because we’re not an SQL shop, we just do email.

Business IT: I run email now for the busines, AND I can deploy BES to SQL, what’s so hard about that?

Corporate IT: That’s not how we deploy BES, and we are in charge of the new BES Servers, so we will deploy them on MSDE.

I knew why I wanted the to deploy with SQL: non-impacting BES moves for users, wireless provisioning, etc.

Turns out though, there is a major performance limitation that Blackberry has built into the 4.x server if you run on MSDE only. Since 4.x is so dependent on the SQL portion of the server now, they limit how much load they’ll put onto the MSDE database, and consequently your overall performance and capacity suffer. I quote from the blackberry performance guide for 4.0 “If you are enabling more than 500 BlackBerry users, you should use Microsoft SQL Server 2000.“

So now, corporate has deployed 4 BES 4.0 servers with MSDE, and we have the following issues:

1. No ability to move users between servers, i.e. NYC to Los Angeles. Movement of a user from one server to the other requires a whole wipe and resync
2. Performance issues. We consistently see 2-3 thread hangs a day with our current setup, whereas the SQL version in 3.6 didn’t have anywhere near that many
3. No option for wireless provisioning. Since they have 4 seperate databases, provisioning wirelessly is close to impossible.

With a little research and a call to Blackberry’s technical experts, our corporate IT guys would have known that SQL was the way to go, and we would be enjoying a much better 4.0 experience.

Blackberry policy 101

So our corporate folk love to use India for their IT stuff, I get that. But why is everyone surprised when the following scenario happens:

We had a blackberry user who kept losing his ability to use send when replying to email on the device.  In researching this problem, we found that several things could be causing it, but most likely it was a bad MIME Service book (sounds complicated?).

In troubleshooting this issue we got a bunch of us together to try and fix it. Now remember, we don’t have any IT access to the new coporate system, that’s run by ABC out of India, so all we can do is help them best we can. Keep in mind that ABC India runs the corporate mail system, including Blackberry
So on the phone we have Me, Blackberry and a corporate representative from ABC offshore. The conversation goes something like this:

• Blackberry : “OK, I’d like to do some testing on this clients device, can you make sure that user’s blackberry doesn’t have any policies on it”
• …. Silence….
• Me: “Hello, ABC, are you there?”
• ABC: “Yes, what do you need?”
• Me: “Blackberry just asked you a question, can you remove the policy from the users blackberry?”
• ABC: “Sure, we can help with that”….. “Umm, how do you do that?”
• Blackberry: “You don’t know how to move the user into a no-policy group?”
• ABC: “Oh, no, can you help me with that?”
• Blackberry: “OK, click ….”

Now, I know it doesn’t sound like a big deal, but ABC India runs blackberry for 20K blackberry users, and they don’t know how to do policy changes for a user?

I sure hope they are getting a good deal with all this!

What you get with India and a process stack.

One of the things our corporate parents like to do is outsource most of their IT to India (I personally have come to love the India culture and it’s people, but I will admit it takes some getting used to, I digress). They have a model I’ve been told is something like 70/70 (70% of IT should be outsourced, and 70% of the outsourced, should be offshore)

Our corporate parents love a strict process, like manufacturing, and they continually think it’s applicable to IT. Given that you can define a strict road map for each process, you should be able to have anyone follow that process – right? This is what makes them think that you can take any IT job and move it to India (again, you might be able to, but not the way they do it). With this concept, you end up with is a set of people in India who know a set of processes, not the technology and nothing more. So anything outside of the process set, you get, well, crap.

They can’t even deploy a video player?

So the company is deploying a video player “they wrote” to the Internet in a couple of weeks, for a finance news channel and others we have.

Apparently in their testing of this player, a few issues were found with certain desktops, but no one ever decided or was able to figure them out.  So, just weeks before a public deployment of this player, they throw the issue over the fence to the west coast team, and within 2hrs, the west coast team has figured it out.

I’m just starting this… bear with me

So I’m gonna try this blogging thing again.  I tried it once before on blogspot or something like that and it wasn’t all that easy.  Anyway…

This blog is called “The Chaos of working here…” it is a bunch of stories about where I work and the stupid crap we have to do and put up with around here, it’s pretty scary.
So I won’t be telling you WHERE I work, you’ll figure it out from some of the many acronyms we have around here, our name is one of them – hint #1.

Enjoy my chaos!