Monthly Archives: May 2016

One Year of Solitude: My Learning Experience as a Lead

It has been a little over a year since I stepped into a role as a technical lead and I thought this might be a good time to reflect on some of the lessons I have learned as I transition from being focused entirely on technical problems to trying to under how those technical pieces fit into a larger picture.

 

Tech is easy. People are hard. And I have no idea how to deal with them.

It is hard to understate this. People are really, really difficult to deal with compared to technology and I have so much to learn about this piece of sysadmin craft. I do not necessarily mean people are difficult in the sense that they are oppositional or hard to work with (although often they are) just that team dynamics are very complicated and the people composing your team have a huge spread in terms of experience, skills, motivations, personalities and goals. These underlying “attributes” are not static either, they change based on the day, the mood and the project making identifying, understanding them and planning around them even harder. The awareness of this underlying milieu composing your team members and thus your team is paramount to your project’s success.

All I can say is that I have just begun to develop an awareness of these “attributes” and am just getting the basics of recognizing different communication styles (person and instance dependent). I can just begin to tell whose motivations align with mine and whose do not. In hunting we call this the difference between “looking and seeing”. It takes a lot of practice to truly “see”, especially if like me, you are not that socially adept.

My homework in this category is to build an RPG-like “character sheet” for each team member, myself included,  and think about what their “attributes” are and where those attributes are strengths and where they can be weaknesses.

 

Everyone will hate you. Not really. But kinda yes.

One the hardest parts of being a team lead, is you are now “in-charge” of technical projects with a project team made up of many different members who are not within your direct “chain-of-command” (at least this is how it works in my world). This means you own the responsibility for the project but any authority you have is granted to you by a manager somewhere higher up the byzantine ladder of bureaucracy. Nominally, this authority allows you to assign and direct work directly related to the project but in practice this authority is entirely discretionary. You can ask team member A to work on item Z but it is really up to her and her direct supervisor if that is what she is going to do. In the hierarchical, authority-based culture and process driven business world that most of us of work in this means you need to be exceedingly careful about whose toes you step on. Authority on paper is one thing, authority in practice is entirely another.

 

Mo’ People, Mo’ Problems

My handful of project have thus far been composed of team members that kind of fall into these rough archetypes.

A portion of the team will be hesitant to take up the project and the work you are asking them to do since you are not strictly speaking their supervisor. They will passively help the project along and frequently you will be required to directly meet with them and/or their supervisor to make sure they are “cleared” for the work you assigned them and to make sure they feel OK about doing it. These guys want to be helpful but they don’t want to work beyond what their supervisor has designated. Get them “cleared” and make sure they feel safe doing the work and you have made a lot of progress.

Another portion of the team will be outright hostile. Either their goals or motivations do not align with the project or even worse their supervisor’s goals or motivations do not align with the project but someone higher up leaned on them and so they are playing along. This is tough. The best you can hope for here is to move these folks from actively resisting to passively resisting. They might be “dead weight” but at least they aren’t actively trying to slow things down any more. I don’t have much a working strategy here – an appeal to authority is rarely effective. Authority does not wanted to bothered by your little squabbles and arguably it has already failed because chain-of-command can make someone play along, but it cannot make they play nice. I try to tailor my communication style to whatever I am picking up from these team members (see the poorly named, Dealing with People You Can’t Stand), do my best to include them (trying to end-run them makes things ten times worse) and inoculate the team against their toxicity. I am fan of saying I deal with problems and not complaints because problems can actually be solved but a lot of times these folks just want to complain. Give them a soap box so they can get it out of their system so you can move on get work done but don’t let them stand on it for too long.

Another group will be unengaged. These poor souls were probably assigned to the project because their supervisor had to put someone on it. A lot of times the project will be outside their normal technical area of operations, the project will only marginally effect them, or both. They will passively assist where they can. The best strategy I have found here is to be concise, do your best not to waste their time, and use their experience and knowledge of the surrounding business processes and people as much as you can. These guys can generate some great ideas or see problems that you would never otherwise see. You just have to find a way to engage them.

The last group will be actively engaged and strongly motivated to see the project succeed. These folks will be doing the heavy lifting and 90% of the actual technical work required to actually accomplish the project. You have to be careful to not let these guys lean to hard on the other team members out of frustration and you have to not overly rely on them or burn them out otherwise you will be really screwed since they are actually the only people truly putting in the nuts-and-bolts work required for the project’s success.

A quick aside, if you do not have enough people in this last group the project is doomed to failure. There is no way a project composed mostly of people actively resisting its stated goals will succeed, at least not under my junior leadership.

Dysfunctional? Yes. But all teams are dysfunctional in certain ways and at certain times. Understanding and adapting to the nature of your team’s dysfunction lets you mitigate it and maybe, just maybe, help move it towards a healthier place.

Until next time, good luck!

A Ticket Too Far… Breaking the Broken

A funny thing happened a while back, one of my manager’s asked me to stop creating tickets on the behalf of customers. This, uh, well, this kind of made me pause for a few reasons. The first and most obvious one is that I cannot remember shit. I always feel terrible when I forget someone’s request and I feel doubly terrible when I forgot it due to an oversight as simple as getting a ticket. The second, is that it is generally considered a Good Thing (TM) to track your customer requests. I won’t even bother supporting that proposition because Tom Limoncelli has pretty much got that covered in Time Management for System Administrators.

The justification for this directive is pretty simple and common-sense and is a great example of how a technical person like me with the best of intentions can actually develop some self-sabotaging behavior.

  • Tickets created for customers by myself with my notes in them are confusing to Tier-1/Tier-2 support folks. It looks like I created the ticket but forgot to own it and I am still working the issue where in actuality I bumped the request all the way back down to Tier-1 where it should of started. Nothing makes a ticket linger in limbo longer, than looking like someone is working it but not being owned by anyone. This tendency for tickets to live in limbo is exacerbated because our ticket system does not support email notification.
  • Customers are confused when a Tier-1/Tier-2 person calls them after picking up a ticket from the queue and asks, “Hey there, I am calling about Request #234901 and your <insert issue here>”.
  • Finally and most importantly, it does nothing to help correct the behavior of customers and teach them the one true way to request assistance from IT by submitting a ticket.

OK. Rebuttal time! (Which sounds kind of weird when you say it out loud). The first two points are largely an artifact of our ticketing system and/or its implementation.

The ticket queue is actually a generic user in the ticket system that tickets can get assigned to by customers. There is no notification when a ticket is created and assigned to this queue, nor any when a ticket is assigned to you. The lack of notification requires a manager or a lead on our team to police the queue, assign tickets to line staff based on who they think is best suited to work a particular issue and then finally notify them via email, phone or in-person.

The arguably confusing series of events where a ticket is created on behalf of a user is again, mainly a technical fault of the system. The requester is set to the customer but line staff that picks up the ticket may just read the notes which has my grubby hands all over them… so whose issue is it? Mine or the customer’s?

That being said – both of these points could largely be alleviated by a smarter ticket system that had proper notification and our Tier-1 guys reading the notes a little more carefully. I can forgive them their trespass since they are extremely interrupt driven and have a tendency to shoot tickets first and ask questions later but still, the appropriate context is there.

The last point, the idea that creating tickets re-enforces bad end-user behavior, is by far the most salient one in my opinion. If you let people get away with not submitting tickets you are short-changing yourself and them. I won’t get credit for the work, the work won’t be documented, we won’t have accurate metrics and I am about 1000% more likely to forget the request and never do it.

Problem: We don’t have a policy requiring users to submit a ticket for a request, it’s more like a guideline. And the further up the support tiers you go, the fewer and fewer requests have tickets. This leaves my team in an interesting spot, we either create the ticket for the customer, tell the customer we won’t work the issue if they don’t create a ticket first (kind of a dick move, especially when there is no teeth to our policy) or not create a ticket at all.

Conclusion: Right idea but we are still focused on the symptom and not the cause. Let’s review.

  • The ticket system has technical deficiencies that lead to less than ideal outcomes. It makes it cumbersome for both technical staff and customers to use it and relies on staff doing the very thing ticket systems are supposed to reduce, interrupt people to let them know they have work assigned to them.
  • A policy is not useful if it does not have teeth. I already feel like a jerk telling a customer “Hey. I am working with another team/customer/whatever but if you submit a ticket someone will take a look”, when they are standing in my cubicle with big old doe eyes. I especially feel like a jerk when I do not even have a policy backing me up. Paraphrasing, Tom Limoncelli, “Your customers judge your competency by your availability. Your manager judges it by your completion of projects. These two dual requirements are directly opposed and balancing them is incredibly important.”
  • By the time I am creating a ticket on behalf of a customer the battle is already lost. I’ve already been interrupted with all the lost efficiency and the danger of mistakes that comes with it.
  • The customers that do not submit tickets get preferential treatment. They get to jump ahead of all the people that actually did submit tickets which hardly seems fair. All that is happening here is that we are encouraging the squeaky wheels to squeak louder.
  • The escalation chain gets skipped. A bunch of these kind of issues should be caught at Tier-1 and Tier-2. By skipping right to Tier-3, we are not applying our skills optimally and also depriving the Tier-1 and Tier-2 guys the chance to chew on a meatier problem. A large part of the reason I am creating tickets for customers is to bump the request back down to Tier-1 and Tier-2 where it should have been dealt with to begin with.

Creating tickets on the behalf of customers is not the problem. It is a symptom of deeper issues in Process and Technology. These issues will not be resolved by no longer generating tickets for customers. Customers will still skip the escalation chain, we will continue to re-enforce bad behavior, less issues will get recorded, our Tier-2 and Tier-3 will still be interrupt driven regardless of whether there is a ticket or not. All that will change is that we will be more likely to forget requests.

The technical problems can be resolved by implementing a new ticket system or by fixing our existing one. The policy problems can be solved by creating a standardized policy for all our customers and then actually ensuring that it has teeth. The people problems can be fixed by consistent and repeated re-training.

That covers the root cause but what about now? What do we do?

  • We create the ticket for the customer – We cannot really do that. It disobeys a directive from leadership and it has all the problems discussed above.
  • We tell the customer to come back with a ticket – This does not really address the root cause, annoys the customers and we do not have policy backing it up. It is not really an option.
  • Do not use a ticket to track the request – And here we are by process of elimination. If things are broken sometimes the best way to fix them is let them break even further.

Until next time . . .

When the world gets bad enough, the good go crazy, but the smart…they go bad.Evil Abed