I have been thinking a lot about documentation lately, mostly about my own shortcomings and trying to understand why the act of documenting seems so difficult and why the quality of the documentation that does get done is often found lacking. Good documentation and good documentation practices are such a fundamental part of the health of an IT shop you would think we as a field would be better at it. My experience is limited and anecdotal (whose is not?) but I have yet to see a shop with solid documentation and solid documentation practices. This extends to myself as well. I can look back at my various positions and roles and there are very few where I actually felt satisfied with the quality of my documentation.
Read on for “aksysadmin’s made-up principles of how to not suck at documentation and do other things good too”.
1. Develop a standardized format, platform and process from the bottom up. Your team uses this, not you.
Leadership has a tendency to standardize on a single standard, platform or process. This is generally considered a good thing. The problem is, leadership does not write technical documentation. We do. And what standard makes sense to them, may not make any sense to the technical staff (*cough* ITIL *cough*). What platform seems adequate to them, may seem unwieldy to sysadmins (*cough* SharePoint *cough*). Standardization may generally be considered a good thing but forcing a standard, platform or process on a team without input or understanding their problem domain and use case is generally considered a bad thing. The harder you make it for your team to document, the less likely they will be to perform a task they are already unlikely to perform.
2. Don’t document how, document why
This is part an internal challenge (IT staff documenting the wrong things) and an external challenge (leadership requiring the wrong kind of things to be documented). I see lots of documentation that is essentially a re-hashed version of a vendor’s manual. Ninety-nine percent of the time your vendor has exhaustive resources on how to do something. It is right there. In the manual. Go read it. Unless it is incredibly unintuitive, and sometimes it is, why would you waste your precious time re-writing an authoritative set of information into a non-authoritative set that requires your team to maintain it? Reading and understanding vendor documentation should be considered a fundamental skill, if your guys cannot read or are unwilling to read vendor manuals you have other problems that need addressing.
What you should document, is why you did things. You will not remember why this particular group was setup or why things are this way instead of that way in six months and your successor certainly will have no idea. Use your documentation to provide context and meaning.
3. Document where to find things
Documenting why something is the way it is great but it is also important to document where things are. I am talking about, things like IP Addresses, Organizational Charts, Passwords and so on. This is another opportunity to avoid work, err work more efficiently. Chances are many of these things have authoritative sources maintained by other people or tools. Why write your IP Addresses down manually in an Excel Spreadsheet when you can use a tool like IPAM to track them? Why track the phone tree for your different workgroups when Active Directory can do that for you? Why spend time doing stuff that is already done? Why indeed?
Figuring out what stuff to document in the where category can be hard to do. I have found the easiest way to do this is to pretend you are brand new. Better yet if you have a brand new team member ask him to track these kinds of information requests as he acquaints himself to your particular little piece of hell. What does he need to know right now to do your job? That is what your replacement will be asking himself after you have ascended.
4. Don’t document break/fix issues
Do not fill up your Wiki, SharePoint, OneNote or file share full of Word .docx with break/fix issues. Your infrastructure and process documentation should be broad and “provide context and meaning” which is pretty much the opposite kind of information than the kind break/fix issues are about – specific configurations, systems or problems.
You already have a place to “document” break/fix issues – it is called your damn ticket system. Use it. Document your fixes in your tickets. If your Tier-1 guys have a habit of closing all but the simplest of tickets with “done” or “fixed”, slap them (and probably yourself as well) and say that their future self just came back in time to hit them for making their job harder. If you do not have ticket system then you have other problems that need to be addressed.
5. Have a panic page
Take the really important stuff from the why documentation and the where documentation and make it into a panic page. A panic page is a short piece of documentation that contains all the information you or anyone else would need to have in order to deal with a “whoops” situation. Think things like vendor contact phone numbers, contract support entitlements, how to file and escalate a case and maybe where to find your co-worker’s emergency scotch. I borrowed this one from my supervisor and it turned out to be prescient suggestion on his part.
6. Have a hard copy
This is an extension of the panic page principle. Panic situations have a way of making electronic documentation inaccessible. “Oh but wait my documentation is in the cloud, I can get to it anywhere with my mobile device, oh I am so smart” you say, yeah well, you will be screwed when you drop your iPad or it runs out of batteries or you happen to live in Alaska which has comparable infrastructure to, say Afghanistan. Have a paper copy on hand, preferably two. Yes it will be harder to maintain but it will be a lot better than having no documentation if your file server explodes or the polar bears take over your data center.
7. Have designated documentation days
As a sysadmin, you generally do not have luxury of setting your own priorities. If your leadership wants documentation, instead of just saying “Hey, we need to document better” at your weekly staff meeting, they need to make it a priority. Nothing does this better than designating a day for documentation. Read-Only Friday is a good one because you are not making changes on Friday anyway, right… righhhhtttt? Of course, you are still going to get interruptions and tickets so designate one person as the interruption blocker and another team member as the documenter (borrowed from the Tom Limoncelli’s excellent Time Management for System Administrators). Rotate individuals as appropriate. These designated documentation days are your time, to make time, to actually Get Shit Done. All those little notes you meant to flesh out with some more context but never had time to… do it now. Organize your stuff. Clean it up. Review it for accuracy. Do this with a frequency related to how fast things change. Until leadership makes it a priority, you will always have another one that trumps it.
These ideas address some of those external and internal challenges you may have and that I know I have. I am more inclined to document stuff if I am not documenting dumb stuff that is already documented elsewhere. I will have an easier time finding the documentation we do have, if it is organized in a way that works for me and my team, both the consumer and producer of it. If I have dedicated time to actually perform the act of documenting it will probably get done. If not, then I am answerable to someone. Of course there can be a large gap between knowing what needs improvement and actually fixing it. Until then.