Dealing with outliers the simple way using R

A decent portion of my new role entails getting my hands on data from various sources, “tidying” it up and putting it into a place where my team can make use of it. As such, I’ve found that even the most curated data sources have issues such as data-typing mistakes, duplicated imports of the same rows and just some general cruft. One of the first things I’ll do to try to gauge the quality of the data set is graph the number of rows per day. The kinds of things I’m interested in generally have a pretty narrow distribution of daily data points or observation counts which shouldn’t vary wildly from day to day.

Something simple like this will get us started:

If we pull those results into R we can visually assess if there’s anything amiss:

Hmm. A couple of days don’t look so good.

We can easily (i.e., lazily) identify the outlying data points using boxplot().

Yep. There are certainly some outliers there.

Boxplot stores the value of outliers which you can use to generate a subset of the original data containing just the rows containing those outliers. You can then go back to your import or ETL process and spot check those dates. In my case the import script had inadvertently been run twice generating duplicate rows on those days.

Hopefully that was helpful!

Until next time. Stay frosty.

Managing the Windows Time Service with SCCM’s Configuration Items

Keeping accurate and consistent time is important in our line of business. Event and forensic correlation, authentication protocols like Kerberos that rely on timestamps and just the simple coordination of things like updates all require accurate timekeeping. Computers are unfortunately notoriously bad at keeping time so we have protocols like NTP and time synchronization hierarchies to keep all the clocks ticking. In an Active Directory environment this is one of those things that (if setup correctly on the Domain Controller holding the PDC Emulator FSMO role) just kind of takes care of itself but what if you have WORKGROUP machines? Well. You’re in luck. You can use SCCM’s Configuration Items to manage configuration settings in devices that are outside of your normal domain environment in isolated networks and beyond the reach of tools like GPOs.

There’s really two pieces to this. We need to ensure that the correct NTP servers are being used so all our domain-joined and WORKGROUP machines get their time from the same source and we need to ensure that our NTP client is running.

 

Setting the correct NTP Servers for the Windows Time Service

To get started create a Configuration Item with a Registry Value based setting. The NTPServer value sets which NTP servers the Windows Time Service (W32Time) pulls from. We can manage it like so:

  • Setting Type: Registry Value
  • Data Type: String
  • Hive Name: HKEY_LOCAL_MACHINE
  • Key Name: SYSTEM\CurrentControlSet\Services\W32Time\Parameters
  • Value Name: NtpServer

The corresponding Compliance Rule is straight forward. We just want to ensure that the same time servers we are using in our domain environment are set here as well.

  • Rule type: Value
  • Setting must comply with the following value: youtimeserver1,yourtimeserver2
  • Remediate non-compliant rules when supported: Yes
  • Report noncompliance if this setting is not found: Yes

  • Rule Type: Existential
  • Registry value must exist on client devices: Yes

 

The Setting should require the existence of the NTPServer key and set its value as specified. If it is set to something else the value will be remediated back to your desired value. You can learn more about setting the NTPServer registry key values and controlling the polling interview at this Microsoft MSDN blog post.

 

Ensuring the Windows Time Service is Running

If the time service isn’t running then you are not going to have accurate time keeping! This is further complicated by the behavior of the Window Times Service on WORKGROUP computers. The time service will stop immediately after system startup, even if the Startup Type is set to Automatic. The W32Time service is configured as a Trigger-Start service in order to reduce the number of services running in Windows 7 and Server 2008 R2 and above. The trigger (of course) that causes it to automatically start is whether or not the machine is domain-joined so for WORKGROUP machines the service status is set to Stopped. Not very helpful in our scenario. Let’s change that.

We can start by just performing a simple WQL query to see if the W32Time service is running:

  • Setting type: WQL Query
  • Data type: String
  • Namespace: root\cimv2
  • Class: Win32_Service
  • Property: Name
  • WQL query WHERE clause: Name like “%W32Time%” and State like “%Running%”

It’s a bit backward but if the query comes back with no results then the configuration state we are looking for does “not exist” and so we’ll mark it as non-compliant. It’s not intuitive but it works:

  • Rule Type: Existential
  • Registry value must exist on client devices: Yes

 

This gives us the status of the Windows Time Service but we still need to remove the [DOMAIN JOINED] trigger so the service will actually start automatically. PowerShell to the rescue!

  • Setting Type: Script
  • Data Type: Interger
  • Script: PowerShell

Discovery Script

Remediation Script

 

  • Value returned by the specified script: Equals 0
  • Run the specified remediation script when this setting is noncompliant: Yes

The Discovery script will return various non-compliant values depending on the configuration state of the endpoint. This will then cause the Remediation script to run which sets the service’s Startup Type to Automatic, removes the [DOMAIN JOINED] trigger and starts the service.

I hope this posts helps you manage your time configuration on all those weird one-off WORKGROUP machines that we all seem to have floating around out there.

Until next time, stay frosty.

Using Azure Cool Blob Storage with Veeam 9.5u3

“Hey, AKSysAdmin. I want to push all our backups to cheap Azure storage. Can you do a proof-of-concept for me and a quick cost write up?”

We are all eagerly awaiting the implementation of Veeam’s Scale-Out Backup Repository Archive Tier functionality in v10. The Archive Tier functionality will allow Veeam customers to leverage to “cheap” cloud storage like AWS’s S3 and Glacier and Azure’s rather hilariously named Cool Blob Storage. In the meantime if you wanted to use Azure Blob Storage right now what are your options?

  • A “middleware” appliance like NetApp’s AltaVault, Microsoft’s StorSimple or a VTL
  • Roll your own IaaS solution in Azure

The first option is pretty straight-forward. You buy an appliance that provides a storage target for your on-prem Veeam Backup and Replication server and send your Backup Copy jobs to that Backup Repository. Once your backups are located there, “magic” happens that handles the hot/warm/cold tier-ing of the data out to Azure as well as the conversion from structured data to unstructured data.

The second option is a little more complicated. You’ll need to spin up an Azure IaaS VM, attach blob storage to it and make it usable to your on-prem Veeam infrastructure.

 

Before we go too much further we should probably talk about the different blob storage types.

Block Blobs

These are pretty much what they sound like, block based storage of large contiguous files. They work great for things are not accessed via randomized read and writes. The individual blocks stored in each blob are referenced by a BlockID and can be uploaded/modified/downloaded simultaneously, assembled and then committed with a single operation . You can see how well this type of storage lends itself to streaming services where large files are split into smaller pieces and uploaded or downloaded sequentially.  The maximum size (as of writing) for a block blob is about 4.75TBs.

Page Blobs

Page blobs are composed of 512-byte pages optimized for random read and write operations. Changes to the pages require immediate commits unlike block blobs. Page blobs work great for things like virtual disks where some other mechanism is organizing the data inside the blob. Page blobs are used for the underlying storage for Azure IaaS data disks. The maximum size (as of writing) for a page blob is 8TBs.

Azure Blob Storage Tiers: Hot, Cool and Archive

Azure Storage Accounts allow you group all your various pieces of blob storage together for the purposes of management and billing. With Blob and General Purpose v2 Storage Accounts you can elect to use storage tiers. Cool Blob Storage has lower storage costs (and higher access costs) and is intended for things like short-term backup and disaster recovery data. Archive storage has even lower storage costs (and even higher access costs) and is designed for data that can tolerate hours of potential retrieval time. Archive storage is intended for long-term backups, secondary backup storage or data that has archival requirements. In order to read the data in an archive storage the blob needs to be rehyrdated which can take up to 15 hours. Blob size is also a factor in rehyrdation time.

I should mention that the option to have your blobs stored in locally redundant storage (LRS) or globally redundant storage (GRS) exists for all of these flavors.

 

This is all great but how do I use it?

Well if you went with the first option you break out your wallet for a capital purchase and follow Veeam’s Deployment Guide for AltaVault or vendor equivalent.

The second option is a little more involved. You need to deploy an instance of Veeam’s Cloud Connect for the Enterprise, add some data disks to the resulting Azure Iaas VM, configure them in Windows, setup a Backup Repository using them and finally add the resulting repository to your on-prem install as a Cloud Backup Repository. For the price of the IaaS VM and the underlying storage you now have a cloud-based backup repository using Azure blob storage.

Here’s why you probably don’t want to do this.

Veeam will support Azure Cool Blob storage fairly soon so you have to ask yourself if it makes sense to buy a purpose built “middleware” appliance to bridge the gap. A few years ago it would of been a no-brainer but with more and more backup vendors supporting cloud storage natively it seems like market for these devices will shrink.

The second option has some issues as well. Your freshly created Cloud Backup Repository is backed by Azure IaaS data disks which sit on top of page blob storage. Guess what page blobs don’t support? Storage tiers. If you create a storage account in the cool tier you’ll notice the only container option you have is for block blobs. If you try and add a data disk to your IaaS VM using a blob storage account you get this error:

Not going to work.

What if you setup a Azure File Storage container and utilized it instead of a data disk? Same problem. Only block blob storage supports archiving tiers at this point in time.

What if you just provisioned extra data disks for your VM, and use Storage Spaces and ReFS to get your storage? Well that will sort of work but there many limitations:

  • Data disks are limited to 4TBs
  • Most IaaS VMs only support 15 data disks
  • If you need more than 15 data disks your IaaS VM is going to get really expensive
  • You have to correctly manage and configure a VM with 15 disks using Storage Spaces
  • All your disks are running on page blob storage which is not really that cheap

The “roll-your-own-IaaS” solution will be performance and capacity limited right out of the gate. It will be complicated and potentially brittle and it doesn’t take advantage of the pricing of storage tiers making it rather pointless in my opinion.

Why you still may want to do this

If the backup dataset that you want to archive is fairly small this might still make sense but if that’s the case I would forgo the entire exercise of trying to cram a round peg into a square hole and look very seriously at a DRaaS provider like Iland where you will get so much more than just cloud storage for your backups for what will likely be a competitive price.

Why even if you still want to do this it’s probably not a good idea

Everything is elastic in the cloud except the bill and unless you have an accurate picture of what you really need you might be surprised once you get that bill. There is a bunch of things that are not really accounted for in your traditional on-premise billing structure: IP addresses, data transfer between virtual networks, IOPS limited performance tiers and so on. In short, there is a lot more to doing the cost analysis than just comparing the cost of storage.

Speaking of – let’s take a look at the current storage prices and see if they really are “cheap”. These prices are based on the Azure Storage Overview pricing and are located in the WestUS2 region of Azure ComCloud.

Standard Page Blobs (Unmanaged Disks)

LRS ZRS GRS RA-GRS
$0.045 per GB N/A per GB $0.06 per GB $0.075 per GB

This also comes with a $0.0005 per 10,000 transactions charge when Standard Page Blobs are attached to a VM as an Unmanaged Disk.

 

Block Blob Pricing

Hot Cool Archive
First 50 terabyte (TB) / month $0.0184 per GB $0.01 per GB $0.002 per GB
Next 450 TB / Month $0.0177 per GB $0.01 per GB $0.002 per GB
Over 500 TB / Month $0.017 per GB $0.01 per GB $0.002 per GB

There are also some operational charges and data transfer costs

Write Operations* (per 10,000) $0.05 $0.10 $0.10
List and Create Container Operations (per 10,000) $0.05 $0.05 $0.05
Read Operations** (per 10,000) $0.004 $0.01 $5
All other Operations (per 10,000), except Delete, which is free $0.004 $0.004 $0.004
Data Retrieval (per GB) Free $0.01 $0.02
Data Write (per GB) Free $0.0025 Free

 

To replace our rather small GFS tape set we’d need somewhere north of 100TBs. The first problem is with the limitation requiring us to use page blob backed data disks, we won’t even be able to meet our capacity requirements (4TBs per data disk, 15 data disks per IaaS VM = 60 TBs).

If we put aside the capacity issue, let’s look at a notional cost just for comparison’s sake: 100TBs * 1024 = 102,400 GBs * $0.045 = $4,608 per month. This doesn’t include the cost of the IaaS VM and associated infrastructure you may need (IP addresses, Virtual Networks, Site-to-Site VPN, etc.) nor any of the associated transaction charges.

The storage charge is more than expected since we’re not really using the technology as intended. Block blob storage in the archive tier gets us a much more respectable number: 100TBs * 1024 = 102,400 GBs * $0.002 = $204.8 per month. BUT we need to factor in the cost of some kind of “middleware” appliance to utilize this storage so tack on an extra $40-$60k (it’s hard to pin this cost down since it will come via a VAR so I could be totally off). If we “op-ex” that cost over three years it’s an additional $1388.00 a month bringing your total to $1593.68 per month for “cheap” storage.

OK. Looks like our “cheap” cloud storage may not be as cheap as we thought. Let’s take a look at our on-premise options.

LTO data tapes… personally I loath them but they have their place. Particularly for archiving GFS data sets that are small. A 24 slot LTO-6 tape library like Dell’s TL2000 is around $20k and 40 LTO-6 tapes with a raw capacity of 100TBs (not including compression) comes to about $602 per month over three years.

What about on-premise storage? A Dell MD1400 with 12 10TB 7.2K RPM NLSAS drives is somewhere in the $15-$20k range and brings 80TBs of usable storage in RAID-60 configuration. Allocated out over three years this comes to roughly $555 per month.

Summary

Technology choices are rarely simple and no matter how much executive and sales folks push “cloud-first” like it’s some kind of magic bullet, cloud services are a technology like any other with distinct pros and cons, use cases and pitfalls. Getting an accurate picture of how much it will cost to shift a previously capital expense based on-premise service to cloud services is actually a fairly difficult task. There are a tremendous amount of things that you get “included” in your on-premise capital purchases that you have to pay for every month once that service is in the cloud and unless you have a good grasp on them you will get a much bigger bill than you expected. I really recommend SysAdmin1138’s post about the challenges of moving an organization to this new cost model if you are considering any significant cloud infrastructure.

If you want to use Azure Blob Storage right now for Veeam the answer is: You can but it’s not going to work the way you want, it’s probably going to cost more than you think and you’re not really using the technology the way it was intended to be used which is asking for trouble. You could buy some middleware appliance but with Scale-Out Backup Repository Archive Tier functionality on the immediate horizon this sounds like a substantial infrastructure investment that you’re only going to get limited return of business value on. It might make sense to wait.

Finally a little bit of disclaimer. I tried to pull the pricing numbers from old quotes that I have (hence the LTO-6 and LTO-8 tapes) to try and keep the math grounded in something like reality. Your prices may vary wildly and I highly encourage you to compare all the different cost options and spend some time to try to capture all of the potential costs of cloud services that may be hidden (i.e., it’s not just paying for the storage). Cloud services and their pricing are constantly changing too so it’s worth checking with Microsoft to get these numbers from the source.

Until next time, stay frosty.

Scheduling Backups with Veeam Free and PowerShell

Veeam Free Edition is an amazing product. For the low price of absolutely zero you get a whole laundry list of enterprise-grade features: VeeamZip (Full Backups), granular and application-aware restore of items, native tape library support and direct access to NFS-based VM storage using Veeam’s NFS client. One thing that Veeam Free doesn’t include however is a scheduling mechanism. We can fix that with a little bit of PowerShell that we run as a scheduled task.

I have two scripts. The first one loads the Veeam PowerShell Snap-In, connects to the Veeam server, gets a list of virtual machines and then backs them up to a specified destination.

 

I had Veeam setup on a virtual machine running on the now defunct HumbleLab. One of the disadvantages of this configuration is I don’t have separate storage to move the resulting backup files onto. You could solve this by simply using an external hard drive but I wanted something a little more… cloud-y. I setup Azure Files so I could connect to cheap, redundant and most importantly off-site, off-line storage via SMB3 to store a copy of my lab backups. The biggest downside to this is security. Azure Files is really not designed to be a full featured replacement for a traditional Windows file server. It’s really more of SMB-as-a-Service offering designed to be programmatically accessed by Azure VMs. SMB3 provides transit encryption but you would still probably be better off using a Site-to-Site VPN between your on-prem Veeam server and a Windows file server running as VM in Azure or by using Veeam’s Cloud Connect functionality. There’s also no functionality replacing or replicating NTFS permissions. The entire “security” of your Azure Files SMB share rests in the storage key. This is OK for a lab but probably not OK for production.

Here’s the script that fires off once a week and copies the backups out to Azure Files. For something like my lab it’s a perfect solution.

 

Until next time, stay frosty!

Morale, Workload and Tickets – A Follow-Up

You guys remember Elaine right? Six months ago, (Morale is Low, Workload is Up) I looked into some ticketing system data to try to discover what is going on in our team and how our seemingly ever increasing workload was being distributed along staff.  We unfortunately came to some alarming conclusions and hopefully mitigated them. I checked in with her recently to see how she was doing. Her response?

Things are just great. Just greatttt…

Let’s dig back in and see what we find. Here’s what I’m hoping we see:

  • Elaine’s percentage of our break/fix work drops to below 20%. She was recently promoted to our tier-2 staff and her skill set should be dedicated more towards proactive work and operational maintenance.
  • George and Susan have come onto their own and between the two of them are managing at least 60% of the ticket queue. They’re our front line tier-1 staff so I would expect the majority of the break/fix work goes to them and they escalate tickets as necessary to other staff.
  • Our total ticket volume drops a bit. I don’t think we’re going to get back to our “baseline” from 2016 but hopefully August was an outlier.

 

Well, shit. That’s not what I was hoping to see.

That’s not great but not entirely unexpected. We did take over support for another 300 users in August so I would expect that workload would increase which it is has by roughly double. However it is troubling because theoretically we are standardizing that department’s technology infrastructure which should lead to a decline in reactive break/fix work. Standardization should generate returns in reduced workload. If we add them into our existing centralized and automated processes the same labor hours we are already spending will just go that much further. If we don’t do that, all we have done is just add more work, that while different tactically, is strategically identical. This is really a race against time – we need to standardize management of this department before our lack of operational capacity catches up with us and causes deeper systemic failures pushing us too far down the “reactive” side of operations that we can’t climb back up. We’re in the Danger Zone.

 

This is starting to look bad.

Looking over our ticket load per team member things are starting to look bleaker. Susan and George are definitely helping out but the only two months where Elaine’s ticket counts are close to them was when she was out of the office for a much needed extended vacation. Elaine is still owning more of the team’s work than she should, especially now that she’s nominally in a tier-2 position. Lets also remember than in August when responsibility for those additional 300 users was moved to my team, along with Susan and George, we lost two more employees in the transition. That works out to a 20% reduction of manpower and that includes our manager as a technical asset (which is debatable). If you look at just reduction in line staff it is even higher. This is starting to look like a recipe for failure.

 

Yep. Confirmed stage 2 dumpster fire.

Other than the dip in December and January when Elaine was on vacation things look more or less the same. Here’s another view of just the tier-1 (George, Frank and Susan) and tier-2 (Elaine and Kramer) staff:

Maybe upgrade this to a stage 3 dumpster fire?

I think this graph speaks for itself. Elaine and Susan are by far doing the bulk of the reactive break/fix work. This has serious consequences. There is substantial proactive automation work that only Elaine has the skills to do. The more of that work that is delayed to resolve break/fix issues the more reactive we become and the harder it is to do the proactive work that prevents things from breaking in the first place. You can see how quickly this can spiral out of control. We’re past the Danger Zone at this point. To extend the Top Gun metaphor – we are about to stall (“No! Goose! Noo! Oh no!”). The list of options that I have, lowly technical lead that I am, is getting shorter. It’s getting real short.

In summation: Things are getting worse and there’s no reason to expect that to change.

  • Since August 2016 we have lost three positions (30% reduction in workforce). Since I started in November 2014 we have seen a loss of eight positions (50% reduction in workforce).
  • Our break/fix workload has effectively doubled.
  • We have had a change in leadership and a refocusing of priorities on service delivery over proactive operational maintenance which makes sense because the customers are starting to feel the friction. Of course with limited operational capacity putting off PM for too long is starting to get risky.
  • We have an incredibly uneven distribution of our break/fix work.
  • Our standardization efforts for our new department are obviously failing.
  • It seems less likely every day that we are going to be able to climb back up the reactive side of the slope we are on with such limited resources and little operational capacity.

Until next time, Stay frosty.

 

SCCM, Asset Intelligence and Adobe SWID Tags, Part II – Custom Fields

There’s some crucial information in Adobe’s SWID tags that does not get automatically collected in SCCM’s Asset Intelligence process via the SMS_SoftwareTag class.

This information is contained in the optional fields of the SWID tag and will allow you to “decode” the LeID field and determine important information for your Enterprise Term Licensing Agreement (ETLA).

 

We’re talking things like whether or not the product is licensed via a volume licensing agreement or a retail license, the activation status along with the particular version of the product (Standard vs. Pro) but as previously mentioned this information is unfortunately not pulled up in the SMS_SoftwareTag class inventory process.

I came across this blog post by Sherry Kissinger and largely cribbed this idea from her. We can use/abuse Configuration Items to run a script that parses the SWID tag files, inserts them into a WMI class that then gets collected via Hardware Inventory and from there we can run reports off of it. Sherry’s post provided the .MOF file and a VB script but I only really speak PowerShell so I rewrote her script

 

 

Set this script up in a Configuration Item, add it to a Baseline and then deploy it. When the Configuration Item runs, it should find any Adobe SWID tags, parse them and create a custom instance of a WMI class (cm_AdobeInfo) containing all of goodies from the SWID tag.

 

By adding Sherry’s custom .MOF file to your Hardware Inventory class settings you can have the SCCM agent pull this class up as part of the Hardware Inventory Cycle.

 

With the following bit of SQL you can build another nice Manager Approved (TM) report:

 

Until next time, stay frosty!

The HumbleLab: Desk of Death – Build a PXE Boot DBAN Station using pfSense and CentOS

I am become Death, destroyer of data.

Data sanitation is a required albeit tedious part of being a Systems Administrator. Last week I noticed that we had quite a few old machines piling up around our office and finding myself with a few spare hours I decided to see if I could speed up the process of scrubbing their hard drives using the ubiquitous Derik’s Boot and Nuke (DBAN). An important note, the mathematics and mechanics of using disks scrubbing software is actually fairly complex, especially when Solid State Drives are involved. My recommendation to you is to check with your security team to make sure your method of disk scrubbing is sufficient. Deguassing and physical destruction using an NSA approved methodology may be the only approved method for certain types of organizations. If you don’t have a security team or a regulatory compliance scheme that you have to comply with that specifies what your media disposal standards are the NIST standards are great place to start.

Disclaimers out of the way, now onto the fun stuff. Here’s what the old system looked like: A help desk tech would boot a CD, load DBAN into memory, select the drive/s to be wiped and then press Enter. This took a few minutes in addition to the time it took to setup the old workstation with power, a keyboard and a monitor. Here’s what I wanted: Plug in a monitor, keyboard and a network cable, turn the computer on, Press F12, pick Network Boot and move on.

To do this I setup an isolated VM on my lab (The HumbleLab) to run a TFTP service in conjunction with the DHCP services already offered by pfSense. Using an old workstation would of also worked just fine if you didn’t have an existing lab setup.

1. Install CentOS and configure networking

You will need to get a minimal install of CentOS setup. There are plenty of guides out there but this one is pretty nice. There’s always the official Red Hat documentation too. Configure a static IP address, fire off a quick su -c yum update to update your system and that should be enough to get you going. I had two physical NICs in my Hyper-V hosts so I dedicated one to my “DBAN” network, created a External Virtual Network with it and added a vNIC connected to that virtual switch to my new CentOS DBAN server.

2. Install and configure the TFTP service

Now we need to get our SYSLINUX bootloaders installed and a TFTP service setup. Lets install both and copy the SYSLINUX bootloaders into the tftpboot path.

Create a pxelinux.cfg directory along with a default configuration file.

Fire up your favorite text editor and edit the newly created configuration file:

Make sure that clients can pull the resulting files from the tftpboot directory:

Last but not least go ahead and start the TFTP service. You could choose to enable the service so it starts automatically but I personally like to start and stop it at will as a kind of weak safety check since it does automatically boot DBAN without user intervention.

 

3. Create a new interface and DHCP scope in pfSense

Unfortunately I lost my screenshots here so we’ll have to just go by feel. You will need to perform the following steps:

  • Add a new vNIC to your pfSesne VM that is connected to your DBAN External Virtual Network.
  • Use the pfSense web interface to assign the new vNIC as an interface.
  • Create a corresponding DHCP scope for your DBAN subnet. This is a good place to stop and test your current configuration. Plug a laptop in and see if you get a DHCP lease and test connectivity to both the pfSense interface and the CentOS DBAN server. If you’re not getting a lease and/or cannot contact both of those interfaces you will need to correct whatever is wrong with your network configuration before you proceed.
  • Modify your DHCP scope to “enable network booting”. Specify the IP address of the DBAN server, set the filename to ‘pxelinux.0’

4. DESTROY!

Plug a victim computer into your DBAN network choose boot from network. You should be presented with that glorious blue screen of impending doom.

 

Deploying VLC Media Player with SCCM

VLC Media Player is an F/OSS media player that supports a dizzying array of media formats. It’s a great example of one of those handy but infrequently used applications that are not included in our base image but generate help desk tickets when an user needs to view a live feed or listen to a meeting recording. Instead of just doing the Next-Next-Finish dance, lets package and deploy it out with SCCM. The 30 minutes to package, test and deploy VLC will pay us back in folds when our help desk no longer has to manually install the software. This reduces the time it takes to resolve these tickets and ensures that the application gets installed in a standardized way.

Start by grabbing the appropriate installer from VideoLAN’s website and copying to whatever location you use to store your source installers for SCCM. Then fire up the Administrative Console and create a New Application (Software Library – Applications – Create Application). We don’t get an .MSI installer so unfortunately we are actually going to have to do a bit of work, pick Manually specify the application information.

Next up, fill out all the relevant general information. There’s a tendency to skimp here but you might as well take the 10 seconds to provide some context and comments. You might save your team members or yourself some time in the future.

I generally make an effort to provide an icon for the Application Catalog and/or Software Center as well. Users may not know what “VLC Media Player” is but they may recognize the orange traffic cone. Again. It doesn’t take much up front work to prevent a few tickets.

Now you need to add a Deployment Type to your Application. Think of the Application as the metadata wrapped around your Deployment Types which are the actual installers. This lets you pull the logic for handling different types of clients, prerequisites and requirements away from other places like separate Collections for Windows 7 32-bit and 64-bit clients and just have one Application with two Deployment Types (a 32-bit installer and a 64-bit installer) that gets deployed to a more generic Collection. As previously mentioned, we don’t have an .MSI installer so we will have to manually specify the deployment installation/uninstallation strings along with the detection logic.

  • Installation: vlc-2.2.8-win32.exe /L=1033 /S –no-qt-privacy-ask –no-qt-updates-notif
  • Uninstallation: %ProgramFiles(x86)%\VideoLAN\VLC\uninstall.exe /S

If you review the VLC documentation you can see that /L switch specifies the language, /S switch specifies a silent install and the –no-qt-privacy-ask –no-qt-updates-notif sets the first-run settings so users don’t receive the prompt.

Without having a MSI’s handy ProductCode for setting up our Detection Logic we will have to rely on something a little more basic: Checking to see if the vlc.exe is present to tell the client whether or not the Application is actually installed. I also like to add a Version check as well so that older installs of VLC are not detected and are subsequently eligible for being upgraded.

  • Setting Type: File System
  • Type: File
  • Path: %ProgramFile(x86)%\VideoLAN\VLC
  • File or folder name: vlc.exe
  • Property: Version
  • Operator: Equals
  • Value: 2.2.8

Last but not least you need to set the User Experience settings. These are all pretty self explanatory. I do like to actually set the maximum run time and estimated installation time to something relevant for the application that way if the installer hangs it doesn’t just sit there for two hours before the agent kills it.

 

From there you should be able to test and deploy your new application! VLC Media Player is a great example of the kind of “optional” that you could just deploy as Available to your entire workstation fleet and close tickets requesting a media player with instructions on how to use the Software Center.

 

 

Until next time, stay frosty!

SCCM, Asset Intelligence and Adobe SWID Tags

Licensing. It is confusing, constantly changing and expensive. It is that last part that our managers really care about come true-up time and so a request in the format of, “Can you give me a report of all the installs of X and how many licenses of A and B we are using?” comes across your desk. Like many of the requests the come across your desk as a System Administrator these can be deceptively tricky. This post will focus on Adobe’s products.

How many installs of Adobe Acrobat XI do we have?

There are a bunch of canned reports that help you right off the bat under Monitoring – Reporting – Reports – Software – Companies and Products. If you don’t have a Reporting Services Point installed yet then get on it! The following reports are a decent start:

  • Count all inventoried products and versions
  • Count inventoried products and versions for a specific product
  • Count of instances of specific software registered with Add or Remove Programs

You may find that these reports are less accurate that you’d hope. I think of them as the “raw” data and while they are useful they don’t gracefully handle things like the difference between “Adobe Systems” and “Adobe Systems Inc.” and detect those as two separate publishers. Asset Intelligence adds a bit of, well, intelligence and allows you to get reports that are more reflective of the real world state of your endpoints.

Once you get your Asset Intelligence Synchronization Point installed (if you don’t have one already) you need to enable some Hardware Inventory Classes. Each of these incurs a minor performance penalty during the Software Inventory client task so you probably only want to enable the classes you think you will need. I find the SMS_InstalledSoftware and SMS_SoftwareTag classes to be the most useful by far so maybe start there.

You can populate these WMI classes by running the Machine Policy Retrieval & Evaluation Cycle client task followed by the Software Inventory cycle. You should now be able to get some juicy info:

 

Lots of good stuff in there, huh? Incidentally if you need a WMI class that tracks software installs to write PowerShell scripts against SMS_InstalledSoftware is far superior to the Win32_Product class because any queries to Win32_Product will cause installed MSIs to be re-configured (KB974524). This is particularly troublesome if there is a SCCM Configuration Item that is repeatedly doing this (here).

There are some great reports that you get from SMS_InstalledSoftware:

  • Software 0A1 – Summary of Installed Software in a Specific Collection
  • Software 02D – Computers with a specific software installed
  • Software 02E  – Installed software on a specific computer
  • Software 06B – Software by product name

All those reports give you a decent count of how many installs you have of a particular piece of software. That takes care of the first part of the request. How about the second?

 

What kind of installs of Adobe Acrobat XI do we have?

Between 2008 and 2010 Adobe started implementing the ISO/IEC 19770-2 SWID tag standard in their products for licensing purposes. Adobe has actually done a decent job at documenting their SWID tag implementation as well as provided information on how decode the LeID. The SWID tag is an XML file that contains all the relevant licensing information for a particular endpoint, including such goodies as the license type, product type (Standard, Pro, etc.) and the version. This information gets pulled out of the SWID tag and populates the SMS_SoftwareTag class on your clients.

 

That’s a pretty good start but if we create a custom report using the following SQL query we can get something that looks Manager Approved (TM)!

See the follow-up to this post: SCCM, Asset Intelligence and Adobe SWID Tags, Part II – Custom Fields

Until next time, stay frosty.

Morale Is Low, Workload Is Up

Earlier this month, I came back from lunch and I could tell something was off. One of my team members, lets call her Elaine, who is by far the the most upbeat, relentlessly optimistic and quickest to laugh off any of our daily trials and tribulations was silent, hurriedly moving around and uncharacteristically short with customers and coworkers. Maybe she was having a bad day I wondered as I made a mental note to keep tabs on her for the week to see if she bounced back to her normal self. When her attitude didn’t change after a few days then I was really worried.

Time to earn my team lead stripes so I took her aside and asked her what’s up. I could hear the steam venting as she started with, “I’m just so f*****g busy”. I decided to shut up and listen as she continued. There was a lot to unpack: She was under-pressure to redesign our imaging process to incorporate a new department that got rolled under us, she was handling the majority of our largely bungled Office 365 Exchange Online post-migration support and she was still crushing tickets on the help desk with the best of them. The straw that broke the camel’s back – spending a day to clean-up her cubicle that was full of surplus equipment because someone commented that our messy work area looked unprofessional…  “I don’t have time for unimportant s**t like that right now!” as she continued furiously cleaning.

The first thing I did and asked her what the high priority task of the afternoon was and figured out how to move it somewhere else. Next I recommended that she finish her cleaning, take off early and then take tomorrow off. When someone is that worked up, myself included, generally a great place to start is to get some distance between you and whatever is stressing you out until you decompress a bit.

Next I started looking through our ticket system to see if I could get some supporting information about her workload that I could take to our manager.

Huh. Not a great trend.

That’s an interesting uptick that just so happens to coincide with us taking over the support responsibilities for the previously mentioned department. We did bring their team of four people over but only managed to retain two in the process. Our workload increased substantially too since we not only had to continue to the maintain the same service level but we now have the additional challenge of performing discovery, taking over the administration and standardizing their systems (I have talked about balancing consolidation projects and workload before). It was an unfortunate coincidence that we had to schedule our Office 365 migration at the same time due to a scheduling conflict. Bottom line: We increased our workload by a not insignificant amount and lost two people. Not great a start.

I wonder how our new guys (George and Susan) are doing? Lets take a look at the ticket distribution, shall we?

Huh. Also not a great trend.

Back in December 2016 it looks like Elaine started taking on more and more of the team’s tickets. August of 2017 was clearly a rough month for the team as we started eating through all that additional workload but noticeably that workload was not being distributed evenly.

Here is another view that I think really underlines the point.

Yeah. That sucks for Elaine.

As far back as a year Elaine has been handling about 25% of our tickets and since then her percentage of the tickets has increased to close to 50%. What makes this worse is not only has the absolute quantity of tickets in August more than doubled compared to the average of the 11 preceding months but the relative percentage of her contribution has doubled as well. This is bad and I should of noticed, a long time ago.

Elaine and I had a little chat about this situation and here’s what I distilled out of it:

  • “If I don’t take the tickets they won’t get done”
  • “I’m the one that learns new stuff as it comes along so then I’m the one that ends up supporting it”
  • “There’s too many user requests for me to get my project work done quickly”

Service Delivery and Business Processes. A foe beyond any technical lead.

This is where my power as a technical lead ends. It takes a manager or possibly even an executive to address these issues but I can do my best to advocate for my team.

The first issue is actually simple. Elaine needs to stop taking it upon herself to own the majority of the tickets. If the tickets aren’t in the queue then no one else will have the opportunity to take them. If the tickets linger, that’s not Elaine’s problem, that’s a service delivery problem for a manager to solve.

The second issue is a little harder since it is fundamentally about the ability of staff to learn as they go, be self-motivated and be OK with just jumping into a technology without any real guidance or training. Round after round of budget cuts has decimated our training budget and increased our tempo to point where cross training and knowledge sharing is incredibly difficult. I routinely hear, “I don’t know anything about X. I never had any training on X. How am I supposed to fix X!” from team members and as sympathetic as I am about how crappy of a situation that is there is nothing I can do about it. The days of being an “IT guy” that can go down The Big Blue Runbook of Troubleshooting are over. Every day something new that you have never seen before is broken and you just have to figure it out.

Elaine is right though – she is punching way above her weight, the result of which is that she owns more and more the support burden as technology changes and as our team fails to evenly adopt the change. A manager could request some targeted training or maybe some force augmentation from another agency or contracting services. Neither are particularly likely outcomes given our budget unfortunately.

The last one is a perennial struggle of the sysadmin: Your boss judges your efficacy by your ability to complete projects, your users (and thus your boss’ peers via the chain of command) judge your efficacy by your responsiveness to service requests. These two standards are in direct competition. This is such as common and complicated problem that there is a fantastic book about it: Time Management for Systems Administrators

The majority of the suggestions to help alleviate this problem require management buy-in and most of them our shop doesn’t have: A easy to use ticket system with notification features, a policy stating that tickets are the method of requesting support in all but the most exigent of circumstances, a true triage system, a rotating interrupt blocker position and so on. The best I can do here is to recommend to Elaine to develop some time management skills, work on healthy coping skills (exercise, walking, taking breaks, etc.) and doing regular one-on-one sessions with our manager so Elaine has a venue for discussing these frustrations privately so at least if they cannot be solved they can acknowledged.

I brought a sanitized version of this to our team manager and we made some substantial progress. He reminded me that George and Susan have only been on our team for a month and that it will take some time for them to come up to speed before they can really start eating through the ticket queue. He also told Elaine, that while her tenacity in the ticket queue is admirable she needs to stop taking so many tickets so the other guys have a chance. If they linger, well, we can cross that bridge when we come to it.

The best we can do is wait and see. It’ll be interesting to see what happens as George and Susan adjust to our team and how well the strategy of leaving tickets unowned to encourage team members to grab them works out.

Until next time, stay frosty.