Tag Archives: sccm

Deploying VLC Media Player with SCCM

VLC Media Player is an F/OSS media player that supports a dizzying array of media formats. It’s a great example of one of those handy but infrequently used applications that are not included in our base image but generate help desk tickets when an user needs to view a live feed or listen to a meeting recording. Instead of just doing the Next-Next-Finish dance, lets package and deploy it out with SCCM. The 30 minutes to package, test and deploy VLC will pay us back in folds when our help desk no longer has to manually install the software. This reduces the time it takes to resolve these tickets and ensures that the application gets installed in a standardized way.

Start by grabbing the appropriate installer from VideoLAN’s website and copying to whatever location you use to store your source installers for SCCM. Then fire up the Administrative Console and create a New Application (Software Library – Applications – Create Application). We don’t get an .MSI installer so unfortunately we are actually going to have to do a bit of work, pick Manually specify the application information.

Next up, fill out all the relevant general information. There’s a tendency to skimp here but you might as well take the 10 seconds to provide some context and comments. You might save your team members or yourself some time in the future.

I generally make an effort to provide an icon for the Application Catalog and/or Software Center as well. Users may not know what “VLC Media Player” is but they may recognize the orange traffic cone. Again. It doesn’t take much up front work to prevent a few tickets.

Now you need to add a Deployment Type to your Application. Think of the Application as the metadata wrapped around your Deployment Types which are the actual installers. This lets you pull the logic for handling different types of clients, prerequisites and requirements away from other places like separate Collections for Windows 7 32-bit and 64-bit clients and just have one Application with two Deployment Types (a 32-bit installer and a 64-bit installer) that gets deployed to a more generic Collection. As previously mentioned, we don’t have an .MSI installer so we will have to manually specify the deployment installation/uninstallation strings along with the detection logic.

  • Installation: vlc-2.2.8-win32.exe /L=1033 /S –no-qt-privacy-ask –no-qt-updates-notif
  • Uninstallation: %ProgramFiles(x86)%\VideoLAN\VLC\uninstall.exe /S

If you review the VLC documentation you can see that /L switch specifies the language, /S switch specifies a silent install and the –no-qt-privacy-ask –no-qt-updates-notif sets the first-run settings so users don’t receive the prompt.

Without having a MSI’s handy ProductCode for setting up our Detection Logic we will have to rely on something a little more basic: Checking to see if the vlc.exe is present to tell the client whether or not the Application is actually installed. I also like to add a Version check as well so that older installs of VLC are not detected and are subsequently eligible for being upgraded.

  • Setting Type: File System
  • Type: File
  • Path: %ProgramFile(x86)%\VideoLAN\VLC
  • File or folder name: vlc.exe
  • Property: Version
  • Operator: Equals
  • Value: 2.2.8

Last but not least you need to set the User Experience settings. These are all pretty self explanatory. I do like to actually set the maximum run time and estimated installation time to something relevant for the application that way if the installer hangs it doesn’t just sit there for two hours before the agent kills it.

 

From there you should be able to test and deploy your new application! VLC Media Player is a great example of the kind of “optional” that you could just deploy as Available to your entire workstation fleet and close tickets requesting a media player with instructions on how to use the Software Center.

 

 

Until next time, stay frosty!

SCCM, Asset Intelligence and Adobe SWID Tags

Licensing. It is confusing, constantly changing and expensive. It is that last part that our managers really care about come true-up time and so a request in the format of, “Can you give me a report of all the installs of X and how many licenses of A and B we are using?” comes across your desk. Like many of the requests the come across your desk as a System Administrator these can be deceptively tricky. This post will focus on Adobe’s products.

How many installs of Adobe Acrobat XI do we have?

There are a bunch of canned reports that help you right off the bat under Monitoring – Reporting – Reports – Software – Companies and Products. If you don’t have a Reporting Services Point installed yet then get on it! The following reports are a decent start:

  • Count all inventoried products and versions
  • Count inventoried products and versions for a specific product
  • Count of instances of specific software registered with Add or Remove Programs

You may find that these reports are less accurate that you’d hope. I think of them as the “raw” data and while they are useful they don’t gracefully handle things like the difference between “Adobe Systems” and “Adobe Systems Inc.” and detect those as two separate publishers. Asset Intelligence adds a bit of, well, intelligence and allows you to get reports that are more reflective of the real world state of your endpoints.

Once you get your Asset Intelligence Synchronization Point installed (if you don’t have one already) you need to enable some Hardware Inventory Classes. Each of these incurs a minor performance penalty during the Software Inventory client task so you probably only want to enable the classes you think you will need. I find the SMS_InstalledSoftware and SMS_SoftwareTag classes to be the most useful by far so maybe start there.

You can populate these WMI classes by running the Machine Policy Retrieval & Evaluation Cycle client task followed by the Software Inventory cycle. You should now be able to get some juicy info:

 

Lots of good stuff in there, huh? Incidentally if you need a WMI class that tracks software installs to write PowerShell scripts against SMS_InstalledSoftware is far superior to the Win32_Product class because any queries to Win32_Product will cause installed MSIs to be re-configured (KB974524). This is particularly troublesome if there is a SCCM Configuration Item that is repeatedly doing this (here).

There are some great reports that you get from SMS_InstalledSoftware:

  • Software 0A1 – Summary of Installed Software in a Specific Collection
  • Software 02D – Computers with a specific software installed
  • Software 02E  – Installed software on a specific computer
  • Software 06B – Software by product name

All those reports give you a decent count of how many installs you have of a particular piece of software. That takes care of the first part of the request. How about the second?

 

What kind of installs of Adobe Acrobat XI do we have?

Between 2008 and 2010 Adobe started implementing the ISO/IEC 19770-2 SWID tag standard in their products for licensing purposes. Adobe has actually done a decent job at documenting their SWID tag implementation as well as provided information on how decode the LeID. The SWID tag is an XML file that contains all the relevant licensing information for a particular endpoint, including such goodies as the license type, product type (Standard, Pro, etc.) and the version. This information gets pulled out of the SWID tag and populates the SMS_SoftwareTag class on your clients.

 

That’s a pretty good start but if we create a custom report using the following SQL query we can get something that looks Manager Approved (TM)!

 

Until next time, stay frosty.

Five things to not screw up with SCCM

With great power comes great responsibility

Uncle Ben seemed like a pretty wise dude when when he dropped this particular knowledge bomb on Peter Parker. As sysadmins we should already be aware of the tremendous amount of power that has been placed into our hands. Using tools like SCCM further serve to underline this point and while I think SCCM is an amazing product and has the ability to be a fantastic force multiplier you can also reduce your business’ infrastructure to ashes within hours if you use it wrong. I can think of two such events where an SCCM Administrator has mistakenly done some tremendous damage: In 2014 a Windows 7 deployment re-imaged most of the computers, including their servers at Emory University and another unfortunate event where a contractor managed to accomplish the same thing at the Commonwealth Bank of Australia back in the early 2000s.

There are a few things you can do to enjoy the incredible automation, configuration and standardization benefits of SCCM while reducing your likelihood of an R.G.E.

Dynamic Collection Queries

SCCM is all about performing an action on large groups of computers. Therefore it is absolutely imperative that your Collections ACTUALLY CONTAIN THE THINGS YOU THINK THEY DO. Your Collections need to start large and gradually get smaller using a sort of matryoshka doll scheme based on dynamic queries and limiting Collections. You should double/triple/quadruple check your dynamic queries to make sure they are doing what you think they are doing when you create them. It is wise to review these queries on a regular basis to make sure an underlying change in something like Active Directory OU structure or naming convention hasn’t caused your query to match 2000 objects instead of your intended 200. Finally, I highly recommend spot-checking Collection members of your targeted Collection before deploying anything particular hairy and/or when deploying to a large Collection because no matter how diligent we are, we all make mistakes.

Maintenance Windows

“The bond traders are down! The bond traders are down! Cry and hue! Panic! The CIO is on his way to your boss’s office!” Not what you want to hear at 7:00 AM as you are just starting on your first cup of coffee, huh? You can prevent this by making sure your Maintenance Windows are setup correctly. SCCM will do what you tell it to do and if you tell it to allow the agent to reboot at 11:00AM instead of 11:00PM, that’s what’s going to happen.

I like setting up an entirely separate Collection hierarchy that is used solely for setting Maintenance Windows and include my other Collections as members. This prevents issues where the same Collection is used for both targeting and scheduling. It also reduces Maintenance Window sprawl where machines are members of multiple Collections all with different Maintenance Windows. It’s important to consider that Maintenance Windows are “union-ed” so if you have a client in Collection A with a Maintenance Window of 20:00 – 22:00 and in Collection B with a Maintenance Window of 12:00 – 21:00 that client can reboot anywhere between 12:00 – 22:00. There’s nothing more annoying than a workstation that was left in a forgotten testing Collection with a Maintenance Window spanning the whole business day – especially after the technician was done testing and that workstation was delivered to some Department Director.

I am also a huge fan of the idea of a “Default Maintenance Window” where you have a Maintenance Window that is in the past and non-reoccurring that all SCCM clients are a member of. This means that no matter what happens with a computer’s Collection membership it isn’t just going to randomly reboot if it has updates queued up and its current Maintenance Window policy is inadvertently removed.

Last but not least, and this goes for really anything that is scheduled in SCCM, pay attention to date and time. Watch for AM versus PM, 24-hour time vs. 12-hour time,  new day rollover (i.e., 08/20 11:59PM to 08/21 12:00PM) and UTC versus local time.

Required Task Sequences

Of all the things in SCCM this is probably one of the most dangerous. Task Sequences generally involve re-partitioning, re-formatting and re-imaging a computer which has the nice little side effect of removing everything previously on it. You’ll notice that both of those incidents I mentioned at the start of this post were caused by Task Sequences that inadvertently ran on a much larger group of computers than was intended. As a general guideline, I council staff to avoid deploying Task Sequences as Required outside of the Unknown Computers Collection. The potential to nuke your line of business application servers and replace them with Windows 10 is reduced if you have done your fundamentals right in setting up your Collections but I still recommend deploying to small Collections, making your Deployment Available instead of Required (especially if you are testing), restricting who can deploy Task Sequences and password protecting the Task Sequence. I would much rather reboot severs to clear the WinPE environment than recover them from backups.

Automatic Deployment Rules

Anything in SCCM that does stuff automatically deserves some scrutiny. Automatic Deployment Rules are another version of Dynamic Collection Queries. You want to use them and they make your life easier but you need to be sure that they do what you think that they do, especially before they blast out this month’s patches to the All Clients collection instead of the Patch Testing collection. Deployment templates can make it harder to screw up your SUP deployments and once again pay attention to the advertisement and deadline time watching for mistakes with UTC vs. local time or +1 day rollover, the Maintenance Window behavior and which collection you are deploying to. And please, please, please test your SUP groups first before deploying them widely. You too can learn from our mistakes.

Source Files Management and Organization

A messy boat is a dangerous boat. There is a tendency for the source files directory that you are using to store all your installers for Application and Package builds to just descends into chaos over time. This makes it increasingly difficult to figure out what installers are still being used and what stuff was part of some long forgotten test. What’s important here is that you have a standard for file organization and you enforce it with an iron fist.

I like to break things out like this:

A picture depicting the Source Files folder structure

Organizing your source files… It’s a Good Thing.

It’s a pretty straight forward scheme but you get the idea: Applications – Vendor – Software Title – Version and Bitness – Installer. You may need to add more granularity to your Software Updates Deployment Package folders depending on your available bandwidth and how many updates you are deploying in a single SUP group. We have had good results with grouping them by year but then again we are not an agency with offices all over rural Alaska.

 

Mitigation Techniques

There are a few techniques you can use to prevent yourself from doing something terrible.

Roll-based Access Control

You can think of Security Scopes as the largest possible number of clients a single admin can break. If you have a big enough team, the clever use of RBAC will allow you limit how much damage individual team members can do. For example: You could divide your 12 person SCCM team into three sub-teams and use RBAC to limit each sub-team to only being able to manage 1/3 of your clients. You could take this idea a step further and give your tier-1 help desk the ability to do basic “non-dangerous” actions but still allow them the ability to use SCCM to perform their job. This is pretty context specific but there is a lot you can do with RBAC to limit the potential scope of an Administrator’s actions.

Application Requirements (Global Conditions)

You can use Application Requirements as a basic mechanism to prevent bad things from happening if they are deployed to the wrong Collection inadvertently.

Look at all these nice, clean servers… it would be a shame if someone accidentally deployed the Java JRE to all of them, wouldn’t it? Well, if you put in a Requirement that checks the value of ProductType in the Win32_OperatingSystem WMI class to ensure the client has a workstation operating system then the Application will fail its Requirements check and won’t be installed on those servers.

 

There’s so much in WMI that you could build some WQL queries that prevent “dangerous” applications from meeting a Requirement of clients outside its intended deployment.

 

PowerShell Killswitch

SCCM is a pull-based architecture. An implication of this is once the clients have a bad policy they are going to act on it. The first thing you should do if you discover a policy is stomping on your clients is to try and limit the damage by preventing unaffected clients from pulling it. A simple PowerShell script that stops the IIS App Pools backing your Management Points and Distribution Points will act as a crude but effective kill switch. By having this script prepped and ready to go you can immediately stop the spread of something bad and then focus your efforts on correcting the mistake.

Sane Client Settings

There is a tendency to crank up some of the client-side polling frequencies in smaller SCCM implementations in order to make things “go faster” however another way to look at the polling interval is that this is the the period of time it takes for all of your clients to have received a bad policy and possibly acted on it. If your client policy polling interval is 15 minutes that means in 15 minutes you will have re-imaged all your clients if you really screwed up and deployed a Required Task Sequence to All Systems. The longer the polling frequency, the more time you have to identify a bad policy, stop it and begin rebuilding before it has nuked your whole fleet.

Team Processes

A few simple soft processes can go a long way. If you are deploying out an Application or Updates to your whole fleet, send out a notification to your business leaders. People are generally more forgiving of mistakes when they are notified of significant changes first. Perform a gradual roll-out over a week or two instead of blasting out your Office 365 installation application to all 500 workstations at once. Setting sane scheduling and installation deadlines in your Deployments helps here too.

If you are doing something that could be potentially dangerous, grab a coworker and do pilot/co-pilot for the deployment. You (the pilot) perform the work but you walk your coworker (the co-pilot) through each step and have them verify it. Putting a second pair of eyes on a deployment avoids things like inadvertently clicking the “Allow clients to restart outside of Maintenance Windows” checkbox. Next time you need to do this deployment switch roles – Bam! Instant cross training!

Don’t be in a hurry. Nine times out of ten, the dangerous thing is simple to deploy but the simple settings cannot be wrong. Take your time to do things right and push back when you are given unrealistic schedules or asked to deploy things outside of your roll-out process. In the mountains we like to say, slow is fast and fast is dead. In SCCM I like to say, slow is fast, and fast is fired.

Read-Only Friday is the holiest of days on the Sysadmin calendar. Keep it with reverence and respect.

Consider enabling the High Risk Deployment Setting. If you do this make sure you tune the settings so your admins don’t get alert fatigue and just learn to click next, next, finish or eventually they will click next, next, finish and go “oops”.

 

I hope this is helpful. If you have other ideas on how not blow up everything with SCCM feel free to comment. I’m always up for learning something new!

Until next time, stay frosty.

 

 

 

Quick and dirty PowerShell snippet to get Dell Service Tag

The new fiscal year is right around the corner for us. This time of year brings all kinds of fun for us Alaskans, spring king salmon runs, our yearly dose of three days worth of good weather and licensing true-up and hardware purchases. Now there’s about a million different ways to skin this particular cat but here’s a quick a dirty method with PowerShell.

 

If you don’t have access to the ridiculously useful Test-NetConnection cmdlet then you probably should upgrade to PowerShell v5 since unlike most Microsoft products PowerShell seems to actually improve with each version but baring that you can just open a TCP socket by instantiating the appropriate .NET object.

 

The slickest way I have ever seen this done though was with SCCM and the Dell Command Integration Suite for System Center which can generate a warranty status report for everything in your SCCM Site by connecting to database, grabbing the all the service tags and then sending that up to Dell’s warranty status API to get all kinds of juicy information like model, service level, ship date, and warranty status. Unfortunately since this was tremendously useful the team overseeing the warranty status web service decommissioned it abruptly back in 2016. Thanks for nothing ya jerks!

 

SCCM SUP Failing to Download Updates – Invalid Certificate Error

I am currently re-building my System Center lab which includes re-installing and re-configuring a basic SCCM instance. I was in the process of getting my Software Update Point (SUP) setup and a few of my SUP groups failed to download their respective updates and deploy correctly. I dimly remember working through this issue in our production environment a few years ago when we moved from SCCM 2012 to 2012 R2 and I cursed myself for not taking better notes and so here we are atoning for our past sins!

Here’s the offending error:

SCCM SUP Download Error Dialog - Invalid Cert

 

SCCM is a chatty little guy and manages to generate some 160 or so different log files, spread out across a number of different locations (see the highly recommended MSDN blog article, A List of SCCM Log Files). Identifying and locating the relevant logs to whatever issue you are troubleshooting is about half the battle with SCCM. Unfortunately I couldn’t seem to find patchdownloader.log in its expected location of SMS_CCM\Logs. Turns out if you are running the Configuration Manager Console from a RDP session patchdownloader.log will get stored in C:\Users\%USERNAME%\AppData\Local\Temp instead of SMS\Logs. Huh. In my case, I am RDPing to the SCCM Server and running the console but I wonder if I run it from a client workstation whether the resulting log will end up locally on that workstation in my %TEMP% folder or whether it will end up on the SCCM SUP server in SMS_CCM\Logs… an experiment for another day I guess.

 

Here’s the juicy bits:

 

A couple of interesting things to note from here:

  • We get the actual error code (0x80073633) which you can use CMTrace’s Error Lookup functionality to “resolve” back to the human readable one the Console presents you with. Sometimes this turns out to be useful information.
  • We get the download location for the update
  • We get the distribution package location that the update is being downloaded to

If I manually browse to the wsus.ds.download.windowsupdate.com URL I manage to download the update without issues. No certificate validation issues which one would expect considering that the connection is going over HTTP according to the log. Makes one wonder how the resulting error was related to an “invalid certificate”…

OK. How do I fix it? Well like most things SCCM the solution is as stupid as it is brilliant. Manually download the update from the Microsoft Update Catalog. Go find the offending update in its respective Software Update Group by referencing the KB number and download it again but this time set your Download Location to the directory that already contains it.

Whoops. Didn’t work.

Take a look at the first attempt to download the content… SCCM Is looking for Windows10.0-KB3172989-x64.cab so it can be downloaded into my %TEMP% directory and then eventually moved off the Deployment Package’s source location at \\SCCM\Source Files\Windows\Updates\2016.

The file I downloaded is not named Windows10.0-KB3172989-x64.cab – it’s actually an .msu file. Use 7-Zip or similar tool to pull the .cab file out of it and now it SCCM SUP should successfully “download” the the update and ship it off to source location for your Deployment Package.

 

FFFUUUU Internet Explorer… a rant about an outage

I am not normally a proponent of hating on Microsoft, mostly because I think much of the hate they get for design decisions is simply because people do not take the time to understand how Microsoft’s new widget of the month works and why it works that way. I also think it is largely pointless. All Hardware Sucks, All Software Sucks once you really start to dig around under the hood. That and Microsoft doesn’t really give a shit about what you want and why you want it. If you are an enterprise customer they have you by the balls and you and Microsoft both know it. You are just going to have to deal with tiles, the Windows Store and all the other consumer centric bullshit that is coming your way regardless of how “enterprise friendly” your sales rep says Microsoft is.

That being said, I cannot always take my own medicine of enlightened apathy and Stockholm Syndrome and this is one of those times. We had a Windows Update get deployed this week that broke about 60% – 75% of our fleet, specifically Internet Explorer 11. Unfortunately we have a few line-of-business web applications that rely on it. You can imagine how that went.

Now there are a lot of reasons why this happened but midway through my support call where we are piecing together an uninstallation script to remove all the prerequisites of Internet Explorer 11 I had what I call a “boss epiphany”. A “boss epiphany” is when you step out your technical day-to-day and start asking bigger questions and is so named because my boss has a habit of doing this. I generally find it kind of annoying in a good-natured way because I feel like there is a disregard for the technical complexities that I have to deal with in order to make things work but I can’t begrudge that he cuts to the heart of the matter. And six hours into our outage what was the epiphany… “Why is this so fucking hard? We are using Microsoft’s main line-of-business browser (Internet Explorer) and their main line-of-business tool for managing workstations in an enterprise environment (SCCM).”

The answer is complicated from (my) technical perspective but the “boss epiphany” is a really good point. This shit should be easy. It’s not. Or I suck at it. Or maybe both. AND that brings me to my rant. Why in the name of Odin’s beard is software deployment and management in Windows so stupid? All SCCM is doing is really just running an installer. For all its “Enterprisy-ness” it just runs whatever stupid installer you get from Adobe, Microsoft or Oracle. There’s no standardization, no packaging or no guarantee anything will actually be atomic. Even MSI installers can do insane things – like accept arguments in long form (TRANSFROMS=stupidapp.mst) but not short form (/t stupidapp.mst) or my particular favorite, search for ProductKey registry keys to uninstall any older version of the application, and then try to uninstall it via the original .MSI. This fails horribly when that .MSI lives in a non-persistent client side cache (C:\Windows\ccmcache). Linux was created by a bunch of dope-smoking neckbeards and European commies and they have had solid standardized package management for like ten years. I remember taking a Debian Stable install up to Testing, and then down-grading to Stable and then finally just upgrading the whole thing to Unstable. AND EVERYTHING WORKED (MOSTLY). Lets see you try that kind of kernel and user land gymnastics with Windows. Maybe I just have not spent enough supporting Linux to hate it yet but I cannot help but admire the beauty of apt-get update && apt-get upgrade when most of my software deployments means gluing various .EXEs and registry keys together with batch files or PowerShell. It’s 2016 and this is how we are managing software deployments? I feel like I’m taking crazy pills here.

 

Lets look at the IEAK as a specific example since I suspect that’s half the reason I got us into this mess. The quotes from this r/sccm thread are perfect here:

  • “IEAK can’t handle pre reqs cleanly. Also ‘installs’ IE11 and marks it as successful if it fails due to prereqs”
  • “Dittoing this. IEAK was a nightmare.”
  • “IEAK worked fine for us apart from one issue. When installing it would fail to get a return from the WMI installed check of KB2729094 quick enough so it assumed it wasn’t installed and would not complete the IE11 install.”
  • “It turns out that even though the IEAK gave me a setup file it was still reaching out to the Internet to download the main payload for IE”
  • “I will never use IEAK again for an IE11 deployment, mainly for the reason you stated but also the CEIP issue.”

And that’s the supported, “Enterprise” deployment method. If you start digging around on the Internet, you see there are people out there deploying Internet Explorer 11 with Task Sequences, custom batch files, custom PowerShell scripts and the PowerShell Deployment Toolkit. Again, the technical part of me understands that Internet Explorer is a complicated piece of software and that there are reasons it is deployed this way but ultimately if it is easier for me to deploy Firefox with SCCM than Internet Explorer… well that just doesn’t seem right now does it?

Until next time… throw your computer away and go outside. Computers are dumb.