Now that you found your favorite patching program, we need to send out notifications and target specific servers for patching. I schedule updates as an offset from Patch Tuesday. Patch Tuesday is always the 2nd Tuesday of the month and is when Microsoft releases it’s monthly patches. I currently do a pilot group which is fired the night of Patch Tuesday, and then +4, +11 and +18 days after Patch Tuesday, which is every Saturday after Patch Tuesday.
Send-UpdateNotification2.ps1 is a notification script I use with SCCM. This will send out notifications at 8AM on Friday for a maintenance window that takes place Saturday night from 10PM to 5AM. Due to the way the months work, for +18 days after Patch Tuesday, I send these out on Thursdays instead of Fridays as the script doesn’t work correctly if the date rolls into the next month. You would run this script as a scheduled task once a day at 8AM. Note that this script needs to run from the SCCM server itself.
If you are using SCCM, you can set maintenance windows based on offsets from Patch Tuesday using New-CMMaintenanceWindow.ps1. This is a Powershell script made by Mattias Benninge. As an example: I run this Powershell script once a year to setup all of my server maintenance windows. This starts the maintenance window every Saturday after Patch Tuesday of the month starting at 10PM going to 5AM Sunday.
New-CMMaintenanceWindow.ps1 -SiteCode ABC -MaintenanceWindowName "+4 days after Patch Tuesday weekend" -AddMaintenanceWindowNameMonth -CollectionID "ABC01192" -patchTuesday -adddays 4 -StartYear 2021 -StartHour 22 -StartMinute 0 -HourDuration 7 -MinuteDuration 0 -SWtype Updates New-CMMaintenanceWindow.ps1 -SiteCode ABC -MaintenanceWindowName "+11 days after Patch Tuesday weekend" -AddMaintenanceWindowNameMonth -CollectionID "ABC01193" -patchTuesday -adddays 11 -StartYear 2021 -StartHour 22 -StartMinute 0 -HourDuration 7 -MinuteDuration 0 -SWtype Updates New-CMMaintenanceWindow.ps1 -SiteCode ABC -MaintenanceWindowName "+18 days after Patch Tuesday weekend" -AddMaintenanceWindowNameMonth -CollectionID "ABC01194" -patchTuesday -adddays 18 -StartYear 2021 -StartHour 22 -StartMinute 0 -HourDuration 7 -MinuteDuration 0 -SWtype Updates
Dump-Computers.ps1 is used to dump computers from a SCCM collection into a plain text file. Note that this script needs to run from the SCCM server itself. If you use WSUS & WSUS groups, take a look at CreateWSUSGroups.ps1. This script would need to run from the WSUS server itself.
Once we gather the plain text files with the different computers, we can copy them to servers.txt. patchtuesday.ps1 is a modification of Send-UpdateNotification2.ps1. We use this script to copy our specific group of computers to servers.txt based on the offset from Patch Tuesday. I run this every day as a scheduled task at 8AM. On Sundays, I run a scheduled task at 1AM and 3AM that runs a batch file, but I don’t want to run this outside of the maintenance windows above, so I use patchem.bat. The batch file has a “counter file” that increments to 2 when servers.txt and serversplus18.txt are equal. When the counter increments to 2, the batch file will no longer try to patch servers. When the date rolls over to the next month for pilot patching, the counter file is wiped out and the whole process starts over again for that month. The one thing to note is when you run the batch file in the scheduled task, be sure to set the current working directory to the proper folder in the scheduled task, otherwise it will not work properly (batch files are an ancient technology and assume the current working directory for all operations, if your current working directory is not set to where the batch file is running from, weird things will happen).
Some servers just refuse to restart on their own after patching, so we can use restart_sniffer.ps1 to nudge them along. This searches servers.txt and then does a restart on a server if it set PendingReboot to True. This script uses Get-PendingRebootStatus.ps1 from TheSystemAdminChannel website. I run this every Sunday at 5AM. Undoubtedly, you’ll find servers that did not patch because they ran out of disk space. The “famous” servers for doing this are ones that have the IIS role installed since they like to log every web visit. Take a look at clean_iis_logs.ps1 for how you can keep 14 days of logs and ditch the rest.
The final step is to run a report checking patch compliance. All of this hard work of getting the monthly updates via an API has been done for us already: https://sqljana.wordpress.com/2017/08/31/powershell-get-security-updates-list-from-microsoft-by-monthproductkbcve-with-api. As an update: the script on this site stopped working in July 2021, so I’m using this file instead: https://github.com/meta-l/MSSecurityUpdates/blob/master/get_updates.ps1
I uploaded this file as Get-SecurityUpdate2.ps1 to my GitHub. You can ignore instructions for getting your own API key, as it appears that is not necessary anymore. I set the APIKey to 1 and it still worked, but I went ahead and left the guy’s posted APIKey in the code. Note that you will need to install the MSRCSecurityUpdates Powershell module to use this script.
Now you can run PatchReport.ps1 to parse the results of this months KBs. This will search the patches.txt file looking for CUs and monthly rollups for Windows Server 2012, 2012R2, 2016 and 2019 and then run those results against servers.txt that we generated from the Patch Tuesday Powershell scripts above. I also created PatchReportSMTP.ps1 to send e-mails of the same report to me.
You can see what CUs are available for this month from https://portal.msrc.microsoft.com/en-us/security-guidance.
A sample patch compliance report is shown below. If a row in the column in InstalledBy is empty, that means that server did not restart after patching and is not fully patched (yet).
Any missing patches will show up like this in orange text:
I commented out the error code about “Cannot connect to computer X”, since I run the same script against the same servers.txt file in two different domains without a 2-way trust. About 9 out of 10 times, a failure to patch is a low disk space issue. In a previous blog posting, I talked about how disabling the HTTP.SYS driver blocked patching and the remedy for that, though something like that happening is pretty rare.
- Soli Deo Gloria