Microsoft Updates on Demand – Part Deux

Now that you found your favorite patching program, we need to send out notifications and target specific servers for patching. I schedule updates as an offset from Patch Tuesday. Patch Tuesday is always the 2nd Tuesday of the month and is when Microsoft releases it’s monthly patches. I currently do a pilot group which is fired the night of Patch Tuesday, and then +4, +11 and +18 days after Patch Tuesday, which is every Saturday after Patch Tuesday.

Send-UpdateNotification2.ps1 is a notification script I use with SCCM. This will send out notifications at 8AM on Friday for a maintenance window that takes place Saturday night from 10PM to 5AM. Due to the way the months work, for +18 days after Patch Tuesday, I send these out on Thursdays instead of Fridays as the script doesn’t work correctly if the date rolls into the next month. You would run this script as a scheduled task once a day at 8AM. Note that this script needs to run from the SCCM server itself.

If you are using SCCM, you can set maintenance windows based on offsets from Patch Tuesday using New-CMMaintenanceWindow.ps1. This is a Powershell script made by Mattias Benninge. As an example: I run this Powershell script once a year to setup all of my server maintenance windows. This starts the maintenance window every Saturday after Patch Tuesday of the month starting at 10PM going to 5AM Sunday.

New-CMMaintenanceWindow.ps1 -SiteCode ABC -MaintenanceWindowName "+4 days after Patch Tuesday weekend" -AddMaintenanceWindowNameMonth -CollectionID "ABC01192" -patchTuesday -adddays 4 -StartYear 2021 -StartHour 22 -StartMinute 0 -HourDuration 7 -MinuteDuration 0 -SWtype Updates

New-CMMaintenanceWindow.ps1 -SiteCode ABC -MaintenanceWindowName "+11 days after Patch Tuesday weekend" -AddMaintenanceWindowNameMonth -CollectionID "ABC01193" -patchTuesday -adddays 11 -StartYear 2021 -StartHour 22 -StartMinute 0 -HourDuration 7 -MinuteDuration 0 -SWtype Updates

New-CMMaintenanceWindow.ps1 -SiteCode ABC -MaintenanceWindowName "+18 days after Patch Tuesday weekend" -AddMaintenanceWindowNameMonth -CollectionID "ABC01194" -patchTuesday -adddays 18 -StartYear 2021 -StartHour 22 -StartMinute 0 -HourDuration 7 -MinuteDuration 0 -SWtype Updates

Dump-Computers.ps1 is used to dump computers from a SCCM collection into a plain text file. Note that this script needs to run from the SCCM server itself. If you use WSUS & WSUS groups, take a look at CreateWSUSGroups.ps1. This script would need to run from the WSUS server itself.

Once we gather the plain text files with the different computers, we can copy them to servers.txt. patchtuesday.ps1 is a modification of Send-UpdateNotification2.ps1. We use this script to copy our specific group of computers to servers.txt based on the offset from Patch Tuesday. I run this every day as a scheduled task at 8AM. On Sundays, I run a scheduled task at 1AM and 3AM that runs a batch file, but I don’t want to run this outside of the maintenance windows above, so I use patchem.bat. The batch file has a “counter file” that increments to 2 when servers.txt and serversplus18.txt are equal. When the counter increments to 2, the batch file will no longer try to patch servers. When the date rolls over to the next month for pilot patching, the counter file is wiped out and the whole process starts over again for that month. The one thing to note is when you run the batch file in the scheduled task, be sure to set the current working directory to the proper folder in the scheduled task, otherwise it will not work properly (batch files are an ancient technology and assume the current working directory for all operations, if your current working directory is not set to where the batch file is running from, weird things will happen).

Some servers just refuse to restart on their own after patching, so we can use force_restart.ps1 to nudge them along. This searches servers.txt for any servers defined in our array and then does a restart on them if it finds them in servers.txt. I run this every Sunday at 5AM. Undoubtedly, you’ll find servers that did not patch because they ran out of disk space. The “famous” servers for doing this are ones that have the IIS role installed since they like to log every web visit. Take a look at clean_iis_logs.ps1 for how you can keep 14 days of logs and ditch the rest.

The final step is to run a report checking patch compliance. All of this hard work of getting the monthly updates via an API has been done for us already: https://sqljana.wordpress.com/2017/08/31/powershell-get-security-updates-list-from-microsoft-by-monthproductkbcve-with-api.

I uploaded this file as Get-SecurityUpdate.ps1 to my GitHub. You can ignore instructions for getting your own API key, as it appears that is not necessary anymore. I set the APIKey to 1 and it still worked, but I went ahead and left the guy’s posted APIKey in the code. Note that you will need to install the MSRCSecurityUpdates Powershell module to use this script.

Towards the end of the function he posted, I added these lines, which finds current month from Get-Date() and feeds it to the Get-SecuriteUpdate function and then dumps it to a plain text file. I added “-Width 300” later on as mysteriously when the exact same script was run on another server, it was truncating the result output at a certain column length, why, I do not know!

$date = Get-Date -Format "yyyy-MMM"
$datestr = $date.ToString()

$rslt = Get-SecurityUpdate -MonthOfInterest $datestr -APIKey '16ce02bef3d9475699b9cc3f0bed1234' -ResultType 'RAW'
$rslt | out-file -Width 300 D:\patching\patches.txt

Now you can run PatchReport.ps1 to parse the results of this months KBs. This will search the patches.txt file looking for CUs and monthly rollups for Windows Server 2012, 2012R2, 2016 and 2019 and then run those results against servers.txt that we generated from the Patch Tuesday Powershell scripts above. I also created PatchReportSMTP.ps1 to send e-mails of the same report to me.

You can see what CUs are available for this month from https://portal.msrc.microsoft.com/en-us/security-guidance.

A sample patch compliance report is shown below. If a row in the column in InstalledBy is empty, that means that server did not restart after patching and is not fully patched (yet).

Any missing patches will show up like this in orange text:

I commented out the error code about “Cannot connect to computer X”, since I run the same script against the same servers.txt file in two different domains without a 2-way trust. About 9 out of 10 times, a failure to patch is a low disk space issue. In a previous blog posting, I talked about how disabling the HTTP.SYS driver blocked patching and the remedy for that, though something like that happening is pretty rare.

  • Soli Deo Gloria

Microsoft Updates on Demand

WSUS and SCCM are great, but maybe you want more control of when Microsoft updates run. There are a few products that can help us with patching: ABC-Update, BatchPatch, WUInstall, and PSWindowsUpdate. I’ve tried all of these and will give my opinion on each.

The first thing we need to do is setup WinRM for every computer we want to update. This is usually pretty easy: just type winrm quickconfig on the computer you want to update. You can also do this through GPO as well:
https://www.techrepublic.com/article/how-to-enable-powershell-remoting-via-group-policy/. This allows remote Powershell access so we can run actions against a list of computers remotely.

The first contender is ABC-Update. I’ve used this one for a while. It comes in a command line and GUI version and is completely free. There is one downside and that is it requires .NET framework to be installed on the computers it patches, which may not be on all of your servers. The command line is relatively easy to understand:

 \\vm-acme-01\netlogon\ABC-Update.exe /S:MSUpdate /C:CriticalUpdates,SecurityUpdates /A:Install
 /R:3 /MailTo:daboss@acme.com /MailFrom:abc-update@acme.com /MailServ:127.0.0.1

We tell ABC-Update to connect to MSUpdate (not WSUS), grab only the Critical and Security Microsoft updates, install them, restart up to 3 times and send us an e-mail when it’s done. Pretty simple. For the GUI version: you can give it a plain text file of computer names or point it at an AD OU and it spawns a process on each computer PSEXEC style that installs a scheduled task that will fire at the time and date of your choosing. You can watch the update status in real time, cancel the updates and re-schedule them as needed. The author is an IT professional and is very responsive and open to feature requests.

Batchpatch requires PSEXEC from Sysinternals to do it’s work. I’ll admit I did not spend much time with Batchpatch. There appears to be no easy way of scheduling updates for a certain time/date or to scope update categories to specific ones. Batchpatch is $399 per year per admin user.

WUinstall is very similar to ABC-Update, however, it only is available as a command line product. The company does have a RMM suite called Xeox that would presumably give you a GUI experience for updating computers if desired. This software is very expensive: to patch 100 clients is $390/year and it goes up from there. There is a 30 day full version trial.

The command line is very similar to ABC-Update:

\\vm-acme-01\netlogon\wuinstall.exe /install /classification update_classification:CS /quiet /autoaccepteula /reboot_if_needed_force /bypass_wsus /rebootcycle 3 /logfile \\vm-logs-01\logs\%computername%.txt

My first trial run of WUInstall ended up with a lot of servers not getting restarted. There are /reboot_if_needed and a /reboot_if_needed_force command line options. Why there is even a difference, I do not know. WUinstall does not require .NET framework to be installed on the servers it patches, so you will be able to patch servers with less software requirements.

PSWindowsUpdate was used at a previous employer, version 1.5, so I decided to give version 2.2 a spin. This is a free Powershell module and has no .NET framework requirement. I did not want to install this Powershell module on all my servers, so I loaded it on my DC’s NETLOGON share and modified $env:PSModulePath to include to UNC path to the PSWindowsUpdate module.

This is an abbreviated snippet from the Powershell code, you can find the files over at my Github. KB890830 is the Windows Malicious Software Removal Tool which is generally useless and wastes patching time, so I exclude it to speed up the patching process. I also had to move the source files from the subfolder \PSWindowsupdate\2.2.02 to just \PSWindowsupdate as it couldn’t find the files when I tried to load the module.

Install-WindowsUpdate -MicrosoftUpdate -Category 'Security Updates', 'Critical Updates' -NotKBArticleID KB890830 -AcceptAll -AutoReboot -Verbose | Out-file dest:\$computername -Force -Append

I use a scheduled task on a server and PSEXEC to kick off the Powershell script to patch servers at 1AM and 3AM after SCCM starts it’s patching at 10PM.

D:\cron\patchadams\psexec.exe -accepteula -d -s @D:\cron\patchadams\servers.txt \\vm-acme-01\netlogon\pspatch.bat

My favorite of all of these patching solutions is PSWindowsUpdate, it’s free and has very little software requirements, followed by ABC-Update.

  • Soli Deo Gloria

Reset the State of Software Center

I recently had to pull Firefox out of Software Center and then made a new Firefox application.  Both the old Firefox and the new Firefox were listed on a particular machine even though I had retired and deleted the old Firefox application.  No matter what I did, the old software persisted!  After some reading: it appears that SCCM tracks Software Center events in WMI.  Even if you remove and reinstall the SCCM client, the “ghost software” remains.  I was able to finally clear off the software icon by doing a complete policy reset using the following WMIC command on the client and then waiting:

WMIC /Namespace:\\root\ccm path SMS_Client CALL ResetPolicy 1 /NOINTERACTIVE

  • Soli Deo Gloria

Dell XPS 13 9350 – The Path to Windows 10

Attempts to do an in-place upgrade on a XPS 13 9350 Windows 8.1 to Windows 10 resulted in lockups around 71%.  The issue appears to be the WiFi driver or more specifically BCM.sys.  If this driver is removed before the Windows 10 upgrade: the upgrade goes flawlessly.

First step is to get a copy of the Windows Development Kit or WDK from Microsoft to obtain the devcon executable.

Next, go into the device manager and get the VEN/DEV id:

Now we remove it!

devcon /r remove "PCI\VEN_8086&DEV_7110"

Now proceed on with the rest of your task sequence.

  • Soli Deo Gloria

A Tale of Two Site Codes

This was an interesting problem.  We are cutting over clients to a new SCCM server with a new site code.  Around 100 clients kept going back to the old site code.  Peeking in LocationServices.log, it kept saying “Group Policy Registration set site code”.  Say what?  We don’t have any GPO like that.

After doing some Googling, I stumbled on this article: https://henkhoogendoorn.blogspot.com/search/label/GPRequestedSiteAssignmentCode and sure enough: GPSiteAssignmentCode was defined!  Someone in the past had made a GPO setting the site code, nuked it, but unfortunately it tattooed the computers forever leaving the old site code.

Solution?  PSEXEC, a list of computers in computers.txt, Notepad++ (to trim trailing spaces) and reg delete:

reg delete HKEY_LOCAL_MACHINE\Software\Microsoft\SMS /f

Then you can do something like psexec @computers.txt -c ccm.bat where ccm.bat holds your ccmsetup command line.

  • Soli Deo Gloria

Bomb Out Task Sequence if Laptop is Not Connected to Ethernet

You would think this would be an easy thing to do in Powershell, but I couldn’t find anything. This WMI code will look for an active Ethernet connection and return errorlevel 0 if it finds an active Ethernet connection and 1 if it does not:

wmic.exe nic where "NetConnectionStatus=2" get NetConnectionID | find "Ethernet"

This has to be put into a batch file and then fired as part of the task sequence.

Why do this?  Well, we want to push Windows 10 through Software Center, however, we don’t want user’s with laptops doing this over the WiFi network.

  • Soli Deo Gloria

 

Update BIOS Using PowerShell and SCCM

https://github.com/adamleinss/PowerShellBIOSUpdate

This is a quick and dirty script for PSADT (http://psappdeploytoolkit.com/) to deploy BIOS updates relating to Intel’s Meltdown/Spectre vulnerability.

PSADT is designed to be used in SCCM deployments, however, it is agnostic enough that it should be able to be used with any software management solution such as PDQ Deploy.

Main drivers in this script:

  • Get-WmiObject Win32_ComputerSystem
  • Get-WmiObject Win32_BIOS

Using a Lenovo M900 as an example:

PS C:\_PUBLIC_REPO> Get-WmiObject Win32_ComputerSystem


Domain              : XXXXXXXXX
Manufacturer        : LENOVO
Model               : 10FM0026US
Name                : XXXXXXXXX
PrimaryOwnerName    : ACME
TotalPhysicalMemory : 8478724096

PS C:\_PUBLIC_REPO> Get-WmiObject Win32_BIOS


SMBIOSBIOSVersion : FWKT86A  
Manufacturer      : LENOVO
Name              : FWKT86A  
SerialNumber      : XXXXXXX
Version           : LENOVO - 1860

Stepping through the code:

$FirmwareUpdateRan = 'FALSE'

Set initial status of $FirmwareUpdateRan to FALSE

$ComputerModel = (Get-WmiObject Win32_ComputerSystem).Model

Set $ComputerModel to 10FM0026US as given for the M900 example above.

$BIOSVersion = (Get-WmiObject Win32_BIOS).Name

Set $BIOSVersion to FWKT86A as given for the M900 example above.

if (($ComputerModel -eq '10FM0026US') -and ($FirmwareUpdateRan -eq 'FALSE') -and ($BIOSVersion -lt 'FWKT86A'))

Once we run one at least one block of firmware update code, $FirmwareUpdateRan will be set to TRUE. Setting this flag will prevent the restart prompt later on if we didn’t run any update code. $BIOSVersion should compared against the version of the BIOS you want to update to. Easiest way of getting this is just running Get-WmiObject Win32_BIOS on the test computer after you run the current BIOS update.

{ $Response = Show-InstallationPrompt -Message 'Executing BIOS update...please close all apps' -ButtonRightText 
'Cancel' -ButtonLeftText 'Continue' -Timeout 600
if ($Response -eq 'Cancel') { exit 12345 }

Show a prompt to end user. The majority of the BIOS updates will force a reboot right away without any warning, thus we display a message to the end user and allow them to cancel it.

New-Item -Path HKLM:SOFTWARE -Name ACMEDesktop -Force
Set-ItemProperty -Path HKLM:SOFTWARE\ACMEDesktop -Name MeltdownFirmwareFix -Value "Yes" -Type String

This is useful for satisfying the detection rule for SCCM. There’s no clean way of determining whether there is a failure of the BIOS update, other than running a compliancy report in your software/hardware inventory reporting tool to make sure the update happened.

set-location $dirfiles\M900

Lenovo’s flash utility doesn’t accept absolute paths: we have to run it from the current directory, so we use set-location to force the location folder.

start-process flash.cmd -ArgumentList '/quiet' -Wait -PassThru

Run the BIOS update

Show-InstallationRestartPrompt -Countdownseconds 600 -CountdownNoHideSeconds 60

This is only shown if the BIOS update didn’t force a reboot. Currently, I only found the T460S and Yoga S1 laptops do not force a reboot. Since reboot isn’t forced, we force one with a 10 minute countdown.

Suspend-BitLocker -MountPoint C: -RebootCount 1 -Confirm:$false

Suspends BitLocker for one reboot, otherwise laptop will go into recovery mode. Note this command is supported for Windows 8 and later only. For Windows 7 you will need to use manage-bde: Manage-bde.exe –protectors –disable c:. I didn’t see any -rc option, so you will need to do something such as a scheduled task to turn it back on.

  • Soli Deo Gloria

SCCM PKI Fun with Certificates

This was fun problem to sort out.  I was asked to jump in and fix a SCCM server already built to work with PKI.  Attempts to get clients registered with the server would end up with bizarre error messages like this:

RegTask: Client is not registered. Sending registration request
RegTask: Reply for registration was empty. Error: 0x8000ffff

I worked on the problem for about 8 hours at work, then went home and setup PKI in my home SCCM lab in about 30 minutes.  I decided I needed to enlist Microsoft PSS on this issue.

After working with Microsoft for about 2.5 hours: they tracked the problem to the certificate on the management point bound to IIS being “too new”.  Essentially, SCCM has legacy code in it that only understands certificates based on the CSP templates (Windows XP/Server 2003) and not KSP/CNG templates (Windows Server 2008 and later).

This is explained in more detail here:

https://www.sevecek.com/EnglishPages/Lists/Posts/Post.aspx?ID=66

The funny part is I was actually using a CA template I had found on the production distribution points that were already up and working, but I guess using the wrong certificate template on DPs doesn’t matter, but using the wrong one on the MP does matter for client registration at least!

I had no access to the CA server, so I couldn’t snoop around on the properties of said certificate templates and they were named “2012 or later IIS”.  Of course the management server is running Windows Server 2012 R2, so why wouldn’t I pick that template?

In the end, you have to use the command line to see the cryptographic provider of the certificates (this doesn’t show up in the GUI):

certutil -repairstore my *

It seems that other people are annoyed by this and according to Microsoft the ability to use CNG or more “modern” certificate templates is coming in a newer build of SCCM:

https://configurationmanager.uservoice.com/forums/300492-ideas/suggestions/17451757-support-v3-and-newer-certificate-templates-for-htt

  • Soli Deo Gloria

0x80004005: An error occurred while retrieving policy for this computer

Started a new job recently and one of the techs was having a problem imaging a laptop with a recently replaced motherboard.  We would PXE boot the laptop and then WinPE would bomb out in 10 seconds stating “0x80004005: An error occurred while retrieving policy for this computer”.

Nothing interesting was found on the SCCM side, however when I finally found the SMSTS.LOG in X:\Windows\temp\SMSTSLOG, I didn’t even have to open the file to figure out the problem: it was dated 2016!  Yup: it was a date and time issue.  If your computer skews too far from the current date and time, SCCM won’t talk to your computer.

You can use the commands date and time within cmd (hit the F8 key…you did enable this functionality, right?) to set the correct date and time.

  • Soli Deo Gloria