Posted on

Microsoft Updates on Demand – Part Deux

Now that you found your favorite patching program, we need to send out notifications and target specific servers for patching. I schedule updates as an offset from Patch Tuesday. Patch Tuesday is always the 2nd Tuesday of the month and is when Microsoft releases it’s monthly patches. I currently do a pilot group which is fired the night of Patch Tuesday, and then +4, +11 and +18 days after Patch Tuesday, which is every Saturday after Patch Tuesday.

Send-UpdateNotification2.ps1 is a notification script I use with SCCM. This will send out notifications at 8AM on Friday for a maintenance window that takes place Saturday night from 10PM to 5AM. Due to the way the months work, for +18 days after Patch Tuesday, I send these out on Thursdays instead of Fridays as the script doesn’t work correctly if the date rolls into the next month. You would run this script as a scheduled task once a day at 8AM. Note that this script needs to run from the SCCM server itself.

If you are using SCCM, you can set maintenance windows based on offsets from Patch Tuesday using New-CMMaintenanceWindow.ps1. This is a Powershell script made by Mattias Benninge. As an example: I run this Powershell script once a year to setup all of my server maintenance windows. This starts the maintenance window every Saturday after Patch Tuesday of the month starting at 10PM going to 5AM Sunday.

New-CMMaintenanceWindow.ps1 -SiteCode ABC -MaintenanceWindowName "+4 days after Patch Tuesday weekend" -AddMaintenanceWindowNameMonth -CollectionID "ABC01192" -patchTuesday -adddays 4 -StartYear 2021 -StartHour 22 -StartMinute 0 -HourDuration 7 -MinuteDuration 0 -SWtype Updates

New-CMMaintenanceWindow.ps1 -SiteCode ABC -MaintenanceWindowName "+11 days after Patch Tuesday weekend" -AddMaintenanceWindowNameMonth -CollectionID "ABC01193" -patchTuesday -adddays 11 -StartYear 2021 -StartHour 22 -StartMinute 0 -HourDuration 7 -MinuteDuration 0 -SWtype Updates

New-CMMaintenanceWindow.ps1 -SiteCode ABC -MaintenanceWindowName "+18 days after Patch Tuesday weekend" -AddMaintenanceWindowNameMonth -CollectionID "ABC01194" -patchTuesday -adddays 18 -StartYear 2021 -StartHour 22 -StartMinute 0 -HourDuration 7 -MinuteDuration 0 -SWtype Updates

Dump-Computers.ps1 is used to dump computers from a SCCM collection into a plain text file. Note that this script needs to run from the SCCM server itself. If you use WSUS & WSUS groups, take a look at CreateWSUSGroups.ps1. This script would need to run from the WSUS server itself.

Once we gather the plain text files with the different computers, we can copy them to servers.txt. patchtuesday.ps1 is a modification of Send-UpdateNotification2.ps1. We use this script to copy our specific group of computers to servers.txt based on the offset from Patch Tuesday. I run this every day as a scheduled task at 8AM. On Sundays, I run a scheduled task at 1AM and 3AM that runs a batch file, but I don’t want to run this outside of the maintenance windows above, so I use patchem.bat. The batch file has a “counter file” that increments to 2 when servers.txt and serversplus18.txt are equal. When the counter increments to 2, the batch file will no longer try to patch servers. When the date rolls over to the next month for pilot patching, the counter file is wiped out and the whole process starts over again for that month. The one thing to note is when you run the batch file in the scheduled task, be sure to set the current working directory to the proper folder in the scheduled task, otherwise it will not work properly (batch files are an ancient technology and assume the current working directory for all operations, if your current working directory is not set to where the batch file is running from, weird things will happen).

Some servers just refuse to restart on their own after patching, so we can use restart_sniffer.ps1 to nudge them along. This searches servers.txt and then does a restart on a server if it set PendingReboot to True. This script uses Get-PendingRebootStatus.ps1 from TheSystemAdminChannel website. I run this every Sunday at 5AM. Undoubtedly, you’ll find servers that did not patch because they ran out of disk space. The “famous” servers for doing this are ones that have the IIS role installed since they like to log every web visit. Take a look at clean_iis_logs.ps1 for how you can keep 14 days of logs and ditch the rest.

The final step is to run a report checking patch compliance. All of this hard work of getting the monthly updates via an API has been done for us already: https://sqljana.wordpress.com/2017/08/31/powershell-get-security-updates-list-from-microsoft-by-monthproductkbcve-with-api. As an update: the script on this site stopped working in July 2021, so I’m using this file instead: https://github.com/meta-l/MSSecurityUpdates/blob/master/get_updates.ps1

I uploaded this file as Get-SecurityUpdate2.ps1 to my GitHub. You can ignore instructions for getting your own API key, as it appears that is not necessary anymore. I set the APIKey to 1 and it still worked, but I went ahead and left the guy’s posted APIKey in the code. Note that you will need to install the MSRCSecurityUpdates Powershell module to use this script.

Now you can run PatchReport.ps1 to parse the results of this months KBs. This will search the patches.txt file looking for CUs and monthly rollups for Windows Server 2012, 2012R2, 2016 and 2019 and then run those results against servers.txt that we generated from the Patch Tuesday Powershell scripts above. I also created PatchReportSMTP.ps1 to send e-mails of the same report to me.

You can see what CUs are available for this month from https://portal.msrc.microsoft.com/en-us/security-guidance.

A sample patch compliance report is shown below. If a row in the column in InstalledBy is empty, that means that server did not restart after patching and is not fully patched (yet).

Any missing patches will show up like this in orange text:

I commented out the error code about “Cannot connect to computer X”, since I run the same script against the same servers.txt file in two different domains without a 2-way trust. About 9 out of 10 times, a failure to patch is a low disk space issue. In a previous blog posting, I talked about how disabling the HTTP.SYS driver blocked patching and the remedy for that, though something like that happening is pretty rare.

  • Soli Deo Gloria

Posted on

Microsoft Updates on Demand

WSUS and SCCM are great, but maybe you want more control of when Microsoft updates run. There are a few products that can help us with patching: ABC-Update, BatchPatch, WUInstall, and PSWindowsUpdate. I’ve tried all of these and will give my opinion on each.

The first thing we need to do is setup WinRM for every computer we want to update. This is usually pretty easy: just type winrm quickconfig on the computer you want to update. You can also do this through GPO as well:
https://www.techrepublic.com/article/how-to-enable-powershell-remoting-via-group-policy/. This allows remote Powershell access so we can run actions against a list of computers remotely.

The first contender is ABC-Update. I’ve used this one for a while. It comes in a command line and GUI version and is completely free. There is one downside and that is it requires .NET framework to be installed on the computers it patches, which may not be on all of your servers. The command line is relatively easy to understand: 

 \\vm-acme-01\netlogon\ABC-Update.exe /S:MSUpdate /C:CriticalUpdates,SecurityUpdates /A:Install
 /R:3 /MailTo:daboss@acme.com /MailFrom:abc-update@acme.com /MailServ:127.0.0.1

We tell ABC-Update to connect to MSUpdate (not WSUS), grab only the Critical and Security Microsoft updates, install them, restart up to 3 times and send us an e-mail when it’s done. Pretty simple. For the GUI version: you can give it a plain text file of computer names or point it at an AD OU and it spawns a process on each computer PSEXEC style that installs a scheduled task that will fire at the time and date of your choosing. You can watch the update status in real time, cancel the updates and re-schedule them as needed. The author is an IT professional and is very responsive and open to feature requests.

Batchpatch requires PSEXEC from Sysinternals to do it’s work. I’ll admit I did not spend much time with Batchpatch. There appears to be no easy way of scheduling updates for a certain time/date or to scope update categories to specific ones. Batchpatch is $399 per year per admin user.

WUinstall is very similar to ABC-Update, however, it only is available as a command line product. The company does have a RMM suite called Xeox that would presumably give you a GUI experience for updating computers if desired. This software is very expensive: to patch 100 clients is $390/year and it goes up from there. There is a 30 day full version trial.

The command line is very similar to ABC-Update:

\\vm-acme-01\netlogon\wuinstall.exe /install /classification update_classification:CS /quiet /autoaccepteula /reboot_if_needed_force /bypass_wsus /rebootcycle 3 /logfile \\vm-logs-01\logs\%computername%.txt

My first trial run of WUInstall ended up with a lot of servers not getting restarted. There are /reboot_if_needed and a /reboot_if_needed_force command line options. Why there is even a difference, I do not know. WUinstall does not require .NET framework to be installed on the servers it patches, so you will be able to patch servers with less software requirements.

PSWindowsUpdate was used at a previous employer, version 1.5, so I decided to give version 2.2 a spin. This is a free Powershell module and has no .NET framework requirement. I did not want to install this Powershell module on all my servers, so I loaded it on my DC’s NETLOGON share and modified $env:PSModulePath to include to UNC path to the PSWindowsUpdate module.

This is an abbreviated snippet from the Powershell code, you can find the files over at my Github. KB890830 is the Windows Malicious Software Removal Tool which is generally useless and wastes patching time, so I exclude it to speed up the patching process. I also had to move the source files from the subfolder \PSWindowsupdate\2.2.02 to just \PSWindowsupdate as it couldn’t find the files when I tried to load the module.

Install-WindowsUpdate -MicrosoftUpdate -Category 'Security Updates', 'Critical Updates' -NotKBArticleID KB890830 -AcceptAll -AutoReboot -Verbose | Out-file dest:\$computername -Force -Append

I use a scheduled task on a server and PSEXEC to kick off the Powershell script to patch servers at 1AM and 3AM after SCCM starts it’s patching at 10PM.

D:\cron\patchadams\psexec.exe -accepteula -d -s @D:\cron\patchadams\servers.txt \\vm-acme-01\netlogon\pspatch.bat

My favorite of all of these patching solutions is PSWindowsUpdate, it’s free and has very little software requirements, followed by ABC-Update.

  • Soli Deo Gloria
Posted on

Failed to Apply Cumulative Update on Server 2016

These errors are always fun to track down. Last month, I had a Windows Server 2016 VM that would always roll back the latest cumulative update for the OS. I tried the usual tricks of running sfc /scannow, dism, safe mode, etc. and nothing worked. I gave up and moved on to something else. Now again this month I had even more servers doing the exact same thing.

After some Googling, I did a deep dive into the logs under C:\windows\logs\cbs\ and looked for the word “error” around the time I tried applying the update and then found this:

Error [0x01803e] CSI 000000f5 (F) Failed execution of queue item Installer: HTTP Installer ({86fee01a-954a-11df-bc0c-cea7dfd72085}) with HRESULT HRESULT_FROM_WIN32(1058). Failure will not be ignored: A rollback will be initiated after all the operations in the installer queue are completed; installer is reliable[gle=0x80004005]

This did ring a bell as months earlier I had found a dev box that would not patch and it ended up that http.sys was disabled on that VM as well (but it was enabled on the prod box???). The common thread between all 3 VMs was that they were all running Apache Tomcat. The web developer had disabled the http.sys driver as this can “hijack” programs from listening on port 80, so to prevent this they just disable it, which also causes PowerShell remote management, printer spooler and branchcache services not to run (because they all rely on the http.sys driver running).

The fix is simple: enable the http.sys driver, run the patches, then disable http.sys:

sc qc http
sc config http start= auto

After patching we can run these commands to set it back to disabled and restart again:

net stop http /y
sc config http start= disabled
sc qc http

Let me just rant and say Microsoft does a very poor job of telling us why the patches failed to apply. They don’t even give a hint of where to go look for the log files.

  • Soli Deo Gloria
Posted on

Bulk Change LNK (Shortcut) Files

I was transferring department folders and files from one server to another. I used Robocopy and FreeFileSync to get the job done. Just for my own future reference, this was the Robocopy command I used:

robocopy \\oldserver\d$\Data\SHARED \\newserver\SHAREDV11 /MIR /COPY:DATSO /DCOPY:DAT /R:10 /W:30 /MT:24 /LOG:"D:\transfer\SHAREDV11.log" /TEE

I fired this off once, then used FreeFileSync to sync any delta changes afterward. This is pretty boring and common, server administrators have been performing this ritual for many years. However, I’ve been in desktop engineering for the last 16 years and only 2 years in server administration, so it was “new to me”.

Weeks later, after I was done, I got an e-mail about invalid shortcuts. In order to save disk space, the users created shortcuts to other folders. These shortcuts of course included the name of the old server. Off to Google we go! One solution I found was this, but didn’t end up using it (our AV software kept deleting it for some reason) http://jacquelin.potier.free.fr/ShortcutsSearchAndReplace/.

Ultimately, I used this code from poster Terrance @ https://superuser.com/questions/495491/change-shortcut-targets-in-bulk

$oldPrefix = "\\OldServer\Archive\"
$newPrefix = "\\NewServer\Archive\"

$searchPath = "Z:\"

$dryRun = $TRUE

$shell = new-object -com wscript.shell

if ( $dryRun ) {
   write-host "Executing dry run" -foregroundcolor green -backgroundcolor black
} else {
   write-host "Executing real run" -foregroundcolor red -backgroundcolor black
}

dir $searchPath -filter *.lnk -recurse | foreach {
   $lnk = $shell.createShortcut( $_.fullname )
   $oldPath= $lnk.targetPath

   $lnkRegex = "^" + [regex]::escape( $oldPrefix ) 

   if ( $oldPath -match $lnkRegex ) {
      $newPath = $oldPath -replace $lnkRegex, $newPrefix

      write-host "Found: " + $_.fullname -foregroundcolor yellow -backgroundcolor black
      write-host " Replace: " + $oldPath
      write-host " With:    " + $newPath

      if ( !$dryRun ) {
         $lnk.targetPath = $newPath
         $lnk.Save()
      }
   }
}

This works great, it even has a “dryrun” option so you can see what it’s going to do before it actually does anything. The only thing to watch out for is if the shortcut was already invalid to begin with. About 1/3 of the shortcuts the user had were already invalid on the old server due to them shifting and renaming various folders and not updating the shortcuts.

  • Soli Deo Gloria
Posted on

Disabling the “Fix Apps That are Blurry” Prompt in Windows 10

I was asked to try to suppress this error message popping up on our kiosk display monitors. When Windows 10 detects that a non-optimal display resolution is being used, it offers the end-user some help. I get why Microsoft is doing this: it is in their best interest to try to help the end-user fix problems on their own whenever possible. However, in this case, it’s a kiosk computer with no keyboard or mouse. No one will ever be able to answer the prompt and due to the size of the display, it’s always going to use a non-optimal display resolution. The prompt doesn’t offer an option “Never ask again”. Off to Google, we go!

Well, that wasn’t so easy, but I’ll get to the punchline and tell you the solution.

reg add "HKCU\Control Panel\Desktop" /v IgnorePerProcessSystemDPIToast /t REG_DWORD /d 1 /f

That command prevents the “fix the blurry” prompt from ever rearing it’s ugly head again. Now the fun part is…where is that documented? It’s nowhere documented at Microsoft’s site. In fact, go ahead and Google IgnorePerProcessSystemDPIToast. You will find very little information on this word. At the time of this posting, that was exactly 7 hits on Google. 7…out of billions of web pages!

One of the more intriguing results was this web site with a bunch of undocumented hacks for Windows 10.

  • Soli Deo Gloria

Posted on

A Very Powerful Freeware File Manager

I recently stumbled upon Altap Salamander. At work, there is a folder with 13000+ sub-folders (don’t ask) I have to work with periodically. Using the built-in Windows file explorer won’t work due to the desktop.ini problem where all you see is a bunch of My Documents folders and not a list of username named folders. I frequently use XYplorer for file operations, however, it would lock-up if I went too deep into the folder structure. At that point, I would usually copy and paste the direct folder path into the native Windows file explorer while RDPed into a server to work around the lock-up issue.

The other issue with the native Windows file explorer is the refreshing view glitch. If Windows file explorer detects any change in the folder list it redraws the folder display view and re-enumerates the folder list starting at folder 1. What a pain!

This is where Altap Salamander comes in to help us. It appears it was a piece of freeware that was created in 1997, then it converted into commercial software and just recently was transitioned back to freeware as of July 2019. You can navigate to any UNC path by going to Commands>Change Directory. This file manager correctly displays and handles folders with large numbers of sub-folders. The software has interesting features including being able to calculate folder sizes,
batch file renamer, advanced file search (including filters on size and date), file type filter view and directory comparison just to name a few. I was able to go to another computer and run it from a remote UNC path, so the program appears to be fully portable and does not require admin installation.

This file manager also lets you see hidden folders that Windows file explorer will hide from you, as I explained in this 2006 blog post and yes, everything I wrote then still holds true today 13 years later (well, kind of, C:\documents and settings is now C:\users, but you get the point)

  • Soli Deo Gloria
Posted on

System Center Orchestrator /Operations Manager 2016 & TLS 1.2

I deleted and re-wrote this blog post since the other one was out of date.  The backstory to this post starts back in July 2018 when we disabled TLS 1.0 across the whole organization.  Much fun ensued and we were running around trying to update the SQL native client and other software to get TLS 1.2 compliant.  One of the things that stopped working were the integration packs that we installed on System Center Orchestrator (SCORCH).  

I opened a premier support case with Microsoft and the first solution was to re-enable FIPS.  However, we later found out that enabling FIPS is really re-enabling TLS 1.0.  We left the registry entries that disabled TLS 1.0 and the ones that enabled FIPS.  The case was left open for Microsoft to figure out how to get TLS 1.2 working. The November 2018 security patches came out from Microsoft and the integration packs on SCORCH broke, AGAIN!  Another premier case was opened with Microsoft. 

After 2 hours of running Procmon and Wireshark, a solution was found.

Here are the steps you need to take:

On the SCORCH server:

1. Remove the SCOM console (If Any).
2. Uninstall SM and OM integration packs from control panel.
3. Reboot the server.
4. Un-deploy the existing SCOM IP from the designer/runbook servers.
5. Unregister the IP. 

6. Download and install the 
System Center 1801+ – Orchestrator Integration Packs (yes, even for SCORCH 2016!)
7.Re-register the IP and redeploy it (NOTE: Before re-registering the IPs again, make sure that the “Microsoft.EnterpriseManagement.Core.dll” and “Microsoft.EnterpriseManagement.OperationsManager.dll” are no longer present in either “c:\windows\assembly” or “c:\windows\microsoft.net\assembly\gac_msil”).

8. Re-install SCOM console (be sure to apply latest UR)

Add these registry entries to each SCOM server: 

reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v2.0.50727" /v "SchUseStrongCrypto" /t REG_DWORD /d 00000001 /f

reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v2.0.50727" /v "SchUseStrongCrypto" /t REG_DWORD /d 00000001 /f

Add these to each SCORCH server (management/runbook):

reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v2.0.50727" /v "SystemDefaultTlsVersions" /t REG_DWORD /d 00000001 /f

reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v2.0.50727" /v "SystemDefaultTlsVersions" /t REG_DWORD /d 00000001 /f

reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319" /v "SystemDefaultTlsVersions" /t REG_DWORD /d 00000001 /f

reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319" /v "SystemDefaultTlsVersions" /t REG_DWORD /d 00000001 /f

Enjoy TLS 1.2 with SCORCH/SCOM.  Note you can check for TLS 1.2 communication by loading Wireshark on the SCORCH management server, then go into the IP pack for SCOM and do a test connection and look for TLS 1.2 in the log.

  • Soli Deo Gloria
Posted on

Bypassing Choose Privacy Settings Screen – Windows 10

This was a fun one to track down.  When we did in-place upgrades on Windows 8.1 to Windows 10 1703, we never got this “Choose privacy settings for your device” screen.  However, going from 8.1 to 1803, this screen will appear once for the first user who logs in with local administrative rights (even though we define certain privacy settings through GPO):

Trying to track this down was hard, but I was inspired by this post on Reddit.

The first stab I tried was logging in as a regular user, running ProcMon and then trying to filter on the registry write operations, but even then, it was too much noise (60K+ events).  I then tried another approach.  When you click the Accept button, there’s a UAC prompt that comes up with a title of “User settings: OOBE”.  I made note of the word “OOBE” and cancelled it making changes. I ran Process Explorer as admin logged in as a regular user, then switched over to logging in as an administrator until I got the privacy screen, switched back to the regular user and then did a search for “OOBE” in the process list.  One of the processes that came up was svchost.exe and it had the following key open:

HKLM\Software\Microsoft\Windows\CurrentVersion\OOBE\Stats

I drilled around in this parent key and found this setting:

HKLM\Software\Microsoft\Windows\CurrentVersion\OOBE\PrivacyConsentStatus

Ah ha!  It was set to REG_DWORD 2, so I set it to 1 and tried logging in again as an administrator.  No prompt to set privacy settings!  I deleted the whole PrivacyConsentStatus key and the prompt still did not show up.  I went back and set PrivacyConsentStatus to 2, logged off and back on, privacy settings page showed back up, I clicked the Accept button on the privacy page and then went back to this registry key to see the results.  PrivacyConsentStatus was set back to 1 and a new entry called PrivacyConsentSID was created with a REG_SZ value with my user account SID.  I deleted PrivacyConsentSID and it seemed to have no effect on the system.

The fix is simple: copy the following into a REG file and then fire it towards the end of OSD

Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\OOBE]
"PrivacyConsentStatus"=dword:00000001
  • Soli Deo Gloria
Posted on

Get Any Edition of Windows 10 Without Access to VLSC

This is a neat little trick I found on the Internet.  If you don’t have access to VLSC and still need to get access to the Enterprise or Education editions of Windows 10, you can use the Media Creation Tool to download them.

Run the following.

MediaCreationTool1803.exe /Eula Accept /Retail /MediaArch x64 /MediaEdition Enterprise

<insert valid Win 10 KMS>.  You can find generic KMS keys here: https://docs.microsoft.com/en-us/windows-server/get-started/kmsclientkeys

Now you can extract the image you want out of the ESD file as a WIM file.  Number 3 is currently the Enterprise SKU:

dism /Get-WimInfo /WimFile:install.esd

dism /export-image /SourceImageFile:install.esd /SourceIndex:3 
/DestinationImageFile:install.wim /Compress:max /CheckIntegrity

Remove pid.txt under sources and check licensing status of machine with the following command after installing the OS:

slmgr /dli

– Soli Deo Gloria