It’s been a while since I’ve done one of these “Case of” blog posts. Back in my desktop engineering days, you could find users doing all sorts of wacky stuff on their computers and those stories of how I found the problem and fixed it made for some interesting posts. Now that am a sysadmin working mainly on servers, it’s only me and a few IT people making changes on the servers.
Another sysadmin asked me to look at why users couldn’t connect to any printers on a particular print server. We have 10 of these print servers, all pretty much identically configured running Windows Server 2019. I won’t bore you with the many hours I spent restarting the server, installing a new printer and sharing it out, fiddling with random registry settings, running Procmon and just trying other random off the wall stuff. The one weird thing was I could install printers from the troubled server under my login, but not of that of a regular user. Yes, I did make that regular user a local administrator: it made no difference.
I surmised that someone in the IT department had been messing with the server, so I used Beyond Compare’s registry comparison feature to compare the problem server with a print server that was working properly.
The problem server is on the left, a working server on the right. You can see that the server-role for print services was missing.
I went into Server Manager and sure enough: the print services role was missing on the troubled server. I thought I had hit the jackpot after re-adding the role, but alas I still could not add a printer from the printer server using a regular user account.
I started doing a “stare and compare” between the two servers and then I noticed something interesting. On the working server, the “View Server” permission for our staff group was checked; on the not working server, it was unchecked.
Upon checking that box, I could now add a printer from the troubled print server without any problems.
I couldn’t stop there, so I had to look up what the View Server permissions were and this is what Microsoft says:
View Server
The View Server permission assigns the ability to view the print server. Without the View Server permission, users cannot see the printers that are managed by the server. By default, this permission is given to members of the Everyone group.
Despite what Microsoft says, I could, as a regular user, go to \\bad_print_server\ and see all of the printers shared on the server, however, I couldn’t install any of them. It seems View Server really means Install Printers from server.
Admin Assistant is a handy piece of software for sysadmins. It used to be known as AutoAdministator back in 2014. Recently, I needed the ability to do a mass shutdown of servers, so I went searching for this tool again. It sits somewhere between PDQ Deploy and psexec in terms of functionality.
Here’s everything that it can do:
What I love about this program is that it’s free: free of ads and free of “Let me get your e-mail address before you download this so I can spam you“. You can just download it and go. It does exactly what it says it does and it does it very well.
Windows 11 released to the world last week and as I predicted: the TPM, Secure Boot and CPU requirements can all be bypassed. Depending on how you are trying to install Windows 11, you have several options. If you are trying to do an in-place upgrade from within Windows 10 itself, you can do a Google search for AllowUpgradesWithUnsupportedTPMOrCPU. Creating this registry key will cause the Windows 11 setup program to ignore the CPU check and will allow you to proceed with TPM 1.2, however, you still need a TPM chip.
To bypass all requirements, you need to run the Windows 11 install from a bootable USB stick. Copy the following into notepad and save as bypass.reg to the USB stick:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\Setup\LabConfig]
"BypassTPMCheck"=dword:00000001
"BypassSecureBootCheck"=dword:00000001
"BypassRAMCheck"=dword:00000001
"BypassStorageCheck"=dword:00000001
"BypassCPUCheck"=dword:00000001
Boot to the Windows 11 setup using the USB stick. During the setup, you will get an error your PC is not supported. Click back to the main screen. At this point, you can hit SHIFT-F10 to get to a CMD prompt, type regedit and then go to File>Import and import bypass.reg above. You can now proceed installing Windows 11.
Techpowerup did a really nice write-up here on the process:
Rufus now has a beta version that will create a bootable ISO with all of these restrictions removed called “Windows 11 Extended Support”: https://github.com/pbatard/rufus/releases/. Note that you can only do clean installs using the bootable USB stick method and the upgrade option does not work from the bootable media.
Now that you found your favorite patching program, we need to send out notifications and target specific servers for patching. I schedule updates as an offset from Patch Tuesday. Patch Tuesday is always the 2nd Tuesday of the month and is when Microsoft releases it’s monthly patches. I currently do a pilot group which is fired the night of Patch Tuesday, and then +4, +11 and +18 days after Patch Tuesday, which is every Saturday after Patch Tuesday.
Send-UpdateNotification2.ps1 is a notification script I use with SCCM. This will send out notifications at 8AM on Friday for a maintenance window that takes place Saturday night from 10PM to 5AM. Due to the way the months work, for +18 days after Patch Tuesday, I send these out on Thursdays instead of Fridays as the script doesn’t work correctly if the date rolls into the next month. You would run this script as a scheduled task once a day at 8AM. Note that this script needs to run from the SCCM server itself.
If you are using SCCM, you can set maintenance windows based on offsets from Patch Tuesday using New-CMMaintenanceWindow.ps1. This is a Powershell script made by Mattias Benninge. As an example: I run this Powershell script once a year to setup all of my server maintenance windows. This starts the maintenance window every Saturday after Patch Tuesday of the month starting at 10PM going to 5AM Sunday.
Dump-Computers.ps1 is used to dump computers from a SCCM collection into a plain text file. Note that this script needs to run from the SCCM server itself. If you use WSUS & WSUS groups, take a look at CreateWSUSGroups.ps1. This script would need to run from the WSUS server itself.
Once we gather the plain text files with the different computers, we can copy them to servers.txt. patchtuesday.ps1 is a modification of Send-UpdateNotification2.ps1. We use this script to copy our specific group of computers to servers.txt based on the offset from Patch Tuesday. I run this every day as a scheduled task at 8AM. On Sundays, I run a scheduled task at 1AM and 3AM that runs a batch file, but I don’t want to run this outside of the maintenance windows above, so I use patchem.bat. The batch file has a “counter file” that increments to 2 when servers.txt and serversplus18.txt are equal. When the counter increments to 2, the batch file will no longer try to patch servers. When the date rolls over to the next month for pilot patching, the counter file is wiped out and the whole process starts over again for that month. The one thing to note is when you run the batch file in the scheduled task, be sure to set the current working directory to the proper folder in the scheduled task, otherwise it will not work properly (batch files are an ancient technology and assume the current working directory for all operations, if your current working directory is not set to where the batch file is running from, weird things will happen).
Some servers just refuse to restart on their own after patching, so we can use restart_sniffer.ps1 to nudge them along. This searches servers.txt and then does a restart on a server if PendingReboot is set to True. This script uses Get-PendingRebootStatus.ps1 from TheSystemAdminChannel website. I run this every Sunday at 5AM. Undoubtedly, you’ll find servers that did not patch because they ran out of disk space. The “famous” servers for doing this are ones that have the IIS role installed since they like to log every web visit. Take a look at clean_iis_logs.ps1 for how you can keep 14 days of logs and ditch the rest.
The final step is to run a report checking patch compliance. All of this hard work of getting the monthly updates via an API has been done for us already.
As an update to this part: these 2 scripts would work for a few months, then stop working and after looking at the code, I discovered I could get what I needed with just 5 lines of code. You can check that out as Get-SecurityUpdateAL.ps1 on my GitHub site. – 6/23/22.
You can ignore instructions for getting your own API key, as it appears that is not necessary anymore. Note that you will need to install the MSRCSecurityUpdates Powershell module to use this script.
Now you can run PatchReport.ps1 to parse the results of this month’s KBs. This will search the patches.txt file looking for CUs and monthly rollups for Windows Server 2012, 2012R2, 2016, 2019, 2022 and then run those results against servers.txt that we generated from the Patch Tuesday Powershell scripts above. I also created PatchReportSMTP.ps1 to send e-mails of the same report to me.
A sample patch compliance report is shown below. If a row in the column in InstalledBy is empty, that means that server did not restart after patching and is not fully patched (yet).
Any missing patches will show up like this in orange text:
I commented out the error code about “Cannot connect to computer X”, since I run the same script against the same servers.txt file in two different domains. About 9 out of 10 times, a failure to patch is a low disk space issue. In a previous blog posting, I talked about how disabling the HTTP.SYS driver blocked patching and the remedy for that, though something like that happening is pretty rare.
WSUS and SCCM are great, but maybe you want more control of when Microsoft updates run. There are a few products that can help us with patching: ABC-Update, BatchPatch, WUInstall, and PSWindowsUpdate. I’ve tried all of these and will give my opinion on each.
The first thing we need to do is setup WinRM for every computer we want to update. This is usually pretty easy: just type winrm quickconfig on the computer you want to update. You can also do this through GPO as well: https://www.techrepublic.com/article/how-to-enable-powershell-remoting-via-group-policy/. This allows remote Powershell access so we can run actions against a list of computers remotely.
The first contender is ABC-Update. I’ve used this one for a while. It comes in a command line and GUI version and is completely free. There is one downside and that is it requires .NET framework to be installed on the computers it patches, which may not be on all of your servers. The command line is relatively easy to understand:
We tell ABC-Update to connect to MSUpdate (not WSUS), grab only the Critical and Security Microsoft updates, install them, restart up to 3 times and send us an e-mail when it’s done. Pretty simple. For the GUI version: you can give it a plain text file of computer names or point it at an AD OU and it spawns a process on each computer PSEXEC style that installs a scheduled task that will fire at the time and date of your choosing. You can watch the update status in real time, cancel the updates and re-schedule them as needed. The author is an IT professional and is very responsive and open to feature requests.
Batchpatch requires PSEXEC from Sysinternals to do it’s work. I’ll admit I did not spend much time with Batchpatch. There appears to be no easy way of scheduling updates for a certain time/date or to scope update categories to specific ones. Batchpatch is $399 per year per admin user.
WUinstall is very similar to ABC-Update, however, it only is available as a command line product. The company does have a RMM suite called Xeox that would presumably give you a GUI experience for updating computers if desired. This software is very expensive: to patch 100 clients is $390/year and it goes up from there. There is a 30 day full version trial.
My first trial run of WUInstall ended up with a lot of servers not getting restarted. There are /reboot_if_needed and a /reboot_if_needed_force command line options. Why there is even a difference, I do not know. WUinstall does not require .NET framework to be installed on the servers it patches, so you will be able to patch servers with less software requirements.
PSWindowsUpdate was used at a previous employer, version 1.5, so I decided to give version 2.2 a spin. This is a free Powershell module and has no .NET framework requirement. I did not want to install this Powershell module on all my servers, so I loaded it on my DC’s NETLOGON share and modified $env:PSModulePath to include to UNC path to the PSWindowsUpdate module.
This is an abbreviated snippet from the Powershell code, you can find the files over at my Github. KB890830 is the Windows Malicious Software Removal Tool which is generally useless and wastes patching time, so I exclude it to speed up the patching process. I also had to move the source files from the subfolder \PSWindowsupdate\2.2.02 to just \PSWindowsupdate as it couldn’t find the files when I tried to load the module.
I use a scheduled task on a server and PSEXEC to kick off the Powershell script to patch servers at 1AM and 3AM after SCCM starts it’s patching at 10PM.
These errors are always fun to track down. Last month, I had a Windows Server 2016 VM that would always roll back the latest cumulative update for the OS. I tried the usual tricks of running sfc /scannow, dism, safe mode, etc. and nothing worked. I gave up and moved on to something else. Now again this month I had even more servers doing the exact same thing.
After some Googling, I did a deep dive into the logs under C:\windows\logs\cbs\ and looked for “, error” (comma, then a space, then the word error) around the time I tried applying the update and then found this:
Error [0x01803e] CSI 000000f5 (F) Failed execution of queue item Installer: HTTP Installer ({86fee01a-954a-11df-bc0c-cea7dfd72085}) with HRESULT HRESULT_FROM_WIN32(1058). Failure will not be ignored: A rollback will be initiated after all the operations in the installer queue are completed; installer is reliable[gle=0x80004005]
This did ring a bell as months earlier I had found a dev box that would not patch and it ended up that http.sys was disabled on that VM as well (but it was enabled on the prod box???). The common thread between all 3 VMs was that they were all running Apache Tomcat. The web developer had disabled the http.sys driver as this can “hijack” programs from listening on port 80, so to prevent this they just disable it, which also causes PowerShell remote management, printer spooler and branchcache services not to run (because they all rely on the http.sys driver running).
The fix is simple: enable the http.sys driver, run the patches, then disable http.sys:
sc qc http
sc config http start= auto
After patching we can run these commands to set it back to disabled and restart again:
I was transferring department folders and files from one server to another. I used Robocopy and FreeFileSync to get the job done. Just for my own future reference, this was the Robocopy command I used:
I fired this off once, then used FreeFileSync to sync any delta changes afterward. This is pretty boring and common, server administrators have been performing this ritual for many years. However, I’ve been in desktop engineering for the last 16 years and only 2 years in server administration, so it was “new to me”.
Weeks later, after I was done, I got an e-mail about invalid shortcuts. In order to save disk space, the users created shortcuts to other folders. These shortcuts of course included the name of the old server. Off to Google we go! One solution I found was this, but didn’t end up using it (our AV software kept deleting it for some reason) http://jacquelin.potier.free.fr/ShortcutsSearchAndReplace/.
This works great, it even has a “dryrun” option so you can see what it’s going to do before it actually does anything. The only thing to watch out for is if the shortcut was already invalid to begin with. About 1/3 of the shortcuts the user had were already invalid on the old server due to them shifting and renaming various folders and not updating the shortcuts.
I was asked to try to suppress this error message popping up on our kiosk display monitors. When Windows 10 detects that a non-optimal display resolution is being used, it offers the end-user some help. I get why Microsoft is doing this: it is in their best interest to try to help the end-user fix problems on their own whenever possible. However, in this case, it’s a kiosk computer with no keyboard or mouse. No one will ever be able to answer the prompt and due to the size of the display, it’s always going to use a non-optimal display resolution. The prompt doesn’t offer an option “Never ask again”. Off to Google, we go!
Well, that wasn’t so easy, but I’ll get to the punchline and tell you the solution.
That command prevents the “fix the blurry” prompt from ever rearing it’s ugly head again. Now the fun part is…where is that documented? It’s nowhere documented at Microsoft’s site. In fact, go ahead and Google IgnorePerProcessSystemDPIToast. You will find very little information on this word. At the time of this posting, that was exactly 7 hits on Google. 7…out of billions of web pages!
One of the more intriguing results was this web site with a bunch of undocumented hacks for Windows 10.
