It’s been a while since I’ve done one of these “Case of” blog posts. Back in my desktop engineering days, you could find users doing all sorts of wacky stuff on their computers and those stories of how I found the problem and fixed it made for some interesting posts. Now that am a sysadmin working mainly on servers, it’s only me and a few IT people making changes on the servers.
Another sysadmin asked me to look at why users couldn’t connect to any printers on a particular print server. We have 10 of these print servers, all pretty much identically configured running Windows Server 2019. I won’t bore you with the many hours I spent restarting the server, installing a new printer and sharing it out, fiddling with random registry settings, running Procmon and just trying other random off the wall stuff. The one weird thing was I could install printers from the troubled server under my login, but not of that of a regular user. Yes, I did make that regular user a local administrator: it made no difference.
I surmised that someone in the IT department had been messing with the server, so I used Beyond Compare’s registry comparison feature to compare the problem server with a print server that was working properly.
The problem server is on the left, a working server on the right. You can see that the server-role for print services was missing.
I went into Server Manager and sure enough: the print services role was missing on the troubled server. I thought I had hit the jackpot after re-adding the role, but alas I still could not add a printer from the printer server using a regular user account.
I started doing a “stare and compare” between the two servers and then I noticed something interesting. On the working server, the “View Server” permission for our staff group was checked; on the not working server, it was unchecked.
Upon checking that box, I could now add a printer from the troubled print server without any problems.
I couldn’t stop there, so I had to look up what the View Server permissions were and this is what Microsoft says:
View Server
The View Server permission assigns the ability to view the print server. Without the View Server permission, users cannot see the printers that are managed by the server. By default, this permission is given to members of the Everyone group.
Despite what Microsoft says, I could, as a regular user, go to \\bad_print_server\ and see all of the printers shared on the server, however, I couldn’t install any of them. It seems View Server really means Install Printers from server.
This should have been easy, but rarely anything in technology is. I listen to a podcast that is just a recording of what is broadcast on AM radio. The problem is if the producer of that show is on vacation or the station is having technical difficulties, it can take days for the podcast to be uploaded.
The first huddle is figuring out what the streaming URL is. This is well hidden, because the parent company probably wants you to go to their website to listen to the stream. You can do this right from a Chrome browser, open the Internet radio stream, then hit the F12 key, go to the Networking tab and looking for anything with m3u8, playlist or stream in the name:
In the above example (which is just a random radio station I picked), the URL is http://ample.revma.ihrhls.com/zc4245. If you copy and paste that into a browser, it starts playing the radio station.
There’s an easier way of getting this information and that’s from this website: https://www.radio-browser.info/. If you type in the station call letters “WISN” in the search, you get a URL of https://stream.revma.ihrhls.com/zc4245, which is very close to what we found before.
Now comes the fun part of figuring out how to record this stream to an audio file. This can be done by using the task scheduler to start up web browser and open our streaming web site address, such as "C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe" "https://stream.revma.ihrhls.com/zc4245". Just throw that into a batch file to make it easier to run.
Then you can use fmedia with the loopback option to record the audio. First, figure out what device you want to record from by running this command:
fmedia --list-dev
You want a Playback/Loopback device that you usually hear things out of, probably your default audio device.
This records the Internet stream for 30 seconds and then saves a MP3 file with a low bit rate of 64kbps, since this is talk radio. If you wanted 3 hours and 30 minutes use --until 3:30. You’ll need to terminate msedge.exe after 3:30 hours, you can do this with a program called pskill. Download the Sysinternals Suite and then put this into a scheduled task after your show ends: pskill -t msedge.exe
This works, but is a bit clunky. Could we just record the Internet audio stream directly? Yes, there’s several options, starting with a GUI one: StreamWriter. This one works great, you just have to turn off the feature for splitting the stream into individual songs (Uncheck save separated tracksin the options). I just have one warning for this solution: this program only outputs the same steam up to 11 audio files with the same name, then it stops writing. It appends a 1, 2, etc. until it gets to 10 and then it stops creating new audio files. The fix is relatively easy with some Powershell code (you would run this some time after your stream ends and recording stops):
This renames all files ending in .AAC to the date the file was created on, such as 09.03.2022.aac and then removes any files ending in .AAC older than 2 days to keep things tidy. I recommend using a batch file to call the Powershell script, it makes it a lot easier to deal with in the task scheduler. In your batch file, just call it like this
ffmpeg can also record Internet radio streams if you prefer a command line option (accepts HH:mm:ss for time notation, I just did 30 seconds here as an example):
VLC supports recording Internet audio streams as well, though the command line is a bit tricky to figure out, but this seems to work (note the stop time is in seconds, so convert the time you want accordingly into the number of seconds):
To remove commercials from the audio file, I like using mp3DirectCut. You can mark the beginning of the commercials with the B key and then the end with the N key to mark that section and then hit the DELETE key and poof, that section of the MP3 file is gone. No re-encoding of the file required. When done, do a File>Save as and save it as a new file.
To get the audio file over to your Android phone, I like using SolidExplorer’s FTP server option and then using WinSCP to FTP the file over to a folder on the phone. For listening: I like mABook. The skip ahead and rewind is user defined, I set mine to 1 minute and 5 minute intervals, so if you don’t want to remove commercials and just want to try skipping them in the playback app, you can usually get good results fast forwarding in either small or large chunks (long chunks usually being around the hour break).
Admin Assistant is a handy piece of software for sysadmins. It used to be known as AutoAdministator back in 2014. Recently, I needed the ability to do a mass shutdown of servers, so I went searching for this tool again. It sits somewhere between PDQ Deploy and psexec in terms of functionality.
Here’s everything that it can do:
What I love about this program is that it’s free: free of ads and free of “Let me get your e-mail address before you download this so I can spam you“. You can just download it and go. It does exactly what it says it does and it does it very well.
Windows 11 released to the world last week and as I predicted: the TPM, Secure Boot and CPU requirements can all be bypassed. Depending on how you are trying to install Windows 11, you have several options. If you are trying to do an in-place upgrade from within Windows 10 itself, you can do a Google search for AllowUpgradesWithUnsupportedTPMOrCPU. Creating this registry key will cause the Windows 11 setup program to ignore the CPU check and will allow you to proceed with TPM 1.2, however, you still need a TPM chip.
To bypass all requirements, you need to run the Windows 11 install from a bootable USB stick. Copy the following into notepad and save as bypass.reg to the USB stick:
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\Setup\LabConfig]
"BypassTPMCheck"=dword:00000001
"BypassSecureBootCheck"=dword:00000001
"BypassRAMCheck"=dword:00000001
"BypassStorageCheck"=dword:00000001
"BypassCPUCheck"=dword:00000001
Boot to the Windows 11 setup using the USB stick. During the setup, you will get an error your PC is not supported. Click back to the main screen. At this point, you can hit SHIFT-F10 to get to a CMD prompt, type regedit and then go to File>Import and import bypass.reg above. You can now proceed installing Windows 11.
Techpowerup did a really nice write-up here on the process:
Rufus now has a beta version that will create a bootable ISO with all of these restrictions removed called “Windows 11 Extended Support”: https://github.com/pbatard/rufus/releases/. Note that you can only do clean installs using the bootable USB stick method and the upgrade option does not work from the bootable media.
I watched the Windows 11 live stream from Microsoft today. There was a few interesting tidbits, such as it will be free upgrade for Windows 10 users (not unexpected), will only come in a 64-bit version (32-bit version is gone) and requires UEFI/Secureboot/TPM 2.0. A lot of people are hyperventilating over the TPM 2.0 requirement, but we are roughly 6 months away from the release of Windows 11 and I have no doubt that this requirement will either be relaxed or a workaround will be found.
Installing the leaked Windows 11 dev build was blocked by Microsoft from installing on bare metal and within days, people figured out how to copy all of the files from the Windows 10 sources folder from the install media and then just overwrite the Windows 10 WIM file with the Windows 11 WIM file and viola, all restrictions were removed. The TPM 2.0 requirement was already bypassed for this build by replacing appraiserres.dll with one from Windows 10. Heck, WinNTSetup will probably have a tick box to just remove the requirement.
The 32-bit version should have been removed a long time ago. If you have something that requires Windows 32-bit (for the 16-bit subsystem support), you should probably be running an emulator for that program or leave it on an older OS. The last time I had to install Windows 32-bit was back in June 2014 for a company we bought from Baldor Generators. There was 1 guy that had to run a MS-DOS program and MS-DOS programs only work on Windows 32-bit (without emulation), so we had to revert his Windows 64-bit back to Windows 32-bit.
However, if I had to do it over again, I would just run winevdm or DOSBOX to run his MS-DOS program and leave him on Windows 64-bit. There’s even more options such as VMWare Workstation, I think it’s finally time to drop MS-DOS support for programs that were created 40 years ago.
Now that you found your favorite patching program, we need to send out notifications and target specific servers for patching. I schedule updates as an offset from Patch Tuesday. Patch Tuesday is always the 2nd Tuesday of the month and is when Microsoft releases it’s monthly patches. I currently do a pilot group which is fired the night of Patch Tuesday, and then +4, +11 and +18 days after Patch Tuesday, which is every Saturday after Patch Tuesday.
Send-UpdateNotification2.ps1 is a notification script I use with SCCM. This will send out notifications at 8AM on Friday for a maintenance window that takes place Saturday night from 10PM to 5AM. Due to the way the months work, for +18 days after Patch Tuesday, I send these out on Thursdays instead of Fridays as the script doesn’t work correctly if the date rolls into the next month. You would run this script as a scheduled task once a day at 8AM. Note that this script needs to run from the SCCM server itself.
If you are using SCCM, you can set maintenance windows based on offsets from Patch Tuesday using New-CMMaintenanceWindow.ps1. This is a Powershell script made by Mattias Benninge. As an example: I run this Powershell script once a year to setup all of my server maintenance windows. This starts the maintenance window every Saturday after Patch Tuesday of the month starting at 10PM going to 5AM Sunday.
Dump-Computers.ps1 is used to dump computers from a SCCM collection into a plain text file. Note that this script needs to run from the SCCM server itself. If you use WSUS & WSUS groups, take a look at CreateWSUSGroups.ps1. This script would need to run from the WSUS server itself.
Once we gather the plain text files with the different computers, we can copy them to servers.txt. patchtuesday.ps1 is a modification of Send-UpdateNotification2.ps1. We use this script to copy our specific group of computers to servers.txt based on the offset from Patch Tuesday. I run this every day as a scheduled task at 8AM. On Sundays, I run a scheduled task at 1AM and 3AM that runs a batch file, but I don’t want to run this outside of the maintenance windows above, so I use patchem.bat. The batch file has a “counter file” that increments to 2 when servers.txt and serversplus18.txt are equal. When the counter increments to 2, the batch file will no longer try to patch servers. When the date rolls over to the next month for pilot patching, the counter file is wiped out and the whole process starts over again for that month. The one thing to note is when you run the batch file in the scheduled task, be sure to set the current working directory to the proper folder in the scheduled task, otherwise it will not work properly (batch files are an ancient technology and assume the current working directory for all operations, if your current working directory is not set to where the batch file is running from, weird things will happen).
Some servers just refuse to restart on their own after patching, so we can use restart_sniffer.ps1 to nudge them along. This searches servers.txt and then does a restart on a server if PendingReboot is set to True. This script uses Get-PendingRebootStatus.ps1 from TheSystemAdminChannel website. I run this every Sunday at 5AM. Undoubtedly, you’ll find servers that did not patch because they ran out of disk space. The “famous” servers for doing this are ones that have the IIS role installed since they like to log every web visit. Take a look at clean_iis_logs.ps1 for how you can keep 14 days of logs and ditch the rest.
The final step is to run a report checking patch compliance. All of this hard work of getting the monthly updates via an API has been done for us already.
As an update to this part: these 2 scripts would work for a few months, then stop working and after looking at the code, I discovered I could get what I needed with just 5 lines of code. You can check that out as Get-SecurityUpdateAL.ps1 on my GitHub site. – 6/23/22.
You can ignore instructions for getting your own API key, as it appears that is not necessary anymore. Note that you will need to install the MSRCSecurityUpdates Powershell module to use this script.
Now you can run PatchReport.ps1 to parse the results of this month’s KBs. This will search the patches.txt file looking for CUs and monthly rollups for Windows Server 2012, 2012R2, 2016, 2019, 2022 and then run those results against servers.txt that we generated from the Patch Tuesday Powershell scripts above. I also created PatchReportSMTP.ps1 to send e-mails of the same report to me.
A sample patch compliance report is shown below. If a row in the column in InstalledBy is empty, that means that server did not restart after patching and is not fully patched (yet).
Any missing patches will show up like this in orange text:
I commented out the error code about “Cannot connect to computer X”, since I run the same script against the same servers.txt file in two different domains. About 9 out of 10 times, a failure to patch is a low disk space issue. In a previous blog posting, I talked about how disabling the HTTP.SYS driver blocked patching and the remedy for that, though something like that happening is pretty rare.
WSUS and SCCM are great, but maybe you want more control of when Microsoft updates run. There are a few products that can help us with patching: ABC-Update, BatchPatch, WUInstall, and PSWindowsUpdate. I’ve tried all of these and will give my opinion on each.
The first thing we need to do is setup WinRM for every computer we want to update. This is usually pretty easy: just type winrm quickconfig on the computer you want to update. You can also do this through GPO as well: https://www.techrepublic.com/article/how-to-enable-powershell-remoting-via-group-policy/. This allows remote Powershell access so we can run actions against a list of computers remotely.
The first contender is ABC-Update. I’ve used this one for a while. It comes in a command line and GUI version and is completely free. There is one downside and that is it requires .NET framework to be installed on the computers it patches, which may not be on all of your servers. The command line is relatively easy to understand:
We tell ABC-Update to connect to MSUpdate (not WSUS), grab only the Critical and Security Microsoft updates, install them, restart up to 3 times and send us an e-mail when it’s done. Pretty simple. For the GUI version: you can give it a plain text file of computer names or point it at an AD OU and it spawns a process on each computer PSEXEC style that installs a scheduled task that will fire at the time and date of your choosing. You can watch the update status in real time, cancel the updates and re-schedule them as needed. The author is an IT professional and is very responsive and open to feature requests.
Batchpatch requires PSEXEC from Sysinternals to do it’s work. I’ll admit I did not spend much time with Batchpatch. There appears to be no easy way of scheduling updates for a certain time/date or to scope update categories to specific ones. Batchpatch is $399 per year per admin user.
WUinstall is very similar to ABC-Update, however, it only is available as a command line product. The company does have a RMM suite called Xeox that would presumably give you a GUI experience for updating computers if desired. This software is very expensive: to patch 100 clients is $390/year and it goes up from there. There is a 30 day full version trial.
My first trial run of WUInstall ended up with a lot of servers not getting restarted. There are /reboot_if_needed and a /reboot_if_needed_force command line options. Why there is even a difference, I do not know. WUinstall does not require .NET framework to be installed on the servers it patches, so you will be able to patch servers with less software requirements.
PSWindowsUpdate was used at a previous employer, version 1.5, so I decided to give version 2.2 a spin. This is a free Powershell module and has no .NET framework requirement. I did not want to install this Powershell module on all my servers, so I loaded it on my DC’s NETLOGON share and modified $env:PSModulePath to include to UNC path to the PSWindowsUpdate module.
This is an abbreviated snippet from the Powershell code, you can find the files over at my Github. KB890830 is the Windows Malicious Software Removal Tool which is generally useless and wastes patching time, so I exclude it to speed up the patching process. I also had to move the source files from the subfolder \PSWindowsupdate\2.2.02 to just \PSWindowsupdate as it couldn’t find the files when I tried to load the module.
I use a scheduled task on a server and PSEXEC to kick off the Powershell script to patch servers at 1AM and 3AM after SCCM starts it’s patching at 10PM.
These errors are always fun to track down. Last month, I had a Windows Server 2016 VM that would always roll back the latest cumulative update for the OS. I tried the usual tricks of running sfc /scannow, dism, safe mode, etc. and nothing worked. I gave up and moved on to something else. Now again this month I had even more servers doing the exact same thing.
After some Googling, I did a deep dive into the logs under C:\windows\logs\cbs\ and looked for “, error” (comma, then a space, then the word error) around the time I tried applying the update and then found this:
Error [0x01803e] CSI 000000f5 (F) Failed execution of queue item Installer: HTTP Installer ({86fee01a-954a-11df-bc0c-cea7dfd72085}) with HRESULT HRESULT_FROM_WIN32(1058). Failure will not be ignored: A rollback will be initiated after all the operations in the installer queue are completed; installer is reliable[gle=0x80004005]
This did ring a bell as months earlier I had found a dev box that would not patch and it ended up that http.sys was disabled on that VM as well (but it was enabled on the prod box???). The common thread between all 3 VMs was that they were all running Apache Tomcat. The web developer had disabled the http.sys driver as this can “hijack” programs from listening on port 80, so to prevent this they just disable it, which also causes PowerShell remote management, printer spooler and branchcache services not to run (because they all rely on the http.sys driver running).
The fix is simple: enable the http.sys driver, run the patches, then disable http.sys:
sc qc http
sc config http start= auto
After patching we can run these commands to set it back to disabled and restart again:
