When RunAs SYSTEM Isn’t Enough

I’ve written several times in the past about running under the SYSTEM account using the well known trick psexec -i -s cmd which runs cmd under SYSTEM, but what if you want to run as TrustedInstaller? More accurately, you can run as SYSTEM with the TrustedInstaller token. I happened to stumble across this trick when trying to delete the files in C:\Windows\WinSxS\Temp\PendingDeletes\ and it just wasn’t happening using the SYSTEM account alone. The program I used is SuperCMD. Just run SuperCMD.exe /TI /Run:cmd.exe using RunAs Administrator on cmd.exe and viola, you are SYSTEM running with the TrustedInstaller token!

Another program that can be used is NSUDO which is based on SuperCMD.

  • Soli Deo Gloria

Edgesuite.net Access Denied Error When Visiting Web sites

Buckle up, this is going to be an interesting one. A few weeks ago, I started getting errors like this visiting web sites like McDonalds, Costco, Meijer, Pick ‘n Save:

Access Denied
You don’t have permission to access “http://www.costco.com/?” on this server.

Reference #18.1371ca17.1714195696.247b9298

https://errors.edgesuite.net/18.1371ca17.1714195696.247b9298

Odd. I started looking around on the Internet which lead me to this thread on the Verizon FIOS site: https://community.verizon.com/t5/Fios-Internet-and-High-Speed/multiple-websites-quot-access-denied-quot-over-verizon-Fios/td-p/1746618

Reading through the thread, I found a post by smith6612 pointing to this link: https://www.akamai.com/us/en/clientrep-lookup/. I went and checked my IP address: it was listed as a web scrapper. These companies use Akamai as a CDN (Content Distribution Network) and according to the store IT support, my IP address was listed as a 10/10 or a very bad IP address in Akamai.

If I used a VPN, I could then access these sites just fine, because I was using a different IP address, but this is a not good long term solution. I attempted to contact Akamai to remove my IP address, but in so many terms, they told me to go pound sand as I was not their customer.

After contacting Spectrum technical support, they suggested a get a new cable modem which should give me a new IP address. I did so and….same IP address. ARGH! Anyways, I was already prepared with a plan B: it appears that if you change the MAC access of your router slightly, even by 1 character, Spectrum will see this as a new router and will give you a different IP address.

I changed the MAC address on the router and restarted the router and the cable modem: I was granted a new IP address! This new IP address was not blacklisted by Akamai.

The option to change a MAC address will vary by router model, but the TP-LINK AX5400 lets you change the MAC address in the router’s web GUI. The other option suggested by Spectrum was to leave the cable modem turned off for 24 hours (for the DHCP lease to expire) and then plug it back in, but that option may not be viable for many people unless you have some type of backup Internet option like a hotspot.

Changing my router’s MAC address and getting a new IP address worked for 24 hours before I was banned again. Not sure why, I removed some extensions from Chrome and tried a new IP address.

It appears after 4 days of removing several Chrome extensions I wasn’t actively using, it’s finally resolved. I suspect it may have to due to 2 Chrome extensions I had loaded in the past: one called “DownloadThemAll!” and another called “Video Download Helper”. I had used DownloadThemAll! to download some audio files from a website (freely available on said website) several months ago. I didn’t want to click 60+ individual links to download them, I wanted to click once and do it batch mode style. My theory is one of the websites I visit was using Akamai’s CDN service saw these extensions loaded in my browser and just assumed I was a bad actor, even though I wasn’t actively using the tools on their website and added me to a naughty list, which was then shared to the rest of Akamai’s customers and thus I was blocked from many websites.

If you have any other helpful tips or interesting stories on this topic, feel free to leave those as comments and they will be approved depending on their quality.

-Soli Deo Gloria

VMWare Workstation Free for Personal Use

A nice gesture from Broadcom: https://blogs.vmware.com/cloud-foundation/2024/05/14/vmware-desktop-hypervisor-pro-apps-now-available-for-personal-use/

Unfortunately, my account is stuck in the verification phase, but someone found the direct download link: https://softwareupdate.vmware.com/cds/vmw-desktop/ws/17.5.2/23775571/. It’s a TAR file, so you will need something like WinRAR to extract it.

If I’m being completely honest: HyperV and Windows Sandbox built into Windows is very good and covers all my virtualization needs

  • Soli Deo Gloria

In Search of the Perfect Android Podcast App

Recently, I subscribed to Club Twit (twit.tv), which includes ad free videos for their tech shows.  I loaded these shows up into Pocket Casts without any problems.  I, however, wanted to watch these on my computer full screen.  I blissfully logged into Pocket Casts on the web to see if I could play my podcasts there, then I was greeted with this message: that will be $40/year.  Are you kidding me?  They have a “deal” where the first year is $20, then year 2 and beyond is $40.

Podcast apps are just aggregators of RSS feeds and there’s no way I’m paying $40/year for the ability to watch videos hosted by a 3rd party.  Researching around, I found Bluestacks which can run Android apps on Windows and indeed, I could load Pocket Casts and have it go full screen without any problems and it would sync my progress of watching the show to my phone.  They even created an icon for Pocket Casts on my desktop, so all I had to do was click on the icon and it would auto-launch Bluestacks and Pocket Casts.

However, this $40 cash grab attempt left a bad taste in my mouth with Pocket Casts. In addition to that, I have issues with their playlist feature.  I can add 3 episodes of a podcast to the queue, but it never plays the last item in the playlist automatically.  So annoying when you are in the shower and not near the phone!

I decided to hunt around for a new podcast app.  Podcast Addict was highly recommended on Reddit, so I loaded it up.  It offered to restore my backup from the year 2011: yes, I had used this app in the past.  Apparently, it saves the app data in a secret backup location on your Google Drive that you can’t see.  Unfortunately, Podcast Addict doesn’t appear to sync between multiple devices, so I uninstalled it.

I then tried Podurama which looked very promising.  They offer syncing between devices and their free tier allows playing of your podcasts on their website.  Unfortunately, I could not get the syncing to work between my computer and phone, so I uninstalled that one as well.

I then stumbled onto Podcast Republic.  While they do not offer playing podcasts from a website, they do offer syncing between devices and right from the get go, they say it’s free.  I loaded it up into Bluestacks and I was able to sync my podcast progress to my phone.  Hooray!  The sync, however, does not seem to be “real-time”, but I was able to do it on demand from Settings>Account & syncing>Sync now.  The app has ads which were not obtrusive at all, just a small banner at the very bottom of the app.   I went ahead and paid the one time fee of $4 to remove the ads.

As an added bonus: I see they support streaming radio stations.  I listen to a radio station on iHeartRadio using their official app.  Unfortunately, they started putting ads on the screen in their app: annoying!   I went to radio-browser.info, found the streaming URL for the station I listen to and added it in Podcast Republic.  No more screen ads!

-Soli Deo Gloria

Reimaged Computers Can’t Register Their DNS record

This one took me a while to solve. The desktop guys kept coming to me stating when they re-imaged a computer, it either didn’t ping or it had the wrong IP address. I found out later they had changed their imaging methodology. Before re-imaging any computer, they first delete the computer account and then re-image it. I would guess that this netjoin hardening change is the reason.

When I went into DNS management, I could clearly see an “Account unknown” in the ACL of the DNS record, which makes sense, because the computer account registered the DNS record, but now that computer account didn’t exist anymore. Until the DNS record is scavenged or deleted manually, the newly imaged computer will be unable to update its own DNS record.

This led me down a path of many dead ends. I wrote a script to compare DHCP leases to DNS records. However, I soon found out that DHCP is not always correct either for the current IP address. If someone moves from location to location, the last DHCP lease is the one you want to use. I then looked into making DHCP the owner and updater of all dynamic DNS records, but this too caused issues such as duplicate DNS records.

I then looked at trying to find any DNS records with “Account unknown” in the ACL, but the script ended up too complex and just didn’t work. It was back to basics: I only cared about recently deleted computer accounts, so why not just look for recently deleted computer accounts and then delete the DNS records for those accounts?

That’s exactly what dns_orphan_fix.ps1 does. It looks back 60 minutes for any deleted computer accounts and then attempts to delete the DNS records for those accounts. I run this in the task scheduler every 30 minutes, so that does mean that DNS records will get deleted twice, but I shouldn’t miss any deleted computer accounts this way. There is a “$dryrun” option that you can flip to $true just to make sure this script will operate the way you think it will operate in your environment before setting it to $false to actually delete DNS records.

  • Soli Deo Gloria

Adding .NET Framework 3.5 – Error Code 0x800f0954

Here we go again: another server, another error. Why can’t things just work properly? Had a consultant e-mail me they couldn’t load .NET Framework 3.5 on Windows Server 2019. “Easy peasy lemon squeezy” I thought. Well, of course, it wasn’t that easy. Attempts to load this feature ended up with error code 0x800f0954. What in the hades is error code 0x800f0954?

Time to hit the Google and wow, there’s a bunch of random articles on this error code. I already had a hunch this had something to do with WSUS. We use SCCM in our environment and SCCM sets the WSUS server in the client registry to a WSUS server without any binaries, an empty WSUS server if you will. I usually fix that by deleting the whole registry key HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU. The SCCM client will recreate this key periodically. Unfortunately, this did not work. Where to look? Our old friend C:\windows\logs\cbs\cbs.log and we find:

2023-11-01 13:12:37, Info CBS External EvaluateApplicability, package: Package_8_for_KB5031005~31bf3856ad364e35~amd64~~10.0.4069.1, package applicable State: Installed, highest update applicable state: Installed, resulting applicable state:Installed
2023-11-01 13:12:37, Info CBS External EvaluateApplicability, package: Package_for_DotNetRollup~31bf3856ad364e35~amd64~~10.0.4069.1, package applicable State: Installed, highest update applicable state: Installed, resulting applicable state:Installed
2023-11-01 13:12:37, Info CBS DLWD: Expecting search returns 1 update, actual:0 [HRESULT = 0x800f0954 - CBS_E_INVALID_WINDOWS_UPDATE_COUNT_WSUS]
2023-11-01 13:12:37, Info CBS DWLD:Failed to do Windows update search [HRESULT = 0x800f0954 - CBS_E_INVALID_WINDOWS_UPDATE_COUNT_WSUS]
2023-11-01 13:12:37, Info CBS FC: WindowsUpdateDownloadFromUUP returns. [0x800F0954]
2023-11-01 13:12:37, Error CBS FC: CFCAcquirerWUClient::Download(136): Result = 0x800F0954
2023-11-01 13:12:37, Error CBS FC: CFCAcquirerWrapper::Execute(147): Result = 0x800F0954
2023-11-01 13:12:37, Info CBS Exec: Failed to download FOD from WU, retry onece. [HRESULT = 0x800f0954 - CBS_E_INVALID_WINDOWS_UPDATE_COUNT_WSUS]

It IS a WSUS problem, but why didn’t deleting the WindowsUpdate registry key help? Well, it appears the WindowsUpdate service only reads this registry key when it starts and if you change or delete this key after it’s running you have to restart the service so it takes note of the new changes. Oh, I like the misspelling of “retry onece” in the logs.

It also didn’t matter if I tried to point Powershell or DISM directly to the binaries in the SxS folder, it wasn’t having anything of that without being able to reach out to Windows Update. Odd.

What’s so frustrating is that I cannot find this error code in any lookup tool such as helpmsg or cmtrace. It’s not documented anywhere I can find. If the program had spit out the whole error message instead of just some random hex code, I could have saved 30 minutes of my life doing something really important, like fixing someone’s Office 365 mailbox that they deleted all of the e-mails out of (oof).

  • Soli Deo Gloria

Get an Extra Month of Internet Service on the Calyx Institute Network

If you use my referral link, you can get an extra month of Internet service on the Calyx Institute network and I get an extra month of Internet service as well.

They use the T-Mobile network and your hotspot will have unlimited data.

Your mileage will vary based on location, but I get around 250Mbps using the hotspot. If you work from home, I highly suggest having a backup Internet option in case your main Internet goes out.

  • Soli Deo Gloria

ERROR_SXS_ASSEMBLY_MISSING Chaos

Tried to add IIS and MSMQ features to a server. Kept getting a 0x80073701 error: missing assembly file. Off to C:\windows\logs\cbs.log we go:

2023-09-06 07:02:33, Error CSI 00000009 (F) STATUS_SXS_ASSEMBLY_MISSING #2625634# from CCSDirectTransaction::OperateEnding at index 0 of 1 operations, disposition 2[gle=0xd015000c]
2023-09-06 07:02:33, Error CSI 0000000a (F) HRESULT_FROM_WIN32(ERROR_SXS_ASSEMBLY_MISSING) #2625476# from Windows::ServicingAPI::CCSITransaction::ICSITransaction_PinDeployment(Flags = 0, a = dbbb65b179c955b3c0186aa84fa6e087, version 10.0.17763.3165, arch amd64, nonSxS, pkt {l:8 b:31bf3856ad364e35}, cb = (null), s = (null), rid = 'Package_4455_for_KB5022286~31bf3856ad364e35~amd64~~10.0.1.7.5022286-8227_neutral', rah = (null), manpath = (null), catpath = (null), ed = 0, disp = 0)[gle=0x80073701]

On Google, I found this post, but I will save you the time: downloading said update, expanding it to a CAB file and then adding the CAB file via DISM did absolutely nothing to fix the problem. Neither did running SFC /scannow or dism /online /cleanup-image /restorehealth.

The fix is to remove the keys referencing the bad KB from the registry under HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Packages, then try re-adding the roles from Server Manager. I suggest using Baretail to watch C:\windows\logs\cbs.log while you are doing this to see if additional errors come back up (you may need to do this fix for multiple KBs. In my case, I would fix one and another KB would pop up).

Before running this script, run regedit using the psexec -s -i cmd trick to run under the SYSTEM account, then go to HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Packages, right-click on Packages and grant SYSTEM full control. Trying to adjust permissions and take ownership of the registry keys within the script was a nightmare, so I went back to basics by removing that logic and just set permissions manually using the registry editor.

You’ll need to run the Powershell script under the same SYSTEM trick above to avoid any permission issues removing the keys:

# Define the root path to search in
$rootPath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Packages"

# Get all child items (keys) under the root path
$keys = Get-ChildItem -Path $rootPath

# Filter the keys based on the presence of the desired values in the name
$filteredKeys = $keys | Where-Object { $_.Name -like '*KB5022286*' -or $_.Name -like '*KB5027222*' }

# Loop through each matching key and remove it
$filteredKeys | ForEach-Object {
    # Extract the key's path
    $keyPath = $_.Name -replace 'HKEY_LOCAL_MACHINE', 'HKLM:'

    # Remove the key
    Remove-Item -Path $keyPath -Recurse -Force
}

Write-Output "Operation completed."

Now for the “root cause analysis”, a buzz word we love to throw around in IT: it appears that someone completely cleared out the contents of C:\Windows\SoftwareDistribution on the server and DISM couldn’t find the source files anymore for these KBs. However, there were other KBs pointed to this folder (which was empty) and they worked just fine? Perhaps these specific KBs actually updated the core IIS files within the OS and that’s why DISM was querying them during the IIS/MSMQ role add?

Perhaps a better solution is to copy the SoftwareDistribution folder from a server running the same server OS where the downloads are not cleared from the folder. Not sure if the GUIDs would match up between the two different servers, but might be worth trying the next time this comes up. If you should try this route yourself, you’ll need to temporarily disable and stop the Windows Update service on both servers as it likes to lock files in this folder.

If you were also curious: Windows Update keeps working just fine after the procedure of removing bad KBs from HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Packages.

  • Soli Deo Gloria

Windows Server 2019 installation has failed

Recently, I’ve been doing in-place upgrades from Server 2012R2 to Server 2019 and having some issues. They usually sit at 32% or 37% for 4 to 6 hours before the setup continues on. At the very end, I’ll sometimes get a “Windows Server 2019 installation has failed” error message and nothing else. I then and go look for setuperr.log on the disk and always see these errors:

2023-08-21 15:07:33, Error [0x0808fe] MIG Plugin {D12A3141-A1FF-4DAD-BF67-1B664DE1CBD6}: WSLicensing: Error reading Server Info hr=0x80070490
2023-08-21 15:07:38, Error CSetupAutomation::Resurrect: File not found: C:\$WINDOWS.~BT\Sources\Panther\automation.dat[gle=0x00000002]
2023-08-21 15:07:38, Error SP CSetupPlatform::ResurrectAutomation: Failed to resurrect automation: 0x80070002[gle=0x00000002]
2023-08-21 15:07:38, Error SP CMountWIM::DoExecute: Failed to mount WIM file C:\$WINDOWS.~BT\Sources\SafeOS\winre.wim. Error 0x80070522[gle=0x00000522]
2023-08-21 15:07:38, Error SP Operation failed: Mount WIM file C:\$WINDOWS.~BT\Sources\SafeOS\winre.wim, index 1 to C:\$WINDOWS.~BT\Sources\SafeOS\SafeOS.Mount. Error: 0x80070522[gle=0x000000b7]
2023-08-21 15:07:38, Error SP ExecuteOperations: Failed execution phase Pre-Finalize. Error: 0x80070522
2023-08-21 15:07:38, Error MOUPG MoSetupPlatform: ExecuteCurrentOperations reported failure!
2023-08-21 15:07:38, Error MOUPG MoSetupPlatform: Using action error code: [0x80070522]
2023-08-21 15:07:38, Error MOUPG CDlpActionPreFinalize::ExecuteRoutine(545): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CDlpActionImpl > > >::Execute(441): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CDlpTask::ExecuteAction(3259): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CDlpTask::ExecuteActions(3413): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CDlpTask::Execute(1644): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CSetupManager::ExecuteTask(2478): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CSetupManager::ExecuteTask(2441): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CSetupManager::ExecuteInstallMode(883): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CSetupManager::ExecuteDownlevelMode(390): Result = 0x80070522
2023-08-21 15:07:39, Error SP CDeploymentBase::CleanupMounts: Unable to unmount the directory C:\$WINDOWS.~BT\Sources\SafeOS\SafeOS.Mount. Error: 0xC142011C[gle=0xc142011c]
2023-08-21 15:07:41, Error MOUPG CSetupManager::Execute(282): Result = 0x80070522
2023-08-21 15:07:41, Error MOUPG CSetupHost::Execute(400): Result = 0x80070522

Using a trick from Sami Laiho, we can look up error codes using net helpmsg <4 digit number> or winrm helpmsg <hexcode>. 0x80070522 comes out to be “A required privilege is not held by the client” which is very odd. What I can tell you is its failure to mount WinRE.WIM is a red herring and has absolutely nothing to do with the actual problem. I suspect when there is any error earlier in the pipeline that cannot be ignored, a generic error is spit out regardless of what actually happened.

If you comb through setupact.log nothing will stand out as being a problem and setupdiag.exe only works on Windows 10 and 11, so we are on our own for figuring this problem out.

Through trial and error, I actually figured out what was going on, so I will list the prep steps I now do which took the process from 4 to 6 hours to about 15 minutes and I didn’t get any in-place setup failures anymore.

  1. Block GPO inheritance on an OU and then move the server computer account to that OU.
  2. Delete everything in C:\windows\system32\GroupPolicy and then restart the server. You may have to turn on showing hidden items to see this folder.
  3. Run secedit /configure /cfg %windir%\inf\defltbase.inf /db defltbase.sdb /verbose to reset local group policy back to in-box defaults.
  4. Make sure all built-in Microsoft services are functioning such as the print spooler (we can disable it after the upgrade).
  5. Run dism /online /cleanup-image /restorehealth to repair any CBS store corruption.
  6. Remove any extraneous roles or programs not needed.
  7. Run psexec -s -i cmd and then launch setup.exe. This runs the setup under the SYSTEM account which has more permissions then just local Administrator. PSEXEC is part of the Sysinternals Suite you can download free from Microsoft.

Depending on what’s in your environment, you may have to go further after the in-place setup is done. As an example: I am upgrading Lansweeper scanning servers and they require .NET Framework 4.8. During the in-place upgrade, .NET framework is removed, so I need to reinstall it. Luckily, I was able to determine that from the Application event log. Another one is the SCCM client. All custom WMI classes that it uses get reset when the in-place upgrade is complete, so I need to uninstall the SCCM client (ccmsetup /uninstall) and then reinstall the SCCM client.

Don’t forget to move your computer account back to its original OU and disable any services needed for security reasons.

Back to the 4 to 6 hour delay: I believe that is because I disabled Internet Explorer on the servers using a SRS rule in a domain GPO. I was told for security reasons that Internet Explorer couldn’t run on our servers anymore and there is no group policy setting for servers to disable Internet Explorer, they only have that for Windows 10 and that’s only for later builds. It seems that Microsoft assumes that the operating system is in a particular state and if it’s not, it has a hard time performing the in-place upgrade.

  • Soli Deo Gloria

Case of Operator Failure

We are retiring an old file server at work. One of the file shares held a bunch of text files of people’s log offs: username, date, time. The problem is I had no idea where this script was running from. RSOP.MSC didn’t show any scripts that would do this in the Logoff section of any applied GPOs and searches of SYSVOL with Agent Ransack came up with nothing.

I decided to use an old Procmon trick from Sami Laiho: https://4sysops.com/archives/using-process-monitor-procmon-remotely/. Basically, this allows us to run Procmon remotely and in another user session, so we can trace events during user logons and logoffs. I did the remote Procmon trick, logged on and off and then took a look at the PML file in Procmon. I searched for the name of the share and behold: I found powershell.exe running a file called logout_oldadmin.ps1.

Doing an e-mail search with the script name, I found some old gpresult HTML reports in old e-mail messages which led me to the GPO name in question that was firing this script off. The question is why didn’t I see in this RSOP.MSC and why didn’t Agent Ransack find the script?

RSOP.MSC hasn’t been supported since 2006 and Microsoft even warns it may not show all of the group policies. Instead, you are supposed to use gpresult /h report.html. Mea culpa.

The Agent Ransack issue…I had a date filter set and didn’t realize it. Oof! Today’s lesson is if you don’t find what you are looking for, use a different tool. When I dropped to a CMD session, mapped a drive to SYSVOL and used “dir”, I could see the files.

  • Soli Deo Gloria