Adam Leinss' Tech Tips

Here we go again: another server, another error. Why can’t things just work properly? Had a consultant e-mail me they couldn’t load .NET Framework 3.5 on Windows Server 2019. “Easy peasy lemon squeezy” I thought. Well, of course, it wasn’t that easy. Attempts to load this feature ended up with error code 0x800f0954. What in the hades is error code 0x800f0954?

Time to hit the Google and wow, there’s a bunch of random articles on this error code. I already had a hunch this had something to do with WSUS. We use SCCM in our environment and SCCM sets the WSUS server in the client registry to a WSUS server without any binaries, an empty WSUS server if you will. I usually fix that by deleting the whole registry key HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU. The SCCM client will recreate this key periodically. Unfortunately, this did not work. Where to look? Our old friend C:\windows\logs\cbs\cbs.log and we find:

2023-11-01 13:12:37, Info CBS External EvaluateApplicability, package: Package_8_for_KB5031005~31bf3856ad364e35~amd64~~10.0.4069.1, package applicable State: Installed, highest update applicable state: Installed, resulting applicable state:Installed
2023-11-01 13:12:37, Info CBS External EvaluateApplicability, package: Package_for_DotNetRollup~31bf3856ad364e35~amd64~~10.0.4069.1, package applicable State: Installed, highest update applicable state: Installed, resulting applicable state:Installed
2023-11-01 13:12:37, Info CBS DLWD: Expecting search returns 1 update, actual:0 [HRESULT = 0x800f0954 - CBS_E_INVALID_WINDOWS_UPDATE_COUNT_WSUS]
2023-11-01 13:12:37, Info CBS DWLD:Failed to do Windows update search [HRESULT = 0x800f0954 - CBS_E_INVALID_WINDOWS_UPDATE_COUNT_WSUS]
2023-11-01 13:12:37, Info CBS FC: WindowsUpdateDownloadFromUUP returns. [0x800F0954]
2023-11-01 13:12:37, Error CBS FC: CFCAcquirerWUClient::Download(136): Result = 0x800F0954
2023-11-01 13:12:37, Error CBS FC: CFCAcquirerWrapper::Execute(147): Result = 0x800F0954
2023-11-01 13:12:37, Info CBS Exec: Failed to download FOD from WU, retry onece. [HRESULT = 0x800f0954 - CBS_E_INVALID_WINDOWS_UPDATE_COUNT_WSUS]

It IS a WSUS problem, but why didn’t deleting the WindowsUpdate registry key help? Well, it appears the WindowsUpdate service only reads this registry key when it starts and if you change or delete this key after it’s running you have to restart the service so it takes note of the new changes. Oh, I like the misspelling of “retry onece” in the logs.

It also didn’t matter if I tried to point Powershell or DISM directly to the binaries in the SxS folder, it wasn’t having anything of that without being able to reach out to Windows Update. Odd.

What’s so frustrating is that I cannot find this error code in any lookup tool such as helpmsg or cmtrace. It’s not documented anywhere I can find. If the program had spit out the whole error message instead of just some random hex code, I could have saved 30 minutes of my life doing something really important, like fixing someone’s Office 365 mailbox that they deleted all of the e-mails out of (oof).

• Soli Deo Gloria

Tried to add IIS and MSMQ features to a server. Kept getting a 0x80073701 error: missing assembly file. Off to C:\windows\logs\cbs.log we go:

2023-09-06 07:02:33, Error CSI 00000009 (F) STATUS_SXS_ASSEMBLY_MISSING #2625634# from CCSDirectTransaction::OperateEnding at index 0 of 1 operations, disposition 2[gle=0xd015000c]
2023-09-06 07:02:33, Error CSI 0000000a (F) HRESULT_FROM_WIN32(ERROR_SXS_ASSEMBLY_MISSING) #2625476# from Windows::ServicingAPI::CCSITransaction::ICSITransaction_PinDeployment(Flags = 0, a = dbbb65b179c955b3c0186aa84fa6e087, version 10.0.17763.3165, arch amd64, nonSxS, pkt {l:8 b:31bf3856ad364e35}, cb = (null), s = (null), rid = 'Package_4455_for_KB5022286~31bf3856ad364e35~amd64~~10.0.1.7.5022286-8227_neutral', rah = (null), manpath = (null), catpath = (null), ed = 0, disp = 0)[gle=0x80073701]

On Google, I found this post, but I will save you the time: downloading said update, expanding it to a CAB file and then adding the CAB file via DISM did absolutely nothing to fix the problem. Neither did running SFC /scannow or dism /online /cleanup-image /restorehealth.

The fix is to remove the keys referencing the bad KB from the registry under HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Packages, then try re-adding the roles from Server Manager. I suggest using Baretail to watch C:\windows\logs\cbs.log while you are doing this to see if additional errors come back up (you may need to do this fix for multiple KBs. In my case, I would fix one and another KB would pop up).

Before running this script, run regedit using the psexec -s -i cmd trick to run under the SYSTEM account, then go to HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Packages, right-click on Packages and grant SYSTEM full control. Trying to adjust permissions and take ownership of the registry keys within the script was a nightmare, so I went back to basics by removing that logic and just set permissions manually using the registry editor.

You’ll need to run the Powershell script under the same SYSTEM trick above to avoid any permission issues removing the keys:

# Define the root path to search in
$rootPath = "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Packages"

# Get all child items (keys) under the root path
$keys = Get-ChildItem -Path $rootPath

# Filter the keys based on the presence of the desired values in the name
$filteredKeys = $keys | Where-Object { $_.Name -like '*KB5022286*' -or $_.Name -like '*KB5027222*' }

# Loop through each matching key and remove it
$filteredKeys | ForEach-Object {
    # Extract the key's path
    $keyPath = $_.Name -replace 'HKEY_LOCAL_MACHINE', 'HKLM:'

    # Remove the key
    Remove-Item -Path $keyPath -Recurse -Force
}

Write-Output "Operation completed."

Now for the “root cause analysis”, a buzz word we love to throw around in IT: it appears that someone completely cleared out the contents of C:\Windows\SoftwareDistribution on the server and DISM couldn’t find the source files anymore for these KBs. However, there were other KBs pointed to this folder (which was empty) and they worked just fine? Perhaps these specific KBs actually updated the core IIS files within the OS and that’s why DISM was querying them during the IIS/MSMQ role add?

Perhaps a better solution is to copy the SoftwareDistribution folder from a server running the same server OS where the downloads are not cleared from the folder. Not sure if the GUIDs would match up between the two different servers, but might be worth trying the next time this comes up. If you should try this route yourself, you’ll need to temporarily disable and stop the Windows Update service on both servers as it likes to lock files in this folder.

If you were also curious: Windows Update keeps working just fine after the procedure of removing bad KBs from HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Packages.

• Soli Deo Gloria

Recently, I’ve been doing in-place upgrades from Server 2012R2 to Server 2019 and having some issues. They usually sit at 32% or 37% for 4 to 6 hours before the setup continues on. At the very end, I’ll sometimes get a “Windows Server 2019 installation has failed” error message and nothing else. I then and go look for setuperr.log on the disk and always see these errors:

2023-08-21 15:07:33, Error [0x0808fe] MIG Plugin {D12A3141-A1FF-4DAD-BF67-1B664DE1CBD6}: WSLicensing: Error reading Server Info hr=0x80070490
2023-08-21 15:07:38, Error CSetupAutomation::Resurrect: File not found: C:\$WINDOWS.~BT\Sources\Panther\automation.dat[gle=0x00000002]
2023-08-21 15:07:38, Error SP CSetupPlatform::ResurrectAutomation: Failed to resurrect automation: 0x80070002[gle=0x00000002]
2023-08-21 15:07:38, Error SP CMountWIM::DoExecute: Failed to mount WIM file C:\$WINDOWS.~BT\Sources\SafeOS\winre.wim. Error 0x80070522[gle=0x00000522]
2023-08-21 15:07:38, Error SP Operation failed: Mount WIM file C:\$WINDOWS.~BT\Sources\SafeOS\winre.wim, index 1 to C:\$WINDOWS.~BT\Sources\SafeOS\SafeOS.Mount. Error: 0x80070522[gle=0x000000b7]
2023-08-21 15:07:38, Error SP ExecuteOperations: Failed execution phase Pre-Finalize. Error: 0x80070522
2023-08-21 15:07:38, Error MOUPG MoSetupPlatform: ExecuteCurrentOperations reported failure!
2023-08-21 15:07:38, Error MOUPG MoSetupPlatform: Using action error code: [0x80070522]
2023-08-21 15:07:38, Error MOUPG CDlpActionPreFinalize::ExecuteRoutine(545): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CDlpActionImpl > > >::Execute(441): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CDlpTask::ExecuteAction(3259): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CDlpTask::ExecuteActions(3413): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CDlpTask::Execute(1644): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CSetupManager::ExecuteTask(2478): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CSetupManager::ExecuteTask(2441): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CSetupManager::ExecuteInstallMode(883): Result = 0x80070522
2023-08-21 15:07:39, Error MOUPG CSetupManager::ExecuteDownlevelMode(390): Result = 0x80070522
2023-08-21 15:07:39, Error SP CDeploymentBase::CleanupMounts: Unable to unmount the directory C:\$WINDOWS.~BT\Sources\SafeOS\SafeOS.Mount. Error: 0xC142011C[gle=0xc142011c]
2023-08-21 15:07:41, Error MOUPG CSetupManager::Execute(282): Result = 0x80070522
2023-08-21 15:07:41, Error MOUPG CSetupHost::Execute(400): Result = 0x80070522

Using a trick from Sami Laiho, we can look up error codes using net helpmsg <4 digit number> or winrm helpmsg <hexcode>. 0x80070522 comes out to be “A required privilege is not held by the client” which is very odd. What I can tell you is its failure to mount WinRE.WIM is a red herring and has absolutely nothing to do with the actual problem. I suspect when there is any error earlier in the pipeline that cannot be ignored, a generic error is spit out regardless of what actually happened.

If you comb through setupact.log nothing will stand out as being a problem and setupdiag.exe only works on Windows 10 and 11, so we are on our own for figuring this problem out.

Through trial and error, I actually figured out what was going on, so I will list the prep steps I now do which took the process from 4 to 6 hours to about 15 minutes and I didn’t get any in-place setup failures anymore.

1. Block GPO inheritance on an OU and then move the server computer account to that OU.
2. Delete everything in C:\windows\system32\GroupPolicy and then restart the server. You may have to turn on showing hidden items to see this folder.
3. Run secedit /configure /cfg %windir%\inf\defltbase.inf /db defltbase.sdb /verbose to reset local group policy back to in-box defaults.
4. Make sure all built-in Microsoft services are functioning such as the print spooler (we can disable it after the upgrade).
5. Run dism /online /cleanup-image /restorehealth to repair any CBS store corruption.
6. Remove any extraneous roles or programs not needed.
7. Run psexec -s -i cmd and then launch setup.exe. This runs the setup under the SYSTEM account which has more permissions then just local Administrator. PSEXEC is part of the Sysinternals Suite you can download free from Microsoft.

Depending on what’s in your environment, you may have to go further after the in-place setup is done. As an example: I am upgrading Lansweeper scanning servers and they require .NET Framework 4.8. During the in-place upgrade, .NET framework is removed, so I need to reinstall it. Luckily, I was able to determine that from the Application event log. Another one is the SCCM client. All custom WMI classes that it uses get reset when the in-place upgrade is complete, so I need to uninstall the SCCM client (ccmsetup /uninstall) and then reinstall the SCCM client.

Don’t forget to move your computer account back to its original OU and disable any services needed for security reasons.

Back to the 4 to 6 hour delay: I believe that is because I disabled Internet Explorer on the servers using a SRS rule in a domain GPO. I was told for security reasons that Internet Explorer couldn’t run on our servers anymore and there is no group policy setting for servers to disable Internet Explorer, they only have that for Windows 10 and that’s only for later builds. It seems that Microsoft assumes that the operating system is in a particular state and if it’s not, it has a hard time performing the in-place upgrade.

• Soli Deo Gloria