This one took me a while to solve. The desktop guys kept coming to me stating when they re-imaged a computer, it either didn’t ping or it had the wrong IP address. I found out later they had changed their imaging methodology. Before re-imaging any computer, they first delete the computer account and then re-image it. I would guess that this netjoin hardening change is the reason.
When I went into DNS management, I could clearly see an “Account unknown” in the ACL of the DNS record, which makes sense, because the computer account registered the DNS record, but now that computer account didn’t exist anymore. Until the DNS record is scavenged or deleted manually, the newly imaged computer will be unable to update its own DNS record.
This led me down a path of many dead ends. I wrote a script to compare DHCP leases to DNS records. However, I soon found out that DHCP is not always correct either for the current IP address. If someone moves from location to location, the last DHCP lease is the one you want to use. I then looked into making DHCP the owner and updater of all dynamic DNS records, but this too caused issues such as duplicate DNS records.
I then looked at trying to find any DNS records with “Account unknown” in the ACL, but the script ended up too complex and just didn’t work. It was back to basics: I only cared about recently deleted computer accounts, so why not just look for recently deleted computer accounts and then delete the DNS records for those accounts?
That’s exactly what dns_orphan_fix.ps1 does. It looks back 60 minutes for any deleted computer accounts and then attempts to delete the DNS records for those accounts. I run this in the task scheduler every 30 minutes, so that does mean that DNS records will get deleted twice, but I shouldn’t miss any deleted computer accounts this way. There is a “$dryrun” option that you can flip to $true just to make sure this script will operate the way you think it will operate in your environment before setting it to $false to actually delete DNS records.
Now that you found your favorite patching program, we need to send out notifications and target specific servers for patching. I schedule updates as an offset from Patch Tuesday. Patch Tuesday is always the 2nd Tuesday of the month and is when Microsoft releases it’s monthly patches. I currently do a pilot group which is fired the night of Patch Tuesday, and then +4, +11 and +18 days after Patch Tuesday, which is every Saturday after Patch Tuesday.
Send-UpdateNotification2.ps1 is a notification script I use with SCCM. This will send out notifications at 8AM on Friday for a maintenance window that takes place Saturday night from 10PM to 5AM. Due to the way the months work, for +18 days after Patch Tuesday, I send these out on Thursdays instead of Fridays as the script doesn’t work correctly if the date rolls into the next month. You would run this script as a scheduled task once a day at 8AM. Note that this script needs to run from the SCCM server itself.
If you are using SCCM, you can set maintenance windows based on offsets from Patch Tuesday using New-CMMaintenanceWindow.ps1. This is a Powershell script made by Mattias Benninge. As an example: I run this Powershell script once a year to setup all of my server maintenance windows. This starts the maintenance window every Saturday after Patch Tuesday of the month starting at 10PM going to 5AM Sunday.
Dump-Computers.ps1 is used to dump computers from a SCCM collection into a plain text file. Note that this script needs to run from the SCCM server itself. If you use WSUS & WSUS groups, take a look at CreateWSUSGroups.ps1. This script would need to run from the WSUS server itself.
Once we gather the plain text files with the different computers, we can copy them to servers.txt. patchtuesday.ps1 is a modification of Send-UpdateNotification2.ps1. We use this script to copy our specific group of computers to servers.txt based on the offset from Patch Tuesday. I run this every day as a scheduled task at 8AM. On Sundays, I run a scheduled task at 1AM and 3AM that runs a batch file, but I don’t want to run this outside of the maintenance windows above, so I use patchem.bat. The batch file has a “counter file” that increments to 2 when servers.txt and serversplus18.txt are equal. When the counter increments to 2, the batch file will no longer try to patch servers. When the date rolls over to the next month for pilot patching, the counter file is wiped out and the whole process starts over again for that month. The one thing to note is when you run the batch file in the scheduled task, be sure to set the current working directory to the proper folder in the scheduled task, otherwise it will not work properly (batch files are an ancient technology and assume the current working directory for all operations, if your current working directory is not set to where the batch file is running from, weird things will happen).
Some servers just refuse to restart on their own after patching, so we can use restart_sniffer.ps1 to nudge them along. This searches servers.txt and then does a restart on a server if PendingReboot is set to True. This script uses Get-PendingRebootStatus.ps1 from TheSystemAdminChannel website. I run this every Sunday at 5AM. Undoubtedly, you’ll find servers that did not patch because they ran out of disk space. The “famous” servers for doing this are ones that have the IIS role installed since they like to log every web visit. Take a look at clean_iis_logs.ps1 for how you can keep 14 days of logs and ditch the rest.
The final step is to run a report checking patch compliance. All of this hard work of getting the monthly updates via an API has been done for us already.
As an update to this part: these 2 scripts would work for a few months, then stop working and after looking at the code, I discovered I could get what I needed with just 5 lines of code. You can check that out as Get-SecurityUpdateAL.ps1 on my GitHub site. – 6/23/22.
You can ignore instructions for getting your own API key, as it appears that is not necessary anymore. Note that you will need to install the MSRCSecurityUpdates Powershell module to use this script.
Now you can run PatchReport.ps1 to parse the results of this month’s KBs. This will search the patches.txt file looking for CUs and monthly rollups for Windows Server 2012, 2012R2, 2016, 2019, 2022 and then run those results against servers.txt that we generated from the Patch Tuesday Powershell scripts above. I also created PatchReportSMTP.ps1 to send e-mails of the same report to me.
A sample patch compliance report is shown below. If a row in the column in InstalledBy is empty, that means that server did not restart after patching and is not fully patched (yet).
Any missing patches will show up like this in orange text:
I commented out the error code about “Cannot connect to computer X”, since I run the same script against the same servers.txt file in two different domains. About 9 out of 10 times, a failure to patch is a low disk space issue. In a previous blog posting, I talked about how disabling the HTTP.SYS driver blocked patching and the remedy for that, though something like that happening is pretty rare.
WSUS and SCCM are great, but maybe you want more control of when Microsoft updates run. There are a few products that can help us with patching: ABC-Update, BatchPatch, WUInstall, and PSWindowsUpdate. I’ve tried all of these and will give my opinion on each.
The first thing we need to do is setup WinRM for every computer we want to update. This is usually pretty easy: just type winrm quickconfig on the computer you want to update. You can also do this through GPO as well: https://www.techrepublic.com/article/how-to-enable-powershell-remoting-via-group-policy/. This allows remote Powershell access so we can run actions against a list of computers remotely.
The first contender is ABC-Update. I’ve used this one for a while. It comes in a command line and GUI version and is completely free. There is one downside and that is it requires .NET framework to be installed on the computers it patches, which may not be on all of your servers. The command line is relatively easy to understand:
We tell ABC-Update to connect to MSUpdate (not WSUS), grab only the Critical and Security Microsoft updates, install them, restart up to 3 times and send us an e-mail when it’s done. Pretty simple. For the GUI version: you can give it a plain text file of computer names or point it at an AD OU and it spawns a process on each computer PSEXEC style that installs a scheduled task that will fire at the time and date of your choosing. You can watch the update status in real time, cancel the updates and re-schedule them as needed. The author is an IT professional and is very responsive and open to feature requests.
Batchpatch requires PSEXEC from Sysinternals to do it’s work. I’ll admit I did not spend much time with Batchpatch. There appears to be no easy way of scheduling updates for a certain time/date or to scope update categories to specific ones. Batchpatch is $399 per year per admin user.
WUinstall is very similar to ABC-Update, however, it only is available as a command line product. The company does have a RMM suite called Xeox that would presumably give you a GUI experience for updating computers if desired. This software is very expensive: to patch 100 clients is $390/year and it goes up from there. There is a 30 day full version trial.
My first trial run of WUInstall ended up with a lot of servers not getting restarted. There are /reboot_if_needed and a /reboot_if_needed_force command line options. Why there is even a difference, I do not know. WUinstall does not require .NET framework to be installed on the servers it patches, so you will be able to patch servers with less software requirements.
PSWindowsUpdate was used at a previous employer, version 1.5, so I decided to give version 2.2 a spin. This is a free Powershell module and has no .NET framework requirement. I did not want to install this Powershell module on all my servers, so I loaded it on my DC’s NETLOGON share and modified $env:PSModulePath to include to UNC path to the PSWindowsUpdate module.
This is an abbreviated snippet from the Powershell code, you can find the files over at my Github. KB890830 is the Windows Malicious Software Removal Tool which is generally useless and wastes patching time, so I exclude it to speed up the patching process. I also had to move the source files from the subfolder \PSWindowsupdate\2.2.02 to just \PSWindowsupdate as it couldn’t find the files when I tried to load the module.
I use a scheduled task on a server and PSEXEC to kick off the Powershell script to patch servers at 1AM and 3AM after SCCM starts it’s patching at 10PM.
I was transferring department folders and files from one server to another. I used Robocopy and FreeFileSync to get the job done. Just for my own future reference, this was the Robocopy command I used:
I fired this off once, then used FreeFileSync to sync any delta changes afterward. This is pretty boring and common, server administrators have been performing this ritual for many years. However, I’ve been in desktop engineering for the last 16 years and only 2 years in server administration, so it was “new to me”.
Weeks later, after I was done, I got an e-mail about invalid shortcuts. In order to save disk space, the users created shortcuts to other folders. These shortcuts of course included the name of the old server. Off to Google we go! One solution I found was this, but didn’t end up using it (our AV software kept deleting it for some reason) http://jacquelin.potier.free.fr/ShortcutsSearchAndReplace/.
This works great, it even has a “dryrun” option so you can see what it’s going to do before it actually does anything. The only thing to watch out for is if the shortcut was already invalid to begin with. About 1/3 of the shortcuts the user had were already invalid on the old server due to them shifting and renaming various folders and not updating the shortcuts.
This is a quick and dirty script for PSADT (http://psappdeploytoolkit.com/) to deploy BIOS updates relating to Intel’s Meltdown/Spectre vulnerability.
PSADT is designed to be used in SCCM deployments, however, it is agnostic enough that it should be able to be used with any software management solution such as PDQ Deploy.
Once we run one at least one block of firmware update code, $FirmwareUpdateRan will be set to TRUE. Setting this flag will prevent the restart prompt later on if we didn’t run any update code. $BIOSVersion should compared against the version of the BIOS you want to update to. Easiest way of getting this is just running Get-WmiObject Win32_BIOS on the test computer after you run the current BIOS update.
{ $Response = Show-InstallationPrompt -Message 'Executing BIOS update...please close all apps' -ButtonRightText
'Cancel' -ButtonLeftText 'Continue' -Timeout 600
if ($Response -eq 'Cancel') { exit 12345 }
Show a prompt to end user. The majority of the BIOS updates will force a reboot right away without any warning, thus we display a message to the end user and allow them to cancel it.
This is useful for satisfying the detection rule for SCCM. There’s no clean way of determining whether there is a failure of the BIOS update, other than running a compliancy report in your software/hardware inventory reporting tool to make sure the update happened.
set-location $dirfiles\M900
Lenovo’s flash utility doesn’t accept absolute paths: we have to run it from the current directory, so we use set-location to force the location folder.
This is only shown if the BIOS update didn’t force a reboot. Currently, I only found the T460S and Yoga S1 laptops do not force a reboot. Since reboot isn’t forced, we force one with a 10 minute countdown.
Suspends BitLocker for one reboot, otherwise laptop will go into recovery mode. Note this command is supported for Windows 8 and later only. For Windows 7 you will need to use manage-bde: Manage-bde.exe –protectors –disable c:. I didn’t see any -rc option, so you will need to do something such as a scheduled task to turn it back on.
We run a program called QATRAX which doesn’t come as a MSI file for installing it. The program has many sins: one of them being that it can only be installed to C:\program files\traxstar (it’s a x86 program) and it has to have write access to this folder. Attempts to use a re-packager to convert this to a MSI file have failed, because all repackagers detect it should really go in C:\program files (x86), but of course that won’t work.
The program was made in the 1990s, so figuring out what files it copies to to system is quite easy. You can use a tracing utility such as Process Monitor to watch for install changes.
# copy qatrax support files
copy-item *.dll C:\windows\syswow64 -Force
We could add registry entries using Powershell code line-by-line, but doing a registry export to a REG file and then import it is far easier in my opinion.
One thing to note is that we need to create an uninstall string manually so it shows up in Programs and Features as being installed. This will also report back to WMI that the program is really installed to help with software inventory. You should be able to get this from a system with the software already installed.
The program also has no native way of switching between dev and prod environments, so I wrote another wrapper that overwrites the values in the registry with certain IP addresses depending on which environment the user wants to be in.
# set everyone full-control to traxstar registry key. this is a hack to allow
# switching between dev and prod environments
$acl = Get-Acl 'HKLM:\SOFTWARE\Wow6432Node\TraxStar Technologies LLC'
$rule = New-Object System.Security.AccessControl.RegistryAccessRule ("Everyone","FullControl","Allow")
$acl.SetAccessRule($rule)
$acl |Set-Acl -Path 'HKLM:\SOFTWARE\Wow6432Node\TraxStar Technologies LLC'
Make a folder in C:\program files
# create install folder (and yes, traxstar has to be in the x64 folder even though it's
# x86 or it will break
mkdir "C:\program files\traxstar\traxclient"
Grant everyone rights to write to this folder
# qatrax requires write access to its own folder
icacls 'C:\program files\traxstar\traxclient' /grant:r everyone:f
This will need to be run as the end user to use $env:username, but if your ExecutionPolicy doesn’t allow it, you can remove it.
After running this for a few days: you can do a “dir *.* > list.txt” and then import this into an Excel spreadsheet using the import data from text file feature.
The –% option (that’s dash-dash%) basically just says “Powershell, just pass these arguments along and don’t try to interpret them”. This functionality requires Powershell v3 or later.
Probably would have been better to use Start-Process and check if the exitcode is not zero. Note to use Dell Command to turn on the TPM chip you need to set a BIOS password and for 64-bit systems you need to use the 64-bit version of CCTK.
Here was a fun installer to get working silently. This one uses something called InstallAnywhere. It is a java based installer and if you Google InstallAnywhere silent, you will happen upon several command line options. The correct set for this version of the installer (2009 version) can be found here.
Here’s the command to install it silently:
xaclient_Hgenas400_P36001.exe -i silent
We can also record settings into a file and play those back. To record:
We kick off the installer, tell PowerShell to wait for the process to end and return an object (-Passthru), grant Everyone read and execute rights to the icon and then copy that icon to the the system shared desktop.
If you execute this code, however, the PowerShell script never progresses. This is because the installer runs the full program as a child process from the installer and until the program is closed, it waits for the installer’s termination forever.
The program is Java based and executes two processes: Infor XA Power-Link and javaw. We can create a loop waiting for these two processes, then kill them:
Do {
$status = Get-Process -Name "Infor XA Power-link" -ErrorAction SilentlyContinue
If (!($status)) {
Write-Host 'Waiting for process to start' ;
Start-Sleep -Seconds 2
}
Else { Write-Host 'Process has started' ;
$started = $true
Stop-Process -name "Infor XA Power-Link"
Stop-Process -name javaw
}
}
Until ( $started )
I picked 2 seconds to keep checking the process list and not hammer the CPU. So obviously for this to work, we need to remove the -Wait and -Passthru options from Start-Process, but then how do we check if the program installed OK? We can check if the program executable exists and then return the proper exit code:
if (Test-Path('C:\infor\ERP XA Client\Infor XA Power-Link.exe')) {
$LASTEXITCODE = 0
exit 0
} #end if
else {
$LASTEXITCODE = 1
exit 1
} #end else
Anything other than exit code 0 is usually a failure (MSIs usually return exit code 3010 to indicate a reboot).
Using the exit command with a specific number seems to pass the exit code properly back to SCCM. Based on my Google-fu: it’s then best to wrap your PowerShell script in a batch file and then fire that from SCCM to get the exit code of any non-native command ran from within a PowerShell script: