BSOD Land

One Friday morning several months ago I encountered a very perplexing problem. A bunch of tickets called into the Help Desk about Windows 2000 machines BSODing. BSOD = Blue Screen of Death: a techie’s favorite (or not so favorite) term to describe a dead Windows machine. We determined that a patch pack pushed out the night before was likely the cause. However, the BSOD only seemed to happen on reboot and only on certain Omnitech 3200 machines and not all of them, nor all the time! Upon rebooting some of these machines several times they worked fine until later on when they were rebooted again. I even took one of them out of commission that was crashing, turned off automatic rebooting, wrote a reboot script and left the machine to reboot for 24 hours continuously. Not once did it crash!

The event log showed nothing, a Google search on the error came up with nothing and the crash could not be consisently produced on demand. The BSOD itself was very useless: an INVALID_PAGE_FAULT in NTOSKRNL. NTOSKRNL, as you know, is the heart of the Windows 2000 operating system. It was painfully obvious that it was not the cause of the crashes. We would either rebuild the machines which would fix the problem and then reapply the security patches or just tell the user to keep rebooting the system (incidentally, booting the PCs into safe mode always worked).

After several weeks of rebuilding machines we grew very tired of the situation. No one seemed to have an answer and someone was even called in on a Saturday to reboot a PC with this problem! Something had to be done! Having trial copy of Winternals Administrator Pak 5 I had access to Crash Analyser. What this tool does is it uses Microsoft’s own debugging tools to decipher the dump file and then it in turn deciphers the Microsoft debug summary to make a best guess as to what caused the crash. I grabbed the C:\WINNT\MEMORY.DMP file after turning on crash dumping on one of the machines causing an issue. Upon running the utility I found out that idechdr.sys was the file causing the crash! So, after discovering this, I went to each machine called in and uninstalled the IDE drivers in safe mode and then let Windows 2000 redetect them on the next reboot. This solution finally worked!

This does not explain however what caused this in the first place. I had an Omnitech 3200 under my desk as my work PC and never once did it fail on me with the BSOD. It’s very likely that your company won’t go out and purchase the Administrator’s Pak based on cost, so you can read Dirk Smith’s excellent article entitled How to solve Windows system crashes in minutes which uses only the freeware debugging tools directly from Microsoft.

– Soli Deo Gloria

Universal Network Boot Disk

What better way is there to compliment your new universal Ghost image and impress your boss then a universal network boot disk? A PXE server? OK, so maybe it’s not the greatest thing in the world, but it sure beats carrying around lots of floppies with you. Not to digress, but why is it that we are still using floppies? The IBM PC was invented in 1981 and here it is 2005 and just last year Microsoft sent out a white paper to companies pleading with them to support booting from USB Flash Devices (UFD). Come on guys, wake up! UFDs are bigger, more reliable and a lot more fun than floppies. Now that Windows PE 2005 supports booting from UFDs we need to pressure these companies to support booting from them.

OK, back to the network boot disk. The one I’m talking about is Bart’s Network Boot Disk. One word: freeware. Yes, freeware! I love Bart! He’s also the one that makes PE Builder. Check it out: it’s very cool stuff! Download the full BFD package and extract it to a directory. Now execute “bfd msnet A:” from the command line. This will make a self-booting network boot disk using the MS-DOS 7.1 files (an interesting side note here is that Bart has had legal problems with Microsoft and PE Builder. It’s a mystery then why he would bundle the MS-DOS 7.1 files directly into this package even though these files are available in particularly every corner of the Internet). Congrats, you just made a universal network boot disk!

Listed on the same page are driver CAB files for practically every NIC ever made! Now here’s the slick part: you can just drop in the CAB files you need into A:libndis and the disk will rebuild itself accordingly! How cool is that? You’ll notice that the drivers haven’t been updated for at least 2 years and may not include support for the latest NICs. That was the problem when we got in motherboards supporting the Intel 915 chipset. The network boot disk would not find the NIC and even when we manually picked it off the menu the driver would not work (it was a variant of the Intel Pro 100VE). Let’s look into how the boot disk works. Every PCI device has a unique hexadecimal id. Let’s boot from the network boot disk you created and run pciscan -v:

We can clearly see that there are vendor ids and device ids. Based on these two pieces of information the boot disk can determine what driver to load. If you download PCISCAN from his web site it gives a much better explaination then I give. PCISCAN gets its information from nic.map. Let’s look for this vendor id in this file:

ret=”SMCPWR2.COM”
ven=10B8 “SMC”
dev=0005 “SMC9432TX EtherPower II 10/100”
ven=1011 “DEC”
dev=0002 “DC21040”
0014 “DC21041”
0009 “DC21140”
0019 “DC21143”

There is it! I booted the disk using Virtual PC 5 and this is the type of NIC it emulates. So 1011=DEC and 0009 = model DC21140. Let’s take a look inside one of these CAB files sitting in A:libndis:

  • e100b.dos
  • e100b.ini
  • ndis.pci
  • ndis.txt

If we look into ndis.pci for the above driver this is what it looks like:

ret=”E100B”
ven=8086 “Intel”
dev=1002 “PRO 100 Mobile Adapters”
1031 “PRO/100 VE Network Connection”
1032 “PRO/100 VE Network Connection”
1035 “PRO/100 VM Network Connection”

So in the case of us getting the new computers in with the Intel 915 chipset we just had to get a new .DOS file and update the .PCI file with the correct hexadecimal id. You can get the latest .DOS file easily by visiting Intel’s web site. They have DOS drivers for all their NICs. We can use PCISCAN or PCI32 to find the hexadecimal id of the NIC and then add that with the appropriate description. Finally, we have to repackage them back up into a CAB file. You can download makev3.zip from here to do that.

The other advantages to this disk are that it randomizes the NETBIOS name so you can use it in multiple computers at the same time. You can also setup a profile which will save the work group or domain name so you don’t have to keep entering it each time. I editted the disk so that it just boots without sitting at the menu asking if you want emm386 support or not. The other thing I changed is the prompt for the second password. This can be fixed by editting the msnet.bat in msnet.cab. Make the following changes:

—————-in this section——————-
:_logon
echo MSNET: Network logon as “%p_user%”
net logon %p_user% %w_passwd% /yes /savepw:no
———————end————————-

change to:

—————-change this——————-
net logon %p_user% %w_passwd% /domain:%logondomain% /yes /savepw:no
———————end————————-

—————-in this section——————-
echo MSNET: Starting network services
net start workstation
if errorlevel 1 goto _abort
———————end————————

add:

——————add this———————–
echo %w_passwd%> password.txt
———————end————————-

That gets rid of the second prompt for the password. Again, note that you will have to use makev3 to repackage the files extracted from msnet.cab back into a CAB file. Note that since this is a DOS boot disk you’ll also need WINS support on your network to make it work. DNS alone will not cut it! The bad news is the new 64-bit processors won’t do 16-bit applications or true DOS anymore! Hopefully Windows PE will be well supported for booting purposes by the time 64-bit processing becomes popular. You can port this boot disk to a UFD as well, making boot time in 12 seconds or less so when you have inpatient techs like DAVE it will go faster for them!

– Soli Deo Gloria

Making a Windows 2000/XP Hardware Independent Ghost Image

About a year and a half ago I was a apart of a 3 member team whose mission it was to create a standard Windows 2000 image for our desktops. The company was currently running Windows 98 SE as a desktop standard. There was a base image and an image for each department and it was messy, real messy! We sat down and discussed what services to leave enabled or disabled, what to include in the default user profile, etc. on the new image. When we were done the image was a beautiful thing. Even though we wrote documentation on everything we did there are always things that get missed. We had a total of 12 Ghost images: 4 for PCs, 6 for laptops (3 with wireless drivers and 3 without) and 2 specialty Ghost images. In January 2005 we merged with another company. This company had about 8 Ghost images bringing the total number of company Ghost images to 20! Under the new management they wanted the patch levels on each image maintained every month! Before that, we were just using SMS 2003 to maintain the patch level of the workstation. You would image a box and then SMS 2003 would push down the new patches. Try calculating the time it takes to open an image, run the patches on it, verify the patch installation with MBSA, do application testing on the image to make sure the patches didn’t break anything and finally reseal the image. Now take that calculation times 20. You could easily justify a full time position just for this task!

Not wanting to update 12 images each month I decided to make a hardware independent image for Windows 2000. When I was done, I got it down to 4 images: 1 base and 3 specialty images! Since the base image was used 95% of time for new machines I could take time to get the other ones updated. It took me 3 solid weeks hunting all over the Internet and ghosting countless machines to get it all working. The documentation I wrote for doing this is on my web page here. Just last month I started a new job at another company and again saw images for each individual piece of hardware. Time to test my skills once again!

The desktop platform at the new company is Windows XP Professional. Here’s one “problem” I found with my instructions. Under the section Finding the IDE Driver Used To Setup [SysprepMassStorage] I stated to search for 82801DB in the INF files you extract from the Intel Chipset setup. Well, I was working on a Dell Optiplex 280 and the image was giving a STOP 0x7B message at startup. I had included support for the IDE chipset driver, but it still wouldn’t work. I went back to the INF file and look what I found:

PCIVEN_8086&DEV_2651.DeviceDesc=”Intel(R) 82801FB Ultra ATA Storage Controllers – 2651″
PCIVEN_8086&DEV_2652.DeviceDesc=”Intel(R) 82801FB Ultra ATA Storage Controllers – 2652″
PCIVEN_8086&DEV_2653.DeviceDesc=”Intel(R) 82801FBM Ultra ATA Storage Controllers – 2653″
PCIVEN_8086&DEV_266F.DeviceDesc=”Intel(R) 82801FB/FBM Ultra ATA Storage Controllers – 266F”

Multiple versions of the 82801FB! Obviously, I had picked the wrong one, but which was the right one? We can solve this little problem by using PCI32. This is a freeware program made by Craig Hart that has 15,000+ PCI devices in its database. I’ll use my home PC as an example:

Vendor 8086h Intel Corporation
Device 24CBh 82801DB/DBL (ICH4/ICH4-L) UltraATA/100 EIDE Controller
Command 0007h (Memory Access, BusMaster, )
Status 0280h (Medium Timing, )
Revision 02h, Header Type 00h, Bus Latency 00h
Self test 00h (Self test not supported)
PCI Class Storage, type IDE
PCI EIDE Controller Features :
BusMaster EIDE is supported
Primary Channel is at I/O Port 01F0h and IRQ 14
Secondary Channel is at I/O Port 0170h and IRQ 15
Subsystem ID 80891043h Unknown
Subsystem Vendor 1043h ASUSTeK Computer Inc
Address 0 is an I/O Port : 00000000h
Address 1 is an I/O Port : 00000000h
Address 2 is an I/O Port : 00000000h
Address 3 is an I/O Port : 00000000h
Address 4 is an I/O Port : 0000F000h
Address 5 is a Memory Address (anywhere in 0-4Gb) : FEBFB400h
System IRQ 9, INT# A

If you can read the second line it says:

Device 24CBh 82801DB/DBL (ICH4/ICH4-L) UltraATA/100 EIDE Controller

That’s what we want. In the case of the Dell Optiplex 280 it was the 266F one: go figure! Incidently, we can use PCI32 for much cooler things like updating a universal network boot disk! Or finding that pesky model number without cracking the case. In the case of the Dell Optiplex 620 I found that it had 2 different versions of the same chipset on the same board! So make sure you include support for what ever is in the computer.

The other problem I ran into again was the HAL issue. I commented out the following line in my sysprep.inf:

UpdateUPHAL=ACPIPIC_UP,C:WINNTInfHal.inf

Upon trying the image on a Dell Latitude D620 I was greeted with a message from Windows stating there was a hardware problem and did I want to start Windows. If I answered yes it would BSOD and then reboot before I could even read the message! If you hit F5 right after you pick “Start Windows XP” it will give you an option to “Disable Automatic Restarting on System Failure”. Way to go Microsoft! I saw it was a stop 0x7B message. I checked the IDE setup in [SysprepMassStorage] section and saw that was setup correctly. Having gone through this once before I guessed it was a HAL issue and I was right. By uncommenting the above line and changing WINNT to WINDOWS the image came right up. This line forces the HAL from a Uniprocessor HAL to APCI HAL. What is the difference between these two HALs? I have no idea, but functionally they appear to be the same! I have yet to find good explanation: does anyone have one?

Making a hardware independent image is big business. The original makers of Ghost made a program called the Universal Imaging Utility. They want $19 per workstation for what we did here. Granted, it’s a drop and go solution, but with enough patience you can get a very simliar result with my instructions. In fact, if you want your image to support gobs of hardware you can head over to the Device Drivers subforum over at MSFN and pick up their Driver Packs, which include support for virtually everything in existence. Note that the Driver Packs are for Windows XP only.

Incidentally, Microsoft claims to solve all this in Windows Vista by using a program called Ximage. Among the cool features it lists:

  • This WIM image format is hardware-agnostic, meaning that you need only one image to address many different hardware configurations.
  • The WIM image format allows you to service an image offline. You can add or delete certain operating system components, patches, and drivers without creating a new image.
  • The WIM image format allows for non-destructive deployment. This means that you can leave data on the volume to which you apply the image because the application of the image does not erase the disk’s existing contents.

Time will tell if Ximage makes it to the final build of Vista! Let’s make sure that it does.

– Soli Deo Gloria