To err is human…

Back in August of 1998 on a Friday night in a little Best Buy store I got my first work order. It was to install a ZIP drive into an IBM Aptiva. Having played with computers as a hobby I thought this was going to be a snap. I attempt to open the case and no go. I kept on pulling up and forward on the case and it would just not go! I headed to the Internet and found out that there was a pull release switch under the front bezel. With the case off I then found the drive bay being covered by a metal blank. Taking a pair of pliers I yanked and twisted that piece of thin metal to a certain point, then I started to use my fingers. You guessed it: slice of a finger! I was bleeding all over the customer’s computer! By this time it was near closing time and I was very frustrated at this point. I finally installed the ZIP drive, tested with a ZIP disk and threw it back together (wiping all the blood off of it of course).

A few months later I had become good friends with my boss and he told me he thought I was a idiot at first. “Why?”, I asked. “That first installation on your first day you did was really bad. You forgot to remove the front cover on the drive bay where the ZIP drive was sitting and you disconnected the floppy drive in the process! I thought you were a complete idiot.” Mea culpa! I had rushed the job: testing the ZIP drive without the front bezel on and then just slapping it together and throwing it back on the shelf. Thank goodness my career was not judged on just that one day. All be told, I’ve cracked motherboards, destroyed data and even blew up a NT 4.0 server acting as a print router (thankfully not on the same day)!

All these incidents are great because I learned from them. Fixing computers is almost like a game of chess: everyone understands all the basic rules of chess, but not everyone is a grandmaster. The more chess games you play the better you become. You might favor one opening and your opponent another. You might have all the chess knowledge in the world, but if you act impulsively you will likely lose to a less experienced player. You will make mistakes whether you like it or not.

Case in point: when faced with a crashing system in the past I would usually just rebuild a system from scratch. This might be OK in the consumer arena where people back up their data religiously to CD-Rs and other removable devices (can you feel the sarcasm?). However, that wouldn’t fly on the VP’s computer who stores all of his personal documents and kids pictures on his laptop. Pretend the computer is your opponent. You don’t want to be checkmated or stalemated, you want to win! How do you do it?

First, ask the right questions to the user:

* When did this start happening?
* Has anything changed within the last X days?
* How often do you do this function?

Then, start the troubleshooting:

* Check the event logs (assuming a NT system). Log anything suspicious.
* If you get an error message log that.
* Check startup entries using Autoruns and the processes running by using Process Explorer.

Now, research the problem:

* Input the error code or problem into Google. You’ll be surprised the wealth of information out there!

* Check Microsoft’s Knowledgebase. People all over the world contact Microsoft for tech support and you will surprised at the amount of knowledge there.

* If you still cannot figure the problem out post what you are experiencing on Experts Exchange or USENET.

You can post on USENET via Google Groups. USENET is a world wide messaging state dating back to the early 90’s. Just do a general search on what you are having a problem with (e.g. input “Outlook 98” if you were having problems with Outlook 98) to pinpoint the news group (message board) that handles such problems and post your problem there. For example: I like to frequent the news group microsoft.public.win2000.setup_deployment. This news group focuses specifically on deployment problems with Windows 2000. Not Windows 9x, not NT 4.0 or XP: just Windows 2000. The people that frequent this news group are usually very familiar with the subject at hand and this is usually the case for many other news groups. There are over 40,000 USENET news groups relating to subjects such as computers, tv sitcoms, bands, politics and anything else you can dream of. Google Groups has archived USENET messages all the way back to the early 90’s, so this is another great place to search for conversations on specific error messages and problems.

Finally, implement the solution. Sometimes people will give you 6 or 7 possible solutions. This might require more research on your part. If the problem is happening on Windows 2000 and someone gives you a fix for Windows NT 4.0 you need to realize that will probably not fix your specific problem.

How do you become a better PC Technician? Here’s what I think:

1. Experience: the more you play with computers the better you get.

2. Willingness to learn: if you think you know it all you will fail miserably.

3. Thirst for knowledge: you might think this is the same as #2, but it is not. I may be willing to learn something new, but not ambitious to go out and learn about other new things.

4. Know your limitations: Clint Eastwood said it so it must be true. Seriously, some problems go past your knowledge. In those cases you have to park your ego and go ask for someone else’s help.

– Soli Deo Gloria

Windows Vista

Microsoft recently announced the next version of Windows will be called Windows Vista. August 24th will mark the 10th anniversary of the launch of Windows 95. I remember that era very well. I was running Windows 3.1 on DOS 6.22 on a 486SX/33. When my mom went shopping at the supermarket I would always go to the magazine rack. I would read articles about “Chicago”, the codename for Windows 95 and check out screenshots of builds in progress. It was all very exciting to computer users because it was the first 32-bit Windows version for consumers. I remember reading about long filenames, the start menu, built in WINSOCK and plug and play support. Today, we don’t even think or appreciate these features. Before Windows 95 you had to name everything in the 8+3 format. Imagine MP3s being limited to the 8+3 format: madonna1.mp3, madonna2.mp3, etc. If you wanted to get on the Internet with Windows 3.1 you had to get Trumpet Winsock. By the way: did anyone ever register Trumpet Winsock? It was shareware, but no one ever seemed to care about that.

My favorite operating system of all time is Windows 2000 Professional. I remember ordering a Hands On Training (HOT) kit for $129 that included Beta 3 copies of Windows 2000 Pro, Server and Advanced server, a training CD, a sales CD and a free t-shirt. The kit also included final build, Not For Resale (NFR) copies of Pro, Server and Advanced Server, with the server versions having 10 CALs each. I just loved the shadow under the mouse: it made the operating system seem 3D like! Now we have Windows XP: product activation, skins and bloat. OK, OK, I admit it: I have a Macintosh theme on my Windows XP laptop! I look at Windows XP as a Windows 2000 add-on. It took 3 years and over 5000 developers to create Windows 2000 and it’s still going strong.

I can’t wait to try out a late beta of Windows Vista. I heard they are getting rid of the old DOS like underpinnings and are replacing the setup routine with a Windows PE like environment. You can keep up on the development of Windows Vista (a.k.a. Longhorn) by visiting Paul Thorrutt’s web site. Paul also has a nice, text based newsletter called WinInfo that he sends out packed with the latest geek news. I highly recommend it. Best of all it’s FREE!

Here are two videos on the daily workings of making a Windows build:

Windows XP Daily Build Cycle

Windows 2000 Daily Build Cycle

Voxware Meta Sound Codec (needed to listen to Windows 2000 video)

– Soli Deo Gloria

BSOD Land

One Friday morning several months ago I encountered a very perplexing problem. A bunch of tickets called into the Help Desk about Windows 2000 machines BSODing. BSOD = Blue Screen of Death: a techie’s favorite (or not so favorite) term to describe a dead Windows machine. We determined that a patch pack pushed out the night before was likely the cause. However, the BSOD only seemed to happen on reboot and only on certain Omnitech 3200 machines and not all of them, nor all the time! Upon rebooting some of these machines several times they worked fine until later on when they were rebooted again. I even took one of them out of commission that was crashing, turned off automatic rebooting, wrote a reboot script and left the machine to reboot for 24 hours continuously. Not once did it crash!

The event log showed nothing, a Google search on the error came up with nothing and the crash could not be consisently produced on demand. The BSOD itself was very useless: an INVALID_PAGE_FAULT in NTOSKRNL. NTOSKRNL, as you know, is the heart of the Windows 2000 operating system. It was painfully obvious that it was not the cause of the crashes. We would either rebuild the machines which would fix the problem and then reapply the security patches or just tell the user to keep rebooting the system (incidentally, booting the PCs into safe mode always worked).

After several weeks of rebuilding machines we grew very tired of the situation. No one seemed to have an answer and someone was even called in on a Saturday to reboot a PC with this problem! Something had to be done! Having trial copy of Winternals Administrator Pak 5 I had access to Crash Analyser. What this tool does is it uses Microsoft’s own debugging tools to decipher the dump file and then it in turn deciphers the Microsoft debug summary to make a best guess as to what caused the crash. I grabbed the C:\WINNT\MEMORY.DMP file after turning on crash dumping on one of the machines causing an issue. Upon running the utility I found out that idechdr.sys was the file causing the crash! So, after discovering this, I went to each machine called in and uninstalled the IDE drivers in safe mode and then let Windows 2000 redetect them on the next reboot. This solution finally worked!

This does not explain however what caused this in the first place. I had an Omnitech 3200 under my desk as my work PC and never once did it fail on me with the BSOD. It’s very likely that your company won’t go out and purchase the Administrator’s Pak based on cost, so you can read Dirk Smith’s excellent article entitled How to solve Windows system crashes in minutes which uses only the freeware debugging tools directly from Microsoft.

– Soli Deo Gloria

Universal Network Boot Disk

What better way is there to compliment your new universal Ghost image and impress your boss then a universal network boot disk? A PXE server? OK, so maybe it’s not the greatest thing in the world, but it sure beats carrying around lots of floppies with you. Not to digress, but why is it that we are still using floppies? The IBM PC was invented in 1981 and here it is 2005 and just last year Microsoft sent out a white paper to companies pleading with them to support booting from USB Flash Devices (UFD). Come on guys, wake up! UFDs are bigger, more reliable and a lot more fun than floppies. Now that Windows PE 2005 supports booting from UFDs we need to pressure these companies to support booting from them.

OK, back to the network boot disk. The one I’m talking about is Bart’s Network Boot Disk. One word: freeware. Yes, freeware! I love Bart! He’s also the one that makes PE Builder. Check it out: it’s very cool stuff! Download the full BFD package and extract it to a directory. Now execute “bfd msnet A:” from the command line. This will make a self-booting network boot disk using the MS-DOS 7.1 files (an interesting side note here is that Bart has had legal problems with Microsoft and PE Builder. It’s a mystery then why he would bundle the MS-DOS 7.1 files directly into this package even though these files are available in particularly every corner of the Internet). Congrats, you just made a universal network boot disk!

Listed on the same page are driver CAB files for practically every NIC ever made! Now here’s the slick part: you can just drop in the CAB files you need into A:libndis and the disk will rebuild itself accordingly! How cool is that? You’ll notice that the drivers haven’t been updated for at least 2 years and may not include support for the latest NICs. That was the problem when we got in motherboards supporting the Intel 915 chipset. The network boot disk would not find the NIC and even when we manually picked it off the menu the driver would not work (it was a variant of the Intel Pro 100VE). Let’s look into how the boot disk works. Every PCI device has a unique hexadecimal id. Let’s boot from the network boot disk you created and run pciscan -v:

We can clearly see that there are vendor ids and device ids. Based on these two pieces of information the boot disk can determine what driver to load. If you download PCISCAN from his web site it gives a much better explaination then I give. PCISCAN gets its information from nic.map. Let’s look for this vendor id in this file:

ret=”SMCPWR2.COM”
ven=10B8 “SMC”
dev=0005 “SMC9432TX EtherPower II 10/100”
ven=1011 “DEC”
dev=0002 “DC21040”
0014 “DC21041”
0009 “DC21140”
0019 “DC21143”

There is it! I booted the disk using Virtual PC 5 and this is the type of NIC it emulates. So 1011=DEC and 0009 = model DC21140. Let’s take a look inside one of these CAB files sitting in A:libndis:

  • e100b.dos
  • e100b.ini
  • ndis.pci
  • ndis.txt

If we look into ndis.pci for the above driver this is what it looks like:

ret=”E100B”
ven=8086 “Intel”
dev=1002 “PRO 100 Mobile Adapters”
1031 “PRO/100 VE Network Connection”
1032 “PRO/100 VE Network Connection”
1035 “PRO/100 VM Network Connection”

So in the case of us getting the new computers in with the Intel 915 chipset we just had to get a new .DOS file and update the .PCI file with the correct hexadecimal id. You can get the latest .DOS file easily by visiting Intel’s web site. They have DOS drivers for all their NICs. We can use PCISCAN or PCI32 to find the hexadecimal id of the NIC and then add that with the appropriate description. Finally, we have to repackage them back up into a CAB file. You can download makev3.zip from here to do that.

The other advantages to this disk are that it randomizes the NETBIOS name so you can use it in multiple computers at the same time. You can also setup a profile which will save the work group or domain name so you don’t have to keep entering it each time. I editted the disk so that it just boots without sitting at the menu asking if you want emm386 support or not. The other thing I changed is the prompt for the second password. This can be fixed by editting the msnet.bat in msnet.cab. Make the following changes:

—————-in this section——————-
:_logon
echo MSNET: Network logon as “%p_user%”
net logon %p_user% %w_passwd% /yes /savepw:no
———————end————————-

change to:

—————-change this——————-
net logon %p_user% %w_passwd% /domain:%logondomain% /yes /savepw:no
———————end————————-

—————-in this section——————-
echo MSNET: Starting network services
net start workstation
if errorlevel 1 goto _abort
———————end————————

add:

——————add this———————–
echo %w_passwd%> password.txt
———————end————————-

That gets rid of the second prompt for the password. Again, note that you will have to use makev3 to repackage the files extracted from msnet.cab back into a CAB file. Note that since this is a DOS boot disk you’ll also need WINS support on your network to make it work. DNS alone will not cut it! The bad news is the new 64-bit processors won’t do 16-bit applications or true DOS anymore! Hopefully Windows PE will be well supported for booting purposes by the time 64-bit processing becomes popular. You can port this boot disk to a UFD as well, making boot time in 12 seconds or less so when you have inpatient techs like DAVE it will go faster for them!

– Soli Deo Gloria

Making a Windows 2000/XP Hardware Independent Ghost Image

About a year and a half ago I was a apart of a 3 member team whose mission it was to create a standard Windows 2000 image for our desktops. The company was currently running Windows 98 SE as a desktop standard. There was a base image and an image for each department and it was messy, real messy! We sat down and discussed what services to leave enabled or disabled, what to include in the default user profile, etc. on the new image. When we were done the image was a beautiful thing. Even though we wrote documentation on everything we did there are always things that get missed. We had a total of 12 Ghost images: 4 for PCs, 6 for laptops (3 with wireless drivers and 3 without) and 2 specialty Ghost images. In January 2005 we merged with another company. This company had about 8 Ghost images bringing the total number of company Ghost images to 20! Under the new management they wanted the patch levels on each image maintained every month! Before that, we were just using SMS 2003 to maintain the patch level of the workstation. You would image a box and then SMS 2003 would push down the new patches. Try calculating the time it takes to open an image, run the patches on it, verify the patch installation with MBSA, do application testing on the image to make sure the patches didn’t break anything and finally reseal the image. Now take that calculation times 20. You could easily justify a full time position just for this task!

Not wanting to update 12 images each month I decided to make a hardware independent image for Windows 2000. When I was done, I got it down to 4 images: 1 base and 3 specialty images! Since the base image was used 95% of time for new machines I could take time to get the other ones updated. It took me 3 solid weeks hunting all over the Internet and ghosting countless machines to get it all working. The documentation I wrote for doing this is on my web page here. Just last month I started a new job at another company and again saw images for each individual piece of hardware. Time to test my skills once again!

The desktop platform at the new company is Windows XP Professional. Here’s one “problem” I found with my instructions. Under the section Finding the IDE Driver Used To Setup [SysprepMassStorage] I stated to search for 82801DB in the INF files you extract from the Intel Chipset setup. Well, I was working on a Dell Optiplex 280 and the image was giving a STOP 0x7B message at startup. I had included support for the IDE chipset driver, but it still wouldn’t work. I went back to the INF file and look what I found:

PCIVEN_8086&DEV_2651.DeviceDesc=”Intel(R) 82801FB Ultra ATA Storage Controllers – 2651″
PCIVEN_8086&DEV_2652.DeviceDesc=”Intel(R) 82801FB Ultra ATA Storage Controllers – 2652″
PCIVEN_8086&DEV_2653.DeviceDesc=”Intel(R) 82801FBM Ultra ATA Storage Controllers – 2653″
PCIVEN_8086&DEV_266F.DeviceDesc=”Intel(R) 82801FB/FBM Ultra ATA Storage Controllers – 266F”

Multiple versions of the 82801FB! Obviously, I had picked the wrong one, but which was the right one? We can solve this little problem by using PCI32. This is a freeware program made by Craig Hart that has 15,000+ PCI devices in its database. I’ll use my home PC as an example:

Vendor 8086h Intel Corporation
Device 24CBh 82801DB/DBL (ICH4/ICH4-L) UltraATA/100 EIDE Controller
Command 0007h (Memory Access, BusMaster, )
Status 0280h (Medium Timing, )
Revision 02h, Header Type 00h, Bus Latency 00h
Self test 00h (Self test not supported)
PCI Class Storage, type IDE
PCI EIDE Controller Features :
BusMaster EIDE is supported
Primary Channel is at I/O Port 01F0h and IRQ 14
Secondary Channel is at I/O Port 0170h and IRQ 15
Subsystem ID 80891043h Unknown
Subsystem Vendor 1043h ASUSTeK Computer Inc
Address 0 is an I/O Port : 00000000h
Address 1 is an I/O Port : 00000000h
Address 2 is an I/O Port : 00000000h
Address 3 is an I/O Port : 00000000h
Address 4 is an I/O Port : 0000F000h
Address 5 is a Memory Address (anywhere in 0-4Gb) : FEBFB400h
System IRQ 9, INT# A

If you can read the second line it says:

Device 24CBh 82801DB/DBL (ICH4/ICH4-L) UltraATA/100 EIDE Controller

That’s what we want. In the case of the Dell Optiplex 280 it was the 266F one: go figure! Incidently, we can use PCI32 for much cooler things like updating a universal network boot disk! Or finding that pesky model number without cracking the case. In the case of the Dell Optiplex 620 I found that it had 2 different versions of the same chipset on the same board! So make sure you include support for what ever is in the computer.

The other problem I ran into again was the HAL issue. I commented out the following line in my sysprep.inf:

UpdateUPHAL=ACPIPIC_UP,C:WINNTInfHal.inf

Upon trying the image on a Dell Latitude D620 I was greeted with a message from Windows stating there was a hardware problem and did I want to start Windows. If I answered yes it would BSOD and then reboot before I could even read the message! If you hit F5 right after you pick “Start Windows XP” it will give you an option to “Disable Automatic Restarting on System Failure”. Way to go Microsoft! I saw it was a stop 0x7B message. I checked the IDE setup in [SysprepMassStorage] section and saw that was setup correctly. Having gone through this once before I guessed it was a HAL issue and I was right. By uncommenting the above line and changing WINNT to WINDOWS the image came right up. This line forces the HAL from a Uniprocessor HAL to APCI HAL. What is the difference between these two HALs? I have no idea, but functionally they appear to be the same! I have yet to find good explanation: does anyone have one?

Making a hardware independent image is big business. The original makers of Ghost made a program called the Universal Imaging Utility. They want $19 per workstation for what we did here. Granted, it’s a drop and go solution, but with enough patience you can get a very simliar result with my instructions. In fact, if you want your image to support gobs of hardware you can head over to the Device Drivers subforum over at MSFN and pick up their Driver Packs, which include support for virtually everything in existence. Note that the Driver Packs are for Windows XP only.

Incidentally, Microsoft claims to solve all this in Windows Vista by using a program called Ximage. Among the cool features it lists:

  • This WIM image format is hardware-agnostic, meaning that you need only one image to address many different hardware configurations.
  • The WIM image format allows you to service an image offline. You can add or delete certain operating system components, patches, and drivers without creating a new image.
  • The WIM image format allows for non-destructive deployment. This means that you can leave data on the volume to which you apply the image because the application of the image does not erase the disk’s existing contents.

Time will tell if Ximage makes it to the final build of Vista! Let’s make sure that it does.

– Soli Deo Gloria

Congratulations Matty!

This post is dedicated to Matt: my co-worker at my last job. He just found another job and I am very happy for him. Matt is very technical and through in solving computer problems. He single handedly rebuilt a NT 4.0 box that died in the cancer department and brought up the cancer software on a Windows 2000 machine with little to no documentation or media. He was also the administrator of our form routing server and tape backup man. He was also great in getting us to Windows 2000 from Windows 98 SE as a desktop standard for various departments. Hey Matt, you remember the Decision Support upgrade? Wow, what a pain that was! Oh what fun it is to convert a 10 year old program using scripts programmed by God knows who dumping data into a mainframe. However, Matt did it.

  • Remember “Pizza Thursdays” in the cafe?
  • Hitting the golf ball just right into the 1970’s golf ball returner?
  • Videos from Ebaumsworld?
  • A. Vicious?
  • Formscape?
  • People also calling me Matt and you Adam?
  • Throwing rubber balls at people’s heads?

All the best to you buddy.

– Soli Deo Gloria

The Beginning

Well, not really. I’ve been posting on news groups (USENET) since 1996 and was on FIDONet back in 1995 (remember the BBS days?). I’ve always found it fascinating how the Internet allows people to exchange information. What prompted me to start a blog was Mark Russinovich starting one here. To see what kind of geek I am you can check out my web page.

To start off this blog I’ll describe how I fixed an Excel upgrade problem yesterday. I was assigned a task of upgrading Excel 2000 to Excel 2002 on a PC running Windows XP Professional. Easy, right? Upon installing it and testing it as myself it worked fine. The next day the user called me stating it was crashing. Indeed, it was crashing quite hard. The application seemed fine until you went to File>Open and then it went to never-never land. Since it worked as me this boiled it down to a permissions or NT profile issue. I reved him back to Excel 2000 which surprisingly worked fine. The next day I went back to reinstall Excel 2002 and check things out while the user was at a meeting. Upon checking the local administrator’s group I saw that the user was already an administrator on the machine, thus eliminating a permissions issue.

I then started up Filemon which is a excellent little freeware utility by our friends at Sysinternals. I setup a filter in Filemon to just show me the excel.exe entries. The last file it read was C:\windows\system32\davclnt.dll. I decided for fun to rename this file to see if that would fix the problem. Of course, this action was not really logical….after all, it would have used the same file logged in as me, right? Any ways, I renamed the file and the appeared back within seconds. Drats! Windows File Protection (WFP) rears its ugly head. Introduced in Windows 2000 WFP protects critical Windows system files by looking for changes. If you touch a critical file by renaming, deleting, or overwriting it, Windows copies the “good file” from a secret folder called dllcache. For a while you could disable WFP by setting “SFCDisable” to FFFFFF9D in the registry. However, Microsoft later removed this feature with their service packs for Windows 2000 and XP.

To get around this you can hex edit the sfc_os.dll file. However, this didn’t work for me and is a bit messy. I like using XP Lite for this purpose. There is a trial version available for download that the author states “is yours to keep!“. In the trial version there is an option to turn WFP On, Off or to Disable it. So I can turn it off, do my dirty work and when I reboot, WFP turns itself back on. Well, getting back to davclnt.dll: renaming it did not work. I started to think the problem was in the HKEY_CURRENT_USER (HKCU) part of the registry (logged in as that user of course). So I started up Regmon this time pausing it just before I went to File>Open. I then saw something interesting. Under HKCUNetwork there was a bunch of drive mappings and one of those mappings was pointing to a bogus server! I exported this key out (always export trees before you start deleting them!) and then deleted it. Tada, problem solved!

Now the interesting question is: why didn’t this happen with Excel 2000? Who knows. In this instance deleting the user’s NT profile may have been quicker. However, I’ve only been at this company for a month and therefore I am not well versed with all desktop standards. Besides, the user was gone and his NTUSER.DAT file was 4MB: that’s a lot of information to just throw away!

That’s it for now. Look for an upcoming article on making hardware independent ghost images for Windows 2000 and XP.