Vista’s Last Mile
By Ina Fried
Staff Writer, CNET News.com
Published: November 8, 2006 4:00 AM PST
REDMOND, Wash.–The last stop for Vista is a windowless conference room in
Building 26, on Microsoft’s sprawling campus in the Seattle suburbs.
Each day, members of the Windows team gather inside this “shiproom” to go
over the bugs that remain, and to debate which of these can still be fixed
in the days left until the product is declared finished, a milestone that is
expected any time now.
The intense “end game,” as these final weeks are known, is a well-worn
tradition inside the shiproom, which is on the third floor of the Windows
development building. The small room, with its dated, dark wood conference
table has been the war room for every Windows release since Windows 2000.
On the wall are knick-knacks from past projects, as well as clocks showing
the minutes ticking away in a dozen time zones. The clocks serve as a
reminder that Microsoft has a deadline to meet. The company has scheduled a
November 30 press event in New York to announce the availability to
businesses of Windows Vista, while computer makers need to get the final
code in order to finish their testing and get Vista on PCs in time for a
broad launch in January.
The once-daily shiproom meetings have become twice-a-day events as the
product has neared completion. Projected onto a screen is a list of
unresolved issues that need to be addressed before Vista can leave. There
were about five dozen such issues at a meeting last Wednesday morning.
Sven Hallauer, who heads up the process, moved quickly through the list as
about 40 programmers, nearly all with a laptop in tow, worked to keep up. At
each sticking-point, the person responsible for tracking the issue gave a
one-sentence report on where things were.
In one case, there was a bug in the Slovenian release of Vista. It was
quickly tabled as not pressing, given that Slovenian is not in the first or
even second wave of localized versions of Vista. Other reports came in–this
software program has a hitch, this particular laptop has trouble waking up
Some of the glitches were already known. Many were things that have already
been fixed, and a few were too new and need investigating. None appeared to
be a show-stopper.
Hallauer had predicted that the morning’s meeting would be fairly
short–maybe a half-hour. After 20 minutes, the group decided that things
seemed pretty good. Perhaps they wouldn’t need to revise the code again.
At the afternoon meeting, though, the team was forced to revisit that
decision. It turned out that there was an issue within Vista’s new
diagnostics: if a piece of software failed to install properly, the system
would nonetheless get a report that it had been successful. Hallauer and
team decided to spin one more build.
It’s all part of the process. Several times, Hallauer and others have
thought that they had the final version of Vista done, only to find
something that meant the team had to put in another fix.
Two weeks ago, Microsoft thought it had something that promised to be the
final version. But within a couple of days, two new glitches had surfaced.
The issues were arcane, but significant enough. In one case, there was a
potential problem with burning DVDs. If a Vista machine attempted to burn
information to a blank DVD directly from a network drive, there was a chance
that data could be lost, if the network was slow. The other problem had to
do with offline folders: Under certain circumstances, applications weren’t
being notified if the cache was full.
“That could end up with users losing their data or having a really bad
experience,” Hallauer said.
While it seems natural to go ahead and fix such bugs, changing the code at
this point is a big deal. There isn’t time for the full regression testing,
which investigates whether a fix in one area has some hidden impact
somewhere else in the system. Instead, teams must create solutions that only
touch a part of the code and count on their ability to not break something
And not everyone agrees which things need to be tackled. The battles inside
the shiproom can get testy sometimes. These days, there are certainly folks
who feel Vista is ready to send on its way. Others keep lobbying for
particular fixes, including some requests made late last week.
Hallauer said he doesn’t see his job as just saying “no”–but at this point,
it is certainly about only saying “yes” to the right things. “Through most
of the product cycle, the teams are fairly independent,” he said. “Now that
we are at the end of the release cycle, it is more (about) taking stronger
Sharks and limpets
While Vista is not glitch-free, the product is finally coming together. When
Microsoft does find a bug, it gets classified into one of two categories.
One is “sharks”–bugs that everyone agrees need to be fixed before the
product ships. And then there are “limpets,” which are issues that can be
fixed, but where the need is less critical. In those cases, the fixes are
developed, but don’t get implemented unless a shark comes along that they
can use to float into the code.
Retiring Windows chief Jim Allchin doesn’t like the shark and limpet
analogy. To him, nearly every bug is a shark worth fixing.
“(If) there’s a fix, I want to put it in,” Allchin said. “It should be clear
that date means not much to me, that quality is much more important.”
But Allchin is finding plenty of resistance these days. Microsoft is under a
fair bit of pressure to get Vista out the door.
The latest shark, though, means that he can get in one of the changes he
wanted. For months, the company has been struggling with an issue in the
Vista set-up process. As the operating system was loading, the screen would
appear to freeze up, with no indication that the installation was still
progressing–although it was.
Developers put that problem right. But as a result, a dialog box that asked
users to identify the type of network they have was popping up twice.
To Hallauer, it was an issue that might or might not have justified a new
build. Allchin was convinced it did.
“When I heard about it, I thought, there’s no way…(We’ve got) to fix
this,” Allchin said.
The unrelated software-installing problem let Allchin win the day.
end their days putting the latest builds through their paces.
Until recently, Microsoft has issued a new internal release of Vista every
day. That’s a grueling process. Typically, its servers start cranking away
at the raw code around 7 p.m., compiling it through the night, with the goal
of releasing the new build by early afternoon the next day.
Down the hall from shiproom, Windows unit employees can pick up the latest
builds. About 500 people pick up a DVD with new code in person each day,
with many more getting the code over the network, and some even bringing
their home machines into the office.
That list includes rank-and-file Windows employees, as well as some of the
company’s top brass. Allchin and his technical assistant, for example, are
still trying to find bugs that the servers and development teams have
“I’m doing video calls with my mom in Boston,” Allchin said. “I’m doing
remote assistance to jump into a machine, and then I ‘remote desktop’ from
that machine to another machine.”
Elsewhere, Allchin is testing a multimonitor set-up with four displays,
including some in portrait mode. Paul Donnelly, who manages part of
Microsoft’s Vista test operation, has been doing the same thing for some
time. As the finalization deadline has neared, he has added more systems to
his office. As of last week, he had nine machines crammed into his office.
He is among those who nearly always picks up the daily build.
“Pretty much every day since at least May 2005, I think I have installed on
some machine,” he said.
Donnelly, who tinkers with old cars and classic pinball machines in his
spare time, said that he tries to do the opposite of what an IT manager
He changes all the default settings, for instance. And instead of testing a
clean installation on a new machine, he’ll try to upgrade an older model.
“You find bugs,” he said, “You absolutely find bugs that way.” Luckily, he
said, it is getting harder and harder to find issues that aren’t already on
the radar screen and being addressed.
“We’re on watch right now…keeping an eye on things to make sure that we
haven’t missed anything,” he said. “I haven’t had any ‘heart attack’ issues
arrive in the last few months.”
But Vista’s fortitude does not depend solely on the watchful eyes of Windows
veterans like Allchin and Donnelly.
With each day’s build, Microsoft is running a battery of automated tests
against around 1,000 of the leading software programs. It has written
750,000 lines of code just to create the test patterns, which take 355
servers the better part of the day to run.
“Our job is to try and break the apps and find the bugs,” said Mike Kirby,
who runs the automated test lab. These days, though, the team is just hoping
that each day’s build doesn’t bring up any new bugs.
In another building, individual software and device makers have their own
private offices, where they can work on their own Vista-related issues. One
of the key areas for Microsoft, beyond finishing its code, is getting
hardware and software makers to get their products ready for the new
operating system. To make that as attractive as possible, it has created a
building on its campus just for them: the Platform Adoption Center, better
known inside the company as the “high-touch” lab. The building, one of the
hippest on Microsoft’s largely bland campus, offers an inviting atmosphere
with private offices, a lounge with a Xbox 360 game console and plenty of
“We try to keep them well-fed and well-caffeinated,” said Dave Wascha, who
helps lead Microsoft’s effort to make sure other software makers have their
products ready to go when Vista ships.
The companies that come also get their own rooms that lock with a code
combination that only they know. They can use PCs from Microsoft, or bring
their own machines. Either way, the computers can connect directly to the
Internet without going through Microsoft’s network.
“Essentially, this is their office,” Wascha said.
The center has been home to 16,000 people since 2004, and is booked solid
every week. It has been home to Microsoft’s traditional partners as well as
some of its fiercest rivals, many of whom did not want to be named.
One rival that has been public about the hand it received from Microsoft is
the Mozilla Foundation, creator of the Firefox browser. In August, the
open-source software maker an offer of help from Microsoft.
Another rival that credits Microsoft for helping it get Vista-ready is AOL.
The Internet services provider went through the Redmond lab in July, while
Microsoft engineers traveled to AOL’s campus in Dulles, Va., in August and
“We worked through a ton of issues,” said Julie McCool, the AOL vice
president who manages the team that handled the Vista work. One of the many
efforts the two companies worked together on was coming up with a way to let
customers get a Vista-friendly version of AOL’s software when they stick an
older CD into a new PC. In the end, the companies figured out a way to alert
people that the CD they pulled off a two-year-old magazine doesn’t have the
Vista version and to get that software from the Web instead.
McCool said that AOL has continued to meet weekly with Microsoft. Initially,
the company had plenty of bugs it was working through, but in recent weeks
it has been smooth sailing. “I don’t think we had any big surprises in the
past week,” McCool said.
Eating their own dogfood
Microsoft’s own work force is a key arbiter of whether a release is ready to
go out the door. About 60,000 machines inside Microsoft are running Vista as
part of the company’s “dogfooding” process.
CIO Ron Markezich signed off last week, saying that Vista had met his
goals–a step that has to happen before a product can leave Redmond.
“It’s totally ready to go,” Markezich said.
Microsoft is trying to do a better job with the final testing of Vista than
it has in past versions of Windows. “We have to learn from ourselves,”
Donnelly said. “We don’t have the ability to go to somebody bigger than us
and go, ‘What were the problems with your last release?'”
Donnelly, who has been at the company since Windows NT 3.5, said he
remembers an early NT release over a Labor Day that was particularly hairy.
“I just remember the pizza boxes stacking up in the kitchenettes,” he said.
There’s urgency, but no panic this time, he said: “You just don’t see people
running around like crazy.”