Software Bugged

From the Vault Dept.: Twenty years ago, I wrote for a number of computer magazines. Good money, but it robbed me of time and imaginative brainpower. Here's a 3,000 word piece I wrote for PC/Computing magazine. Don't get bugged if you can't make it to the end.

                                                                                       

Things always seem to go wrong when it’s hot. In this case, some of the heat came from the large vacuum tubes powering the new Mark II computer, but most of it came from a brutally warm day.

The year was 1945, and Grace Hopper, a Navy lieutenant assigned to the Bureau of Ordnance Computation Project at Harvard University, was helping build this new computer to meet the nonstop wartime demand for gunnery calculations.

There’s a special sense of confidence you get when you know you’re doing everything right. Great athletes know this; it can also come from writing that perfect piece of programming code. When the machine came to a stop that hot day, Hopper knew it wasn’t her fault. She suspected something in the machine. She and her assistants inspected the monster, peering into its viscera of wire and glass, and found the problem: a failed relay. Looking closer, they found the cause: a two-inch-long moth. The carcass was tweezed from the machine and scotch-taped into the logbook, and Hopper quipped that worktime had been lost because they were “debugging” the computer.

That incident gave the computer industry a term that now describes a figurative problem so perfectly that it’s easy to forget to credit Hopper and the hapless insect for it. Software bugs are amazingly lifelike. They seem to creep from an unseen lair, causing anything from a minor headache to a major disaster. The bugs we’re considering aren’t viruses: There’s nothing deliberately malicious about these critters. They’re simply the unexpected conflicts that cause an application to take an unforeseen turn.

As any online service user will attest, buggy software can drive an otherwise placid computerist into passionate outbursts. “I don’t see why [fill in the software company] should force us to test their program!” is a common cry, its intensity often inversely proportionate to the problem just suffered.

“It’s a difficult issue,” says Don LaVange, WordPerfect Corporation’s director of testing. “Two things are true: No software is ever bug-free. And nobody ever wants to find a bug. People don’t realize how much software they have. DOS, the BIOS in their machines. All of it has bugs. The issue really has to do with bugs that do unpleasant things.”

Buggy programs can be annoying: In Version 5.0, WordPerfect presented a slew of new features and unnoticed bugs that swamped the company’s usually efficient support staff. They also can be destructive: A flaw in Version 5.0 of the Norton Utilities caused one of its modules to inadvertently give some users’ hard disks a destructive reformat.

Some bugs, however, exist only in the mind of the user. “One of the biggest misunderstandings in this industry is the definition of the term ‘bug,’” says Microsoft’s Monica Harrington. Now in the company’s public relations group, Harrington spent three years as product manager for Microsoft Word and observed all phases of product development and testing. “Some users think it’s anything that causes a program to act differently than they expected,” she says. “A lot of people think there’s a bug at work when a system crashes or when you ask a program to do something and it doesn’t – even if it’s not supposed to.”

No company intent on cultivating good customer relations can simply tell an irate caller to go back and read the manual, appealing as that option might be to a harried support staff. Harrington advocates diplomacy: “We have to consider it from the user’s point of view. If the user thinks there’s a problem, then there is one. Even if it turns out not to be a bug.”

Many company spokespeople recoil at the suggestion that corporate finances may dictate a product’s premature release. Tom Pettibone, vice president of product development at Central Point Software, is more honest – and more philosophical: “Sure, there are market pressures to release a product that may not be ready. You just have to discipline yourself not to.” The release of bug-ridden version 7.0 of Central Point’s PC Tools was a too-illustrative case in point. Nearly 250,000 copies of corrected version 7.1 had to be sent to customers, although the company did so free of charge.

But the market – at least from the IBM-compatible standpoint – is also responsible in another, more direct way for many of the problems that provoke software bugs. “The market is an ever-changing thing,” says Pettibone. “Every time you turn around there are new machines and new operating systems.”

“The real problem is one of compatibility issues,” says Adobe’s Eric Zocher. He’s the Director of Applications Engineering and therefore responsible for overseeing all the engineers working on all platforms and projects. “Users are apt to find problems depending upon their particular combinations of DOS, Windows, CPU, video card, BIOS, and so on.”

Because Adobe develops products for so many platforms and operating systems, the quality assurance department is forced to develop testing strategies to accommodate all probable combinations. “QA has a big matrix for each product that shows us all the configurations we test against. Actually, because of all the possible hardware and software combinations, it becomes a huge set of multidimensional matrices.

“You have more than 100 manufacturers selling more than 1,000 PCs a year, so we need to have those brands represented. We also have the 25 most popular video cards, along with printers and network products. We have to keep up with software, to make sure we have the latest versions of, say, Norton Desktop for Windows and QEMM. We have 9 standard utilities that we test against, but there are hundreds out there.”

The recent release of Adobe’s newest version of Illustrator for Windows put that matrix to the test. “We had to go through a long list of other applications, reading from and writing to those programs as well as doing lots of cutting and pasting. File compatibility is vital: We have to be sure that our files work with programs like Word for Windows, Ventura Publisher, PageMaker, and others. We have a long list of graphics file formats to test against, too. Mix all that together, and you get some idea how complex it can be.”

Says Microsoft’s Harrington: “Most of the troublesome bugs we see are the result of configuration problems. Different combinations of things can cause problems that we don’t find out about until the program has entered the market.”

Pettibone, who also has seen his share of platforms, seconds the thought. “It’s a lot easier to put something out for the Macintosh, because then you know that every machine is fundamentally the same.” He also laments a lack of cooperation among companies, criticizing Microsoft in particular: “They’ve been getting to be a real nuisance. Whenever they put out a new operating system, they already have all their applications tested. We don’t have that kind of access to their code, so it’s tough for us, and doubly tough for a company like Lotus, that is in direct competition with Microsoft in the applications market.”

Hardware and software conflicts breed nasty bugs, but programming errors spawn bugs of their own, especially when many programmers are involved. LaVange provides an amusing illustration of the perils inherent in programming.

“We did an experiment with a bunch of cub scouts who came in to WordPerfect to see what programmers do. We had them play a robot game. One kid was the robot, one kid was a programmer. The programmer had to write a series of instructions to get the robot to pick something up on the other side of the room. If the robot literally interpreted those instructions, it didn’t always work. That’s a tough lesson to learn: you have to write every piece of instruction correctly because your machine won’t be cogitating. As instructions get more complicated, that starts to work into thousands and thousands of lines of code.”

Most software companies began with the work of one or two engineers, but that’s no longer possible with highly ompetitive, mainstream projects. “We have many people writing the code for any one project,” says LaVange, “so the conditions set up by one programmer could conflict with other parts of the project. And human beings make mistakes.”

Programming mistakes are usually fairly unspectacular, although they catch the public’s attention when nasty incidents like last summer’s telephone service interruptions and the Hubble telescope problems occur. But human error is usually easier to discover than the conflicts between programs and machines.

KEEPING BUGS AT BAY

LaVange acknowledges that companies are reluctant to talk too much about problems. “You want to keep the appearance of having only bug-free products. Of course we don’t want to release anything until all the bugs are out. But you can never get all the bugs out. What you can do is take measures to see that the product is reliable, using a large testing team.”

Product testing gobbles a surprising amount of the resources of many software companies, competing for valuable product development staff and personnel. Central Point’s Pettibone would like to see one quality assurance engineer per programmer on staff. But that’s an unrealistic ideal. “The amount we spend on QA is about one-third of the product development time and one-third of the expense.”

Pettibone witnessed that disastrous release of PC Tools. “When the problems with Version 7.0 were discovered, we moved fast. We were honest with our customers. We ran into some situations with DOS 5.0 and the way it works with hardware, and we fixed those problems for our customers at our expense.”

Quality assurance isn’t a uniform process from company to company. Adobe’s QA engineers write the test programs used for its applications, says Zocher. “The font-testing programs are totally nasty, written to run under DOS, Windows, and on the Mac. Every typeface is automatically put to work at every angle. We have scripted testing for the big applications, including good tools for DOS, but I’m not happy with the tools we use for Windows. One problem is that in recent months we’ve had to work with Windows version 3.0, then 3.0A, then different betas of 3.1. And, of course, when we find bugs in Windows, we have to build a workaround or wait for Microsoft to fix it.”

WordPerfect’s in-house testing is less automated because a word processing program inspires more spontaneous computing. The challenge is to anticipate all of the strange combinations that users will concoct, which is done using scripts of pre-determined combinations along good second guesses of the less-predictable ones that particular projects will prompt.

Is there any possibility of standardized testing? According to the Software Engineering Institute, a research and development group on Pittsburgh’s Carnegie Mellon Institute campus, there’s at least a need for such a thing.

According to a report in Electronic Business magazine, the SEI ranks a software company’s operations on a five-level scale that gives lowest recognition to a company whose operations are perceived as “unplanned and chaotic,” moves up one level to acknowledge a stabilized operation with an intuitive routine and sees at the summit a company whose development processes are optimized based on an ongoing analysis of efficiency and usefulness.

Watts Humphrey, who directs the SEI’s software process program, examined several hundred software development projects and determined that “75 to 85 percent are at level one, 15 to 20 percent are at level two and very few are operating at any higher level.” The Malcolm Baldrige National Quality Award recognizes efficient operations, rewarding mature, skillfully-optimized projects. But very few companies even compete for the award.

One company that has competed is Oregon-based Mentor Graphics Corporation, which develops electronic design automation systems. For its project analysis, Mentor uses the FURPS system, developed by the Institute of Electrical and Electronics Engineers. The system measures functionality, usability, reliability, performance, and support, and can therefore serve as a yardstick both to engineers and QA staff.

“Any time you go through the development process you’re going to find bugs,” Pettibone declares. “It means you haven’t finished the product. You need good testers, and you need good machines in-house to test with.”

In-house testing is known as the Alpha stage. Beta testing, that penultimate phase of software development, gives a select group of users a chance to pummel the program. Cooperative computerists with both mainstream and oddball configurations are sought to help the company discover what’s going to happen when the real world gets hold of the new application.

Typically, the user is sent a package containing program diskettes and prototypes of the manual, as well as a system for reporting bugs and inconsistencies--often using an online service like CompuServe to report those problems. Each round of problem corrections prompts a new release of disks; the process can take several weeks and result in three or four rounds of updates.

“When the time comes to send the product to beta testers,” says Pettibone, “it’s difficult to find all the different configurations we need among our volunteers.”

“We have to rely on testers with unusual configurations,” adds Zocher. “They give us a better look at real-world problems. Of course, given that there’s an infinite number of hardware and software configurations, you’ll never find your ideal number.”

Microsoft boasted of having over 10,000 beta-testers working on the final stage of DOS 5.0 development. Central Point doesn’t have anything approaching those resources, says Pettibone, “but we’re now in the thousands range when we release a big product.”

After Adobe ships a new application, says Zocher, “Customer support is given access to our bug-report database. QA is scaled back from 20 to about 3, but they still back up customer support. And if a major application from another company ships after we ship our product, we still have to put that application in our testing matrix.”

Should post-release bug-fixes occur, what happens next is the dreaded maintenance release. For some products, it may prompt a new version number. PC Tools version 7 became version 7.1; dBASE IV 1.0 went to 1.1. Microsoft’s Word for Windows version 2.0 had a quiet improvement that became 2.0a, but customers only knew about that through bulletin boards or trade publications. WordPerfect’s maintenance releases keep the same version number, but the program’s date (accessed through a Help screen) is changed. And WordPerfect advises its customers about such changes in its quarterly newsletter.

However the change is made, the process is called slip streaming. Ideally, customers (and retailers) never notice; the packaging looks the same, so there’s no need for a company to take back unsold merchandise. A customer’s best protection from slip streaming is continued contact with customer support: most companies will ship maintenance releases for little more than the cost of disks and postage.

LIVING IN THE REAL WORLD

Can we ever expect a bug-free program to hit the market? “That’s not a real-world expectation,” Microsoft’s Harrington insists. “All users would have to think alike and come to a program with identical levels of experience and the exact same hardware.”

This may be one of those rare subjects about which competing companies agree. “There will never be a period when we find all the bugs,” LaVange says. “Six months after we release a program we’re still finding them, even though it’s at a declining rate.”

We’re a long way from the days when a bug could simply be tweezed from a machine. Programs are more complicated than could have been imagined in the Mark II days and are getting more complicated still. Complication invites bugs. With the continued development of object-oriented programming, it may be possible one day to assemble an application from bug-free modules, but many programmers agree that it’s still a tall order to expect that kind of buglessness anywhere.

That’s distressing news for the complainers who persist in maintaining the fairy-tale notion that software programs should have no programming faults.


Given the unavoidable presence of bugs, what matters most in the end is the support a developer offers. As long as the company accepts its responsibility to help battle the problems – particularly in those harried prerelease stages – no user needs to feel alone in the battle.

“We like to take a long view in the area of software development,” says Harrington. “It really is a word-of-mouth business, so customer satisfaction is absolutely in Microsoft’s best interest. We depend upon a customer’s reactions. Whether it’s a true software bug or something the customer just doesn’t understand, when we get a problem report, we want to resolve it.”

Which leads to another touchy subject: Who should pay for such support? If it’s the company’s fault, should it be the company’s nickel? And who could easily determine that responsibility? The best anyone can offer right now is assurance that you’re not alone in finding those persnickety problems, and they’re not likely to go away. A recent CompuServe message put it nicely. “I understand that bugs are always going to be there,” the writer mused. “I just wish there were more gnats and mosquitos and fewer killer bees.”

– July 1992