The Scariest Software Bugs and Development Nightmares in History

In the Halloween episode of the Sauce Labs podcast Test Case Scenario, host Jason Baum is joined by testing experts Evelyn, Nikolay, and Marcus. The group dresses up in costumes, gathers around, and shares three software development horror stories from history. Tune in below to hear truly terrifying tales of how a lack of proper testing can cost you time, money, or even life as you know it.

Evelyn Coleman kicks off the episode with a scary story of a school experiment gone wrong. 

Cornell University graduate student Robert Tappan Morris created a computer warm — simply to prove it could be done, according to one of his friends. On November 2, 1988, Morris released the worm into the wild — not from Cornell, where he studied, but from the Massachusetts Institute of Technology (MIT) in hopes he’d never be caught. By exploiting weak passwords and other vulnerabilities, the worm infected an estimated 2,000 computers within 15 hours, leaving the machines “dead in the water”, according to Harvard University’s Clifford Stoll. 

While the worm inflicted an estimated $10 million in damages, it also exposed vulnerabilities and spurred research that improved digital security. And the ghoulish guy who unleashed the worm? He received a felony conviction under the 1986 Computer Fraud and Abuse Act. After serving three years’ probation, completing 400 hours of community service, and paying a fine of $13,326, he landed a job at MIT in 2006, where today he works as a tenured professor. 

Nikolay Advolodkin shares another truly terrifying tale of software development gone wrong. 

He used to work for a company that was responsible for receiving web requests from digital advertisers. His team appended information to the incoming web requests, so they could more easily identify their customers and make the ads more targeted. After updating a 404 page, his team pushed it to production. 

At the time, they had a load balancer with three servers they’d deploy on. After deploying it on the first server, things were OK. So, they began deploying it to the second. Their DevOps guy was soon flooded with alerts. After automated tests started failing, he asked if the team should pause any changes. The team convinced him the alerts were probably just false positives, and subsequently deployed to the last server. 

Suddenly, their chat application started slowing down. The internet crashed for not only their team, but also the parent company’s. After investigating, they found the page they deployed was a few megabytes in size — way bigger than it should have been. Hundreds of thousands of web requests were coming in with every second that passed, and the company’s services were getting redirected to the new 404 page. The failure cost their customers millions of dollars.

The key takeaway? Trivial changes are not always as trivial as they may seem. Always take the time to consider the potential impact of your software deployments. 

Marcus Merrell told a horror story about a disastrous defect with the Oklahoma State Sex Offender Registry.

The Registry allowed people to conduct queries to figure out which offenders lived nearby. At some point, someone noticed that the URL query string, which included information like a person’s social security number, date of birth, and home address, looked eerily similar to a SQL query. In fact, it was named SQL string. 

For at least three years, according to one publication, hackers could extract personal information from the database. 

At some point, someone figured out that the database also allowed people to update records — for example, to take yourself off the list or add someone to the list. Long story short, someone added the Governor of Oklahoma to the list of sexual offenders, says Merrell.

I think it’s fair to say this bug could have cost people their lives.

On June 4, 1996, the European Space Agency launched the Ariane 5 rocket into space.

Just 37 seconds into its inaugural flight, the rocket — which cost $7 billion and 10 years to build — veered off path and began to disintegrate, causing the self-destruct mechanism to trigger two seconds later. Fiery ruble soon fell from the sky, scattering across the swamps of French Guiana.

After a public inquiry, it was determined that the team's decision to repurpose code from the Ariane 4, without updating the necessary values and verifying their underlying assumptions, caused the failure.

In total, the incident cost over $370 million in damages and delayed our understanding of the Earth's magnetosphere by nearly four years.

Willie Conrad once worked for a company that hosted content for large media brands. By using "canned" searches stored in a MySQL database, his team made it possible for visitors to see tailored content based on the landing page they originally viewed.

When it came time to update the queries, Willie exported the records as a CSV file and processed it offline to generate the necessary updates.

"You should have seen me go," said Willie. "I was in my element — My fingers danced over the keyboard, frantically tapping out `grep` and `sed` commands like some kind of underground elite hacker, about to dazzle the world with their speed and cunning after breaking in to the world's most secure missile-defense system. The keyboard was nearly overheating from the frenzy. I thought of every conceivable little detail. Single and double quotes in the query string? No problem. Full UTF-8 support? You bet. Proper special character escaping? Please."

Willie wanted to double-check that everything looked right. The only problem was that there were tens of thousands of queries. So, he looked at a representative sample.

"I found just one little oversight," said Willie. "The very first row of my CSV file had the *names* of the columns, instead of the actual data."

To fix the error, Willie produced an update statement. He never intended for it to run, of course. After all, that would destroy every query in the database!

Willie confidently deleted the first row and then proceeded with the update.

Soon after, Willie received a slew of emails that suggested the wrong content was being served on client websites. Plumbing videos showed up on a gardening site. Investment advice appeared in the place of recipes. The emails kept coming, and his team soon realized that all the queries were set to the literal string, "QUERY" — despite the fact that Willie had deleted that top row.

"It was total chaos," said Willie. "I had just destroyed every single landing page, for every client in our system."

Bewildered about where he went wrong, Willie began to investigate. He discovered that the CSV file had not one but two sheets. While he had deleted the first row from the first sheet, he had neglected to delete the first row from the second sheet — causing all hell to break loose.

From this experience, Willie came to appreciate disaster recovery drills, which make restoring from a backup faster and less error-prone. He's also (understandably) more cautious about leveraging text processing hacks.

Historically, QA teams have been undervalued, regardless of whether they fail or succeed. If they fail, developers assume that flawed code is inevitable. If they succeed, developers never see the costly mistakes their testers have prevented. In either scenario, even the most well-intentioned teams can underestimate the importance of testing.

But these terrifying tales demonstrate that it's critical to prevent bugs from escaping into production. If history has taught us anything, it's that the mantra "always be testing" is relevant whether you're motivated to reduce business risk, keep your job, make your users happy, or conserve valuable resources.