Black Friday: Retail Holiday or Developers’ Nightmare?

In the United States, Thanksgiving has arrived. For many of us, this serves as a chance to see family, play some football, and stuff ourselves with an unholy amount of food. Thanksgiving also indicates an unofficial kickoff to the holiday season and by the time the last plate of leftovers is conquered we all have that one enthusiastic relative that is dusting off a Mariah Carey album. (Sorry about the trademark loss Mariah, you’ll always be the "Queen of Christmas” to us.)

For many brands and their systems engineers, Thanksgiving is the eve of their most stressful day of the year. While others are lining up outside a mall at midnight to “politely move” their neighbors aside for the shot at a heavily discounted TV, most back-end folks are hoping that their servers hold up during shopping’s Armageddon. Black Friday and Cyber Monday may be holidays to some, but to your friendly neighborhood developer or QA engineer, it’s a game of watching how much weight the ice can take before it finally cracks. 

User experience is paramount this time of year, and one of the measures we use to track success is uptime. But as we saw with multiple high-profile outages last year, it doesn’t matter if your platform has 99.99% uptime if that .01% downtime comes at the worst moment imaginable. A mishap when the entire world is watching can have cascading effects. We all know outages are bad for brand equity, but when they create a complete feeling of distrust, an outage could indirectly cause the demise of an entire company. This is clearly an area that we need to take seriously.

These late November shopping holidays have become sort of the norm. Much of the world has “Singles’ Day,” which runs on 11/11 and drives record revenue for platforms like Alibaba. North America claims the weekend after Thanksgiving. Some prudent brands have decided to offer their sales earlier in part to reduce the stress of driving millions of consumers to their platform in a single day. But what if they don’t? Are there ways that testing can save these platforms from an utterly disastrous outage?

Yes, but it goes beyond standard performance testing. Chaos testing or testing for failover and security testing is also of the utmost performance when dealing with a stressful event such as this. If one area falters, the whole house of cards is likely to come crashing down.

Adobe forecasts that $221.8 billion is at stake during this holiday season, with the period of Thanksgiving and Cyber Monday accounting for $37.2 billion of that. Cyber Monday alone was predicted to bring in $12 billion, with Black Friday coming in close behind at $9.6 billion. Suffice it to say, on the two days of the year when $21.6 billion are at stake, you probably don’t want your servers to crash.

If we divide that number by 48, we see that $450 million is spent by consumers every hour of the season. For every hour that you are down, you receive no slice of that pie. For every minute you are down, you see none of the $7.5 million consumers spend per hour. Mid-tier retailers only get a small percentage of that revenue anyway, but a giant like Amazon is looking at an actual revenue loss of millions of dollars in the same situation. 

Speaking more broadly, IDC estimated a few years ago that an outage among Fortune 1000 organizations costs $100,000 per hour. Historically, the average cost of outages among these organizations falls between $1.25 billion and $2.5 billion annually, a number that has certainly increased with the rapid growth of ecommerce. Imagine being the Director of QA and telling your CEO that a 10-minute outage cost your company 7 figures — truly a black Friday it would be.

The worst part is these estimates are only tracking actual revenue (or lack thereof). The long-term damages can be far-reaching and even more devastating — a customer experience failure damages trust, brand equity, and even a sense of security. Our own research shows that many users will abandon a brand after a negative experience and then they’ll go straight to Reddit or Twitter to tell all their friends how much your platform sucks.

But enough of the fear-mongering! The holidays are supposed to be a time of joy and relaxation, right? (Well maybe not in this author’s family, but I digress.) Surely, I have not authored this article merely to say the end is nigh and there is nothing to be done about the impending digital apocalypse.

You can avoid a Turkey Day meltdown by leveraging Sauce Labs Intelligent Software Debugging solution, a suite of tools that empower you to accelerate root cause analysis, prioritize impactful bugs, and remediate issues before they reach your customers.

The robust automated test infrastructure, mobile application diagnostics, and ML-driven failure analysis that make up our debugging solution cuts mean time to resolution (MTTR) by up to 50%, according to Forrester.

Ready to see for yourself? Take an interactive tour, and then sign up for a free trial.

Systems will crash all across the globe this weekend, and while not all will be preventable, better planning and synchronicity among various QA teams may have helped. Giving you these tips this late in the game almost feels cruel and harkens to the Alanis Morisette song “Ironic:” “It’s the good advice, that you just didn’t take.” Perhaps I’ll write next year’s Black Friday blog in March, so you have some time to execute these plans. As for this year? I’ll borrow a turn of phrase from the football games that you will be watching during your fifth plate of stuffing. If you didn’t plan this year, you can always hope for a Hail Mary.