Blog

No Single Point of Failure: Systems That Keep Working

· The Yao Group · 5 min read

A single point of failure is one part of a system that, if it breaks, takes everything down with it. It might be one server, one network link, one supplier or one update that reaches every computer at once.

Most companies learn where theirs are during an outage. A better time is now, while the systems are still running.

Three outages that started in one place

  • June 2021, Fastly. A bug had sat unnoticed in Fastly's software for weeks. Then one customer made a normal settings change that set it off, and 85 percent of Fastly's network began returning errors. Fastly spotted the problem within a minute and had 95 percent of its network back within 49 minutes (Fastly).
  • October 2021, Facebook. During routine maintenance, a command meant to check network capacity took down every connection in Facebook's backbone, the private network joining its data centers. A tool built to catch bad commands had a bug and let it through. Engineers then had to get into data centers in person, and the strong security on those buildings slowed them down (Meta Engineering).
  • July 2024, CrowdStrike. One faulty content update for CrowdStrike's security software crashed about 8.5 million Windows computers (Microsoft). It was not rolled out in stages. Among CrowdStrike's fixes was a staged rollout, which sends a change to a small group first and widens it only after that group is fine (CrowdStrike root cause analysis).

In each case the trigger was small, and the damage was large because nothing stood between that one change and everything else. Google's engineers call this a cascading failure: a failure that grows over time because each part that breaks puts more load on the parts still working (Google SRE book). Large systems are still built this way today, which means the next big outage is likely to start the same way.

Spare parts: the idea behind RAID

In 1987, David Patterson, Garth Gibson and Randy Katz at the University of California, Berkeley published "A Case for Redundant Arrays of Inexpensive Disks", the paper that named RAID. RAID spreads data across several small disks with spare (redundant) information, so the system keeps running when one disk fails. The paper set out five levels of RAID and argued that arrays of small disks promised large gains over one large, costly disk in performance, reliability and power use (UC Berkeley).

The same idea works far beyond disks. Amazon's guide to building reliable cloud systems says to replace one large resource with several small ones, so a single failure has less effect. It also says to spread requests so they "don't share a common point of failure" (AWS Well-Architected, Reliability).

Plan for it in the design: TOGAF

TOGAF is The Open Group's framework for planning how a company's systems fit together. One of its example principles is business continuity: "Enterprise operations are maintained in spite of system interruptions." TOGAF says hardware failure, natural disasters and data corruption should not stop the business. It also says recovery, spare capacity and ease of repair "should be addressed at the time of design" (TOGAF Standard 9.2, Architecture Principles).

Disaster recovery is the plan for getting systems and data back after a serious outage. The U.S. government's guide on this, NIST SP 800-34, treats it as one part of contingency planning, which is the wider work of keeping a system's mission going when something goes wrong (NIST SP 800-34 Rev. 1). A plan that has never been tested is a guess.

From surviving failure to learning from it

In his 2012 book Antifragile, Nassim Nicholas Taleb argues that things should be built to gain from shocks. As the publisher sums up the book, the resilient "resists shocks and stays the same", while the antifragile "gets better and better" (Penguin Random House).

A system can be built to get better from failure too:

  • Break it on purpose. Chaos engineering means running careful experiments that cause failures, to build confidence that a system can handle rough conditions in real use (Principles of Chaos Engineering). Each weak spot found in a test is one that will not surprise you later.
  • Test recovery, then automate it. Amazon's guide says to test how a system fails and check that recovery works. It also says to watch business measures, so automation can start repairs when one crosses a limit (AWS Well-Architected, Reliability).
  • Stage every change. Send updates to a small group first, watch the results, then widen the rollout.
  • Feed every failure back into the design. Each outage, real or practiced, ends with a change that removes the weak spot it found.

Why this needs autonomous security engineering

Availability is one of the three goals of security. The U.S. government's FIPS 199 defines it as making sure people can reach information when they need it. An outage is a security event even when no attacker is involved.

Watching for failures, moving work to spare capacity, rolling out changes in stages and recording what happened all have to happen faster than people can do them by hand. That is the work of Autonomous Security Engineering: your existing tools set up to work together, catch trouble early, recover on their own and keep the evidence your auditors need. A person still decides the steps that can hurt the business most.

Read more about Autonomous Security Engineering, or schedule a discovery call.

Sources

security architecture disaster recovery resilience

More posts

Book a call