Series: Engineering Judgment, Part 1

Complex systems do not usually fail because one engineer made one mistake. They fail because the number of possible states has grown faster than the organization's ability to understand, test, and control them. This is the part of engineering we tend to underestimate.

A new feature is easy to describe, but a new branch in the state space is harder to see. A compatibility path, a retry rule, a migration exception, or a temporary fallback that becomes permanent may each look small in isolation. Together, they create interactions that no roadmap records and behavior that no single person can fully hold in their head. The implementation grows one change at a time while the system grows through combinations.

That combinatorial growth is what I mean by entropy in software. It is not disorder as a vague metaphor, and it is not merely code that looks messy. Software entropy is the growth of possible system behaviors, especially the behaviors that emerge from combinations nobody explicitly designed. Its danger is that it often arrives disguised as progress.

Functionality Grows Additively. State Space Does Not.

Product roadmaps usually count features one by one, but engineering systems do not experience them that way. If a system has several modules, and each module can be in multiple states, the total number of possible system states grows through combinations. A small feature may add only one state to one module while also creating interactions with every module that depends on it. That is why a change that looks local in code review can become global in production: the feature may be small, but its new behavior surface is not.

This distinction matters because most planning discussions reason about implementation size. They ask how many files will change, how many APIs will be touched, and how many weeks the work will take. Those questions are useful, but they describe the cost of producing the change rather than the cost of absorbing it into the system. A more consequential question is how much new state space the change introduces.

If a feature adds one clean capability inside a well-isolated boundary, that cost may be reasonable. If it creates new interactions across the core execution path, persistence layer, recovery path, and operational tooling, the feature is not small even if the diff is. Many production incidents begin with precisely this mismatch: the implementation looked bounded, but the behavior was not.

Accidents Are Often Normal

Engineers naturally want to find the bug, and the bug matters. In complex, tightly coupled systems, however, a bug is often the final expression of a deeper condition: the system has accumulated enough hidden interactions that some combination will eventually escape the design. Charles Perrow's Normal Accident Theory is uncomfortable because it removes the comfort of blaming only local mistakes. In certain systems, accidents are not exceptional events but normal outcomes of complexity and tight coupling, which means engineering quality must include control of the complexity itself.

A blockchain network halt, a distributed database split-brain, a cascading service outage, or a failed migration rollback will usually be explained afterward as a specific defect. That explanation may be correct, but it is too narrow to guide the next design. Across these incidents, the deeper pattern is often the same:

  • a local change touched a global assumption
  • a fallback path behaved differently from the primary path
  • an old compatibility rule interacted with a new feature
  • recovery logic was less tested than forward execution
  • the team optimized the happy path and underestimated the state space around failure

The incident makes the failure visible, but the entropy was already present in the relationships among those conditions. Fixing the immediate defect is necessary; reducing the chance of recurrence requires shrinking or clarifying the state space that allowed it to matter.

Coupling Turns Small Failures Into System Failures

Complexity is not just the number of parts; it is the way those parts depend on one another. Early systems are usually understandable because their boundaries are still sharp. There may be one team, one architecture, one deployment path, and one shared mental model, so when something breaks, the search space remains small.

Success changes those conditions. More teams, features, and integrations arrive, and the original boundaries become historical suggestions. A module that once had one job acquires several responsibilities and exception paths. A migration intended to remove old behavior keeps both paths alive because one customer still depends on the old one, while a "temporary" compatibility layer quietly becomes part of the platform.

As these dependencies accumulate, local reasoning stops working. A change in one place modifies timing in another; a new cache changes consistency assumptions; a retry rule multiplies load during partial failure; a schema evolution breaks an operational script nobody remembered. Each effect is understandable after the fact, but none is visible from the changed component alone.

This is where systems become dangerous. They do not need to look obviously chaotic; they only need to retain the appearance of modularity after the real coupling has moved underneath the abstractions.

Abstractions Hide Complexity Until They Leak

Abstractions are necessary because no large system can be built without them. They do not delete complexity, however; they move it. A good abstraction moves complexity to a place where it can be controlled, while a bad one hides complexity until the moment when control matters most.

This distinction is especially visible in infrastructure systems. A clean API may hide persistence semantics, a simple transaction interface may hide scheduler behavior, and a runtime abstraction may hide memory, state access, or recovery constraints. Most of the time the abstraction holds, which encourages the organization to forget the layer beneath it. When a rare production condition appears, that hidden layer can suddenly become the only one that matters.

At that point, engineers have to reason through several levels at once:

  • the public interface
  • the implementation details
  • the historical exceptions
  • the operational environment
  • the failure mode currently unfolding

These levels are not independent, and the failure often sits in the interaction among them. The problem is therefore not that abstractions are bad, but that every abstraction carries a maintenance cost. That cost grows as the system evolves away from the assumptions under which the abstraction was first designed.

The Largest Source of Entropy Is Often Organizational

Technical entropy and organizational entropy reinforce each other. A system may begin with one small team and one clear objective; years later, it may have several teams, overlapping ownership, multiple roadmaps, and layers of compatibility promises. The codebase then reflects not only technical decisions but the history of negotiations, deadlines, and responsibilities around it.

This is why some systems remain hard to simplify even when everyone agrees that they are too complex. Removing a feature may require negotiation with another team, deleting an old path may require changing support commitments, and simplifying an architecture may require acknowledging that previous decisions no longer make sense. The technical work is real, but the organizational cost is often the blocker because simplification redistributes obligations as well as code.

When ownership is unclear, entropy wins. Nobody is directly responsible for removing old complexity, while everyone has an incentive to add the requirement they currently need. Growth has owners, schedules, and visible outcomes; simplification often has none of them. Organizations therefore accumulate systems that nobody would intentionally design, even though each local decision was understandable at the time.

Entropy Reduction Is Engineering Work

The natural direction of a successful system is toward more complexity, so entropy reduction has to be deliberate. The best engineering teams are not those that add the most features, but those that preserve the ability to understand and change the system after years of growth. Doing so requires practices whose value is cumulative and therefore easy to underestimate in the short term.

Complexity budgets are one such practice. Before adding a feature, a team can ask what long-term complexity it introduces: a new state, mode, operational path, ownership boundary, or compatibility promise. The answer makes an otherwise hidden cost discussable. It also forces the team to decide who will pay that cost after the feature itself has shipped.

Rollback-first design provides another test of control. If a change cannot be safely rolled back, the team should treat that fact as a risk signal rather than an implementation detail. Rollback is not merely an operations feature; it is evidence that the system can still move between known states. Designing for it exposes dependencies that a forward-only plan can leave hidden.

Deletion is even more direct. Most organizations are skilled at adding functionality, but few are disciplined about removing obsolete modules, dead flags, old compatibility paths, and redundant abstractions. Deletion reduces the number of states the system can enter and the number of histories engineers must understand. Simplicity should therefore take priority over feature parity when a proposed capability does not solve a critical problem whose long-term value exceeds the complexity it introduces.

The hardest engineering decision is often not what to build, but what to stop carrying.

The Real Competition

In the short term, systems compete on capability; in the long term, they compete on their ability to remain understandable. A system that accumulates features faster than it accumulates clarity will eventually slow down. Changes become riskier, incidents become harder to diagnose, and engineers become afraid to touch certain paths. The organization begins working around the system instead of improving it, turning accumulated entropy into a strategic constraint.

The job of engineering leadership is therefore not only to accelerate delivery, but to keep the system within the range where delivery remains possible. That means treating complexity as a first-class cost, protecting iteration speed, and creating explicit ownership for deletion. It also means asking repeatedly whether the system remains understandable enough to evolve, because that capacity can disappear gradually while every local delivery metric still looks healthy.

Doing less is not always discipline; sometimes it is avoidance. In a complex system, however, choosing not to add something can be one of the highest forms of engineering responsibility when the added capability would weaken the system's future control.

A system survives not by accumulating every possible feature, but by accumulating complexity more slowly than its ability to control it.