Why automations break: a taxonomy of failure

Every automation eventually fails. That is not pessimism, it is arithmetic: an automation depends on several external systems, each of which changes on its own schedule without asking you. Over a few years, the probability that none of them changes in a way that matters is close to zero.

What separates a healthy automation stack from a liability is not whether things break. It is whether failures are loud — visible, safe, and fixable — or silent, quietly producing wrong output until someone notices weeks later.

Here are the nine failure modes we actually encounter, roughly ordered by frequency, with what each does and how it is designed against.

1. Credential expiry

Frequency: most common by a wide margin. Damage: low, if detected.

OAuth tokens have lifetimes. Service account passwords hit rotation policies. API keys get revoked when someone tidies up. And the classic: the integration was authenticated with a departing employee's personal account, so it dies the day IT disables them.

Design against it: service accounts rather than personal ones, always. Credentials in a shared vault the business controls. Monitoring for authentication failures specifically, since they are distinguishable from other errors and worth alerting on differently. Where a token has a known lifetime, alert before it expires rather than after.

2. Vendor API changes

Frequency: every integration outlives at least one. Damage: low to moderate.

A field is renamed, an endpoint deprecated, a response format restructured, rate limits tightened, or a required parameter added. Usually announced, usually in a changelog nobody subscribes to.

Design against it: validate response shape rather than assuming it. If an expected field is missing, stop — do not write a null over good data. Pin API versions explicitly where the vendor supports it, so upgrades are a decision rather than an event. And subscribe to the changelogs of the two or three vendors your critical paths depend on.

3. Silent data-shape drift

Frequency: common. Damage: high — this is the dangerous one.

Nothing errors. Someone adds a column to the spreadsheet, renames a CRM pipeline stage, changes a dropdown option, or starts entering dates as DD/MM instead of MM/DD. The automation runs perfectly against a world that has moved.

The date format case is worth dwelling on: 03/04 is valid either way, so nothing throws an error, and you get plausible wrong dates for months.

Design against it: validate values against expected sets, not just types. If a pipeline stage arrives that is not in the known list, that is an error, not a pass-through. Range-check dates and amounts. Assert on cardinality — if a query that normally returns 40 records returns 4,000 or zero, stop and ask.

4. Unhandled exception cases

Frequency: common, concentrated in the first months. Damage: moderate.

The workflow assumed every order has a PO number. Then sales started taking phone orders. Nothing is broken — the logic simply never anticipated this input and now does something unhelpful with it.

Design against it: an explicit default branch on every decision point, and that default should be "route to a human," never "carry on as if this were normal." Most of these surface during parallel running, which is the argument for doing it. More on this in the exception problem.

5. Transient dependency outages

Frequency: constant at low level. Damage: negligible if handled, moderate if not.

An API returns a 503, a network call times out, a platform is briefly degraded. Almost always resolves within minutes.

Design against it: retry with exponential backoff — three or four attempts over a few minutes absorbs the overwhelming majority of these invisibly. Critically, retries must be idempotent: if the workflow is unsure whether an invoice was created, it checks rather than creating a second one. Non-idempotent retries turn a harmless outage into duplicate records, which is a self-inflicted version of a worse problem.

6. Race conditions and sync loops

Frequency: uncommon, but only because two-way sync is uncommon. Damage: high.

Two systems both update. Each update triggers a sync to the other. Each sync triggers another update. In the mild case you burn operations; in the bad case fields flip back and forth and neither system holds the truth.

The related case: both sides change the same field between sync runs. Without an explicit conflict rule, whichever ran last wins, and that is not a decision anyone made.

Design against it: origin tagging so a sync-initiated write does not re-trigger a sync. Explicit conflict rules — last-write-wins, source-of-truth-per-field, or hold-for-human — agreed by the business rather than defaulted by the engineer. And a reconciliation pass that detects drift even when nothing errored.

7. Volume and rate-limit surprises

Frequency: occasional, usually at a predictable moment. Damage: moderate.

The workflow was built and tested against typical volume. Then month-end arrives, or a marketing campaign lands, or a data migration dumps 5,000 records in an hour. The API rate-limits you, the platform's operation quota is exhausted mid-month, or a queue backs up faster than it drains.

Design against it: respect rate limits deliberately with throttling and queueing rather than discovering them. Test against peak volume, not average. Alert on quota consumption at 70% rather than at 100%, when there is still time to act.

8. Orphaned ownership

Frequency: uncommon but near-certain over a long enough horizon. Damage: high, because it compounds every other failure.

This is an organisational failure with technical consequences. The person who built or maintained the automations leaves. Alerts route to their inbox, or to a shared channel nobody owns. Documentation was never written. The stack keeps running until it does not, and then nobody knows what it was supposed to do.

Design against it: alerts to a role or channel with a named owner, never an individual's mailbox. Documentation written for someone who has never seen the system. Credentials in a shared vault. And a periodic review that asks not "is it enabled" but "when did each of these last successfully do its job."

9. Success-triggered obsolescence

Frequency: uncommon. Damage: moderate, and easy to misdiagnose.

The rarest and most interesting. The automation works, the business changes because of it, and the automation is now solving yesterday's problem. Or a system it depends on gets replaced as part of a project nobody connected to the automation.

Design against it: you cannot, technically. This is what a quarterly review is for — a short, honest look at what is running, what it is still worth, and what should be retired. Automations should be decommissioned deliberately, not left running because nobody is sure whether they matter.

The six patterns that cover most of it

Almost everything above is defended by the same small set of practices. Every workflow we build ships with all six.

1. Retry with backoff — absorbs transient failures invisibly.
2. Idempotency — makes retries safe, prevents duplicates.
3. Validate before writing — converts silent drift into a loud stop.
4. An explicit default branch — unanticipated input routes to a human, never onward.
5. Dead-letter handling — nothing is dropped without a visible record and a reason.
6. Alert a named owner — with enough context to know what broke and what it was doing.

None of these are exotic. All of them are routinely omitted, usually because they are invisible in a demo and add perhaps 20% to a build. That 20% is the difference between an asset and a liability.

Auditing what you already run

Five questions, honestly answered, will tell you where you stand.

1. If this failed right now, who would find out, and how?
2. Whose credentials is it using?
3. What happens to a record it cannot process?
4. Where is it documented? (Scenario names are not documentation.)
5. When did it last successfully run — not "is it enabled"?

If several answers are uncomfortable, that is what a takeover assessment is for: inventory, mapping, documentation and a stabilisation plan. You keep the documentation regardless of what you do next.