The work, measured.

Case study 01 · observability

The automation failure your dashboard cannot show you

Self-built lab project — not a paid client delivery. Source, schema and raw test logs are public on GitHub.

Context

A business runs its lead intake, order sync and nightly reporting on automated workflows. Someone checks the dashboard in the morning. It says zero failures.

That reading is worth almost nothing — “zero failures” looks identical whether the day was clean or whether the automation has been dead since 2 a.m. Every monitoring setup in common use reports on runs that failed. None report on runs that never happened: a schedule that quietly stopped, a workflow left deactivated after an edit, a webhook registration lost on restart, a queue worker that died. Each produces no execution, therefore no error record, therefore nothing to see anywhere. The business finds out days later, from a customer.

Challenge

Detect the absence of work, not just its failure — and prove it against six specific failure modes rather than describing them:

1. A schedule silently stopped firing
2. A workflow was left deactivated
3. A webhook registration was lost on restart
4. The queue worker died and nothing is consuming jobs
5. A worker crashed mid-run, so the run never reached a terminal state at all
6. A crashed job was silently replayed by another worker, creating duplicate side effects while reporting success

Number 5 breaks conventional error handling: if an execution never terminates, n8n’s Error Trigger never fires. The entire business case rested on that being true — so it was tested before anything was built.

Approach

Test the assumption first (45 minutes, before ~8 hours of building). A worker container was killed mid-execution. Read directly from n8n’s own execution table rather than the UI, the run sat at status = running, finished = false, stoppedAt = NULL — indefinitely. No error row, no dashboard signal. The assumption held, and the design proceeded on measured ground rather than belief.

The system has four parts:

• A heartbeat written by each monitored workflow: one row when a run starts, closed when it succeeds, plus a last-success timestamp.
• An independent watchdog that alarms on absence — comparing each workflow’s expected interval against its last success, flagging runs that opened but never closed, and detecting silent replays via an attempt counter.
• A dead-man’s switch outside the automation platform entirely. The watchdog pings an external cron-monitor on each successful sweep; if the pings stop, that external service alerts. A monitor that dies with the system it monitors is just one more silent failure.
• Alerting with dedupe and recovery notices — a persistent problem doesn’t spam every sweep, and a resolved one is explicitly announced rather than just going quiet.

Outcome

The full matrix was executed live — nine tests, none skipped. Three found real bugs in my own implementation, which is the part worth reporting:

A restart left the webhook returning 200 OK while nothing ran. The expected failure was a 404. Instead the endpoint responded {"message":"Workflow was started"} — perfectly healthy-looking — while every execution queued forever. Worse than a 404, because the caller and any naive uptime check both see success. What caught it was the external dead-man’s switch: the watchdog’s own schedule was caught in the same stall, its pings stopped, and the external monitor flagged it. That single incident is the entire argument for building the fourth component.

Editing a live monitoring workflow silently disabled it. Saving changes did not republish it; the schedule stopped firing and the monitor went quiet ~18 minutes later. The act of maintaining the monitor turned it off.

An alert delivered successfully but was unreadable — a message field left in literal mode, so Slack received raw template text instead of the workflow name. The API returned "ok": true. Caught only by inspecting the actual API response of a live execution.

A backup → destroy → restore drill on the operational database completed with an exact match, including the edge cases a naive restore gets wrong: a still-open crashed run with a null finish time, and a replayed run carrying attempt number 2.

What this demonstrates

Connecting nodes is not the scarce skill. Knowing what an automation cannot tell you, testing that belief before building on it, and publishing the bugs found in your own work — that is the difference between an automation that looks fine and one you can trust with revenue-carrying processes.

Repo: workflows, schema and raw test logs →

Case study 02 · documents

Zero silent errors: what it takes to trust a document pipeline

Self-built lab project on public real-world datasets — not a paid client delivery.

Context

Pulling data out of a document is the easy half. The half that decides whether a finance, insurance or legal team can use the output is different: stopping a wrong value from entering the system without anyone noticing.

A pipeline that is 95% accurate and silent about the other 5% is unusable in any process with money or compliance attached, because you cannot tell which 5%. A pipeline that flags every value it is not sure about is something an accountant can work with. Those are two different products, and the second is considerably harder to build. That distinction — not the headline accuracy number — is what this project set out to measure.

Challenge

Take 100 real receipts (not synthetic, not cherry-picked — deliberately low-quality phone photos with OCR confidence commonly in the 36–49% range) and answer one question with evidence: how many wrong values does this system accept silently? Then drive that number to zero and keep it there while accuracy improves — verifying at every step that an accuracy gain has not quietly reintroduced silent errors.

Approach

Every field carries its own confidence, its own distrust flags and its own routing decision — a document can have one field accepted and another sent for review, because the field is the unit of trust, not the document. Every extraction attempt is stored append-only with the model, prompt version and input mode that produced it; a human correction is a new row, never an overwrite, so the machine’s original answer survives and you can still measure later how good it actually was. Thirteen runs, each changing one variable so the delta was attributable.

Outcome

Two real silent errors — found, diagnosed, structurally fixed, verified

Both were merchant names, both returned at 0.95 confidence, and both defeated three-pass OCR consensus for the same reason: all three passes made the identical mistake (the name sat at the image edge, so every pass lost it the same way). Consensus checking only catches disagreement; when the passes agree on the same wrong answer it is blind.

The fix was not “run OCR again” but a genuinely independent witness — cross-checking the extracted merchant name against the website domain printed on the receipt. The verification run caught exactly those two documents by name and returned the silent-error count to zero.

A wrong architectural assumption, identified and corrected

Accuracy plateaued around 56%, and the comfortable conclusion was “we’re near the limit of the OCR engine.” Published benchmarks on the same public dataset reach roughly 91–96% F1 — so that conclusion was wrong. The real problem was pipeline shape: image → OCR → plain text → language model, meaning the model never saw the document at all, only a lossy transcription. Sending the page image instead, everything else unchanged, produced a large jump — and fields the system had been unable to find at all dropped to zero, confirming they were never missing, only destroyed in transcription.

Two attractive shortcuts, both tested and both rejected on measurement

1. “The OCR guards are useless now that the model reads the image directly.” The data seemed to agree — those guards identified a wrong value only ~42% of the time. Measured result of relaxing them: auto-acceptance rose from 1% to 64%, and silent errors rose from 1 to 50. Precision was the wrong yardstick: the guards are low-precision but high-recall, and in a pipeline whose entire promise is “never wrong silently,” recall is the thing being bought. Reverted.

2. “If two independent reads of the same image agree, that’s stronger evidence than OCR consensus.” Measured result: auto-acceptance rose from 0 to 26, and silent errors from 0 to 6 — all six where both reads made the same mistake. The same blind spot, one level up: the same model reading the same image twice can produce correlated errors. Self-consistency now only adds suspicion when the reads disagree; it never relaxes an existing guard.

Honest status

One open item is stated rather than hidden: merchant_address is the weakest field at 78%, and most of its errors are one- or two-character differences (NO.53, vs NO. 53,). Whether those are genuine extraction errors or artifacts of the ground truth’s own transcription has not yet been separated out — so it is reported as unverified rather than counted as a win.

Industry practice supports the shape: reported straight-through processing in real deployments typically starts near 60% and reaches 85–90% over months, with roughly 15% of documents routing to human review considered normal and healthy. Human review is the standard shape, not an admission of failure — and the engineering above is what makes that review queue trustworthy.

What this demonstrates

Measuring what does not work, and reverting two changes that made the headline number look better while making the product worse, is the part of this work that transfers to any client system. “I tested the attractive shortcut and rejected it with numbers” is a harder claim to make than a high accuracy score — and for anything carrying financial or regulatory risk, a considerably more useful one.

Case study 03 · infrastructure

Backup → destroy → restore, verified 3/3

Self-built infrastructure test — not a paid client delivery.

The point

“We have backups” and “we’ve tested our backups” are different claims. Most teams can produce a backup file; far fewer have ever restored one and checked that what came back actually works.

What was done

The full cycle was run three times on a self-hosted n8n stack: back up, completely destroy the instance, restore from scratch. Each cycle verified that the encrypted credentials survived the teardown — confirmed by reconnecting a real Slack credential afterwards and watching a workflow authenticate and run, not just by seeing the row reappear in the database.

The restore also had to get the awkward states right: a still-open crashed run with a null finish time, and a replayed run carrying attempt number 2 — the exact rows a naive pg_dump and reload tends to mangle or drop. All three cycles came back with an exact match.

What this demonstrates

If your automation carries real business process, the operational question is not whether a backup exists but whether a restore has ever been rehearsed — and whether the person who set it up knows why the operations database is kept separate from n8n’s own schema, and why a lab worker set to restart: "no" must be configured differently in production.

Repo: docker-compose, backup / restore scripts and operational notes →