The Month My AI Pipeline Failed 100% of the Time and Every Log Said "Success"

MyVitals extracts lab values with one LLM call, then reconciles raw names against a shared dictionary — exact match, fuzzy match, then an LLM call for the leftovers. If that call fails outright, the pipeline doesn't fail the upload — it degrades: unmatched metrics auto-create a new dictionary entry instead of finding the right one. Good design, on its own.

Here's the problem: an env var meant for the OCR endpoint got set on the variable that goes to Chat Completions instead. Every metric-matching call, for about a month, hit 400 Invalid Model. Every single one. And because of the graceful degradation above, every upload still reported success.

Why nobody noticed

The visible symptom wasn't an error — it was a canonical-metrics dictionary slowly fragmenting into near-duplicates. Nothing about that looks like a bug from the outside. The metric everyone watches, "did uploads succeed," stayed at 100%.

Lesson: when a stage is designed to degrade instead of fail loudly, the success of the operation it's embedded in tells you nothing about whether that stage ran. You need a metric for the stage itself.

What actually caught it

Not an alert, not a test — there's no integration harness exercising the real pipeline. A routine cost audit noticed the LLM-matching cost line had gone quiet, which prompted someone to actually read the error logs. The diagnosis — 400 Invalid Model — had been sitting in the database the whole time, inside a raw error string, in a tooltip nobody had reason to open.

The fix wasn't more data, it was structuring what already existed: typed failure classification (error type, HTTP status, retryability) grouped by that classification instead of raw message text. 4× · 36% of matching calls · invalid_model is a bug report. Four chronological stack traces is a wall nobody reads.

The retry budget has to know your reverse proxy, personally

Extraction runs synchronously inside the upload request — no job queue yet. Nginx's proxy_read_timeout is 120s. If retry logic doesn't know that number: three 90-second retries run for 4.5 minutes against a socket Nginx already closed. Client gets a 504, server keeps working on nothing, report sits at "processing" forever.

Fix: a shared wall-clock budget (105s, deliberately under 120s) passed through both extraction and matching, so retries stop before the proxy would kill the connection anyway. Retries only happen for genuinely transient failures — a 401 isn't going to succeed on attempt two, and every retry is a real charge. A provider's Retry-After is honored as sent, not clamped — ignoring a rate limiter's instructions just earns another 429 faster.

Your retry budget and your proxy's timeout are the same constraint wearing two config files. Tune only one and you haven't fixed anything — you've just moved where the zombie request dies.

Takeaways

• Give every "designed to degrade" stage its own visible failure-rate metric.
• Structure error data at write time, not read time — you won't go back and re-parse a month of raw strings later.
• Budget synchronous retries against your actual proxy timeout, explicitly, in the same unit.
• Log the alias and the resolved concrete model, separately — "it worked yesterday" isn't debuggable if you can't tell whether the model changed underneath you.

---

👉 Try MyVitals now — the pipeline now tells on itself when something's wrong.