A field-ticket extractor we helped stand up ran clean for two months. Then in March it started dropping the charge code off a specific service vendor’s tickets. Not erroring. Not failing. Just quietly returning null where a code used to be. The pipeline’s dashboards were all green. Row counts matched. Null rates were within tolerance because plenty of tickets legitimately have no charge code. The vendor had changed their PDF template, moved the code into a differently labeled column, and the model started guessing wrong without ever saying so.
Nobody noticed until accounting flagged a handful of unallocated costs during monthly close. By then the pipeline had been wrong for three weeks.
This is the failure mode the rest of this series didn’t cover. We spent five posts getting you to a working AI batch pipeline. This one is about the harder problem: knowing it still works six months later.
Why AI pipelines fail quietly
A traditional pipeline fails loud. The API times out, the schema doesn’t match, a foreign key blows up, the DAG turns red, and someone gets paged. You built monitoring for exactly those events and it catches them.
An LLM extraction pipeline can be completely healthy by every one of those measures and still be wrong. The API returned a 200. The JSON parsed. The row landed in the table. The value in it is garbage. Every check you inherited from normal data engineering passes, because those checks were designed to catch broken plumbing, not broken judgment.
The inputs move underneath you too. Your code doesn’t change, your prompt doesn’t change, and the extractor still degrades because a vendor reformatted a document, a model provider shipped a silent update, or the mix of documents arriving this month looks nothing like the mix you tuned on. Offline testing against a fixed set catches the changes you make. It does nothing about the changes that happen to you.[1]
So the question is not “did the pipeline run.” The pipeline always runs. The question is “did it get the right answer,” and answering that takes a different set of tools than the ones you already have.
Build a ground truth set from your first 500 documents
Before you can tell whether a pipeline is still right, you need a fixed reference for what right looks like. That’s a ground truth set: a collection of real documents paired with the correct extracted values, verified by a human who knows the domain.
The good news is you probably already have it and don’t realize it. When you built the pipeline, you ran documents through review. The first few hundred that a human looked at, corrected, and approved are your ground truth. Pull the first 500 processed field tickets, run tickets, or JIBs where a reviewer confirmed the output, freeze the input files, and store the confirmed values alongside them. That’s the whole exercise.
A few things that make the difference between a useful set and a decorative one:
- Cover the document variety you actually see. If you process tickets from twelve vendors, all twelve belong in the set. A ground truth set that’s 90% one vendor tells you that vendor still works and nothing else.
- Include the ugly ones on purpose. The handwritten gauge sheet, the JIB with the line items in a weird order, the run ticket that got scanned at an angle. Those are where extraction breaks first, so they’re the ones worth locking down.
- Store the source file, not just the text. If your OCR layer changes, you want to re-run from the original PDF, not from text that a previous version already mangled.
Five hundred is a starting number, not a rule. For a narrow extractor over one document type you might get honest coverage from 150. For a classifier deciding among a dozen categories you’ll want more. Size it to cover the variety, not to hit a target.
Regression suites catch what you break
Once you have ground truth, a regression suite is straightforward: run the whole set through the current pipeline, compare every extracted field against the confirmed answer, and report where they disagree. Field-level accuracy, not document-level. A ticket where you nailed nine of ten fields and dropped the charge code is not a 100% pass.
Run this before every change that touches extraction. That includes the changes people don’t think of as changes:
- A prompt edit. Tightening the instruction to fix one vendor’s dates can quietly break how the model reads another vendor’s totals. Prompt changes are the sneakiest regressions because they feel local and aren’t.
- A model version bump. Moving to a newer model is usually an upgrade on average and a downgrade on the three document types you cared about most. You only find out which by running the set.
- An OCR or preprocessing change. Swap the PDF text layer and every downstream extraction is now working from different input, whether or not you meant it to.
The value is boring and real. You make a change, you run 500 documents in a couple of minutes, and you see the exact fields that moved before the change ships. When a case gets fixed, it goes into the suite so it can’t silently break again later. Over a year that suite becomes the memory of every failure mode you’ve already paid for once.
Confidence scoring and where to route the doubt
We covered confidence scoring in human-AI workflows as the mechanism for deciding what a person looks at. It does double duty as a monitoring signal, so it’s worth being precise about what “confidence” even means here, because the naive version misleads you.
Asking a model “how confident are you, 0 to 1” gets you a number, and that number is worth less than people assume. Models tend to sound sure. A field the model completely fabricated will often come back at 0.95. Self-reported confidence is a weak signal on its own.
The stronger signals are structural. Did the extracted value pass a format check: is the API number the right shape, does the date parse, is the amount numeric? Do the parts reconcile: do the JIB line items sum to the invoice total, does the run ticket’s opening minus closing gauge match the stated volume? Did two independent passes agree? A field that fails a format check or breaks an internal sum is genuinely low-confidence regardless of what the model says about itself.
Route on the composite. High confidence and clean validation goes straight through. Anything that fails a structural check or lands below threshold goes to the review queue. The routing threshold should track the stakes, and upstream stakes are not uniform: a misread volume on a run ticket flows straight into revenue, so its bar sits higher than a chemical note on a field ticket that nobody allocates against.
Monitor the score distribution, not just the failures
Here’s where most teams stop short. They monitor whether the pipeline ran and whether individual documents cleared threshold. They don’t monitor the shape of the outputs over time, and that shape is where the quiet failure shows up first.
Go back to the charge-code story. The day that vendor changed their template, the extractor didn’t start erroring. It started returning low-confidence guesses on that vendor’s tickets and, worse, confident wrong values on the field it thought it still understood. The individual documents mostly cleared threshold. But the distribution moved. The share of that vendor’s tickets landing in review crept up. The rate of null charge codes ticked past its normal band. The average confidence on that document type sagged.
None of those trip a pass/fail alarm. All of them are visible if you’re watching the trend. The monitoring that catches silent drift looks at rates and distributions, not events:
- Auto-approval rate by document type, week over week. A vendor whose tickets used to sail through and now don’t is telling you something changed at the source.
- Null and empty-field rates per field. A field that’s null 4% of the time settling into 15% is a template change or a prompt regression, not noise.
- Confidence distribution, not just the mean. A growing left tail means a slice of your inputs stopped matching what the model was tuned on.
- Correction rate from review, by field and by vendor. This is your cleanest ground-truth-adjacent signal, because a human already decided the model was wrong.
You don’t need a platform for this. A dbt model over your extraction log and a weekly query that flags any metric drifting more than a set band off its trailing baseline will catch the template change before close does. Sampling a small slice of auto-approved output for a human to spot-check each week backs it up, because the point is to find the errors that cleared threshold without complaint.[1:1]
The upstream document types fail in specific ways
Generic LLM monitoring advice treats every document as interchangeable. Upstream documents aren’t, and knowing how each one goes wrong tells you which metric to watch.
Field tickets drift when a pumper’s routine changes or a new hand starts filling them out differently, and when handwriting quality varies enough that the same field reads clean one week and ambiguous the next. Watch per-writer or per-lease correction rates.
Run tickets are the highest-stakes and the most checkable. Opening gauge, closing gauge, temperature, BS&W, and net volume have to reconcile arithmetically. A run ticket where the numbers don’t foot is a hard error you can catch without any model help, and a rising rate of failed reconciliations is your earliest warning that extraction quality slipped.
JIBs are the template-change magnet, because they come from operators you don’t control and every operator formats theirs differently. When a non-operated partner changes their billing system, their JIB layout changes with it, and your extractor was tuned on the old one. The line-items-sum-to-total check is what saves you here. Monitor it per sending operator.
Completion reports are long, semi-structured, and full of fields that are legitimately absent on any given well, which makes null rates a noisy signal and reconciliation checks scarce. These lean harder on the regression suite and on human spot-checks than the others, because there’s less structure to validate against automatically.
Reliable versus quietly wrong
What separates a pipeline you can trust from one you can’t has little to do with the accuracy number on day one. Both can start at 95%. What matters is whether the pipeline tells you when it stops being right.
A reliable extraction pipeline fails visibly. When a vendor changes a template, the reconciliation checks start failing, the correction rate for that vendor climbs, and the confidence distribution shifts, all of it before the bad numbers reach close. The pipeline is wrong for an afternoon instead of three weeks.
A quietly wrong pipeline looks identical on every dashboard right up until accounting finds the discrepancy. It has no ground truth to check against, no regression suite to run before a prompt edit ships, and no distribution monitoring to notice the drift. It was probably built to hit an accuracy target once, in a demo, and then trusted indefinitely.
The work in this post is unglamorous and none of it is exotic. A frozen set of confirmed documents. A script that compares against them. A handful of trend queries over your own logs. A weekly spot-check. That’s the whole apparatus, and it’s the difference between finding out your pipeline broke on a Tuesday afternoon and finding out from your comptroller six weeks later.
If you built the pipeline from the last post in this series, this is the part that keeps it earning its keep. Build the ground truth set from the documents you’ve already reviewed, wire up the regression run before your next prompt change, and put the distribution queries on a weekly schedule. Do that and the next template change becomes a Tuesday, not a close-week fire drill.
Offline evaluation catches the changes you make; online evaluation on sampled production traffic catches the changes that happen to you, including provider model updates that ship without a version bump. See “Evaluation-First AI Product Engineering: Golden Sets, Drift Monitoring, and Release Gates for LLM Features,” Medium (2026). https://medium.com/@falvarezpinto/evaluation-first-ai-product-engineering-golden-sets-drift-monitoring-and-release-gates-for-llm-2c3bfb3f1e7b ↩︎ ↩︎