How we designed and tested a cheaper architecture for a client's AI classification pipeline, and what the tests found along the way.
Problem
Every accuracy gain made it more expensive to run
The client used a language model to assign business attributes to records at scale. Each proposed accuracy improvement raised the cost per record, and at that volume a gain that cannot be afforded across every record is not a gain.
Underneath sat a data problem. The authoritative categories the system leaned on were often stale or too general, so the model was being asked to infer something the source data no longer reliably carried. The brief was to raise accuracy while lowering the cost per record.
Prototype
Cheap, certain steps first, and the model only at the end
We designed a deterministic-first funnel. Cheap, certain steps run first: structured attribute maps, a vocabulary of legal purpose clauses and name patterns, then lightweight scoring with no language model. Only what survives both layers reaches the model, which handles the genuine residual.
We tested the funnel against the existing approach, red-teamed it with an independent second model before committing to it, and ran a cheap-model substitution test scored as match, partial, miss or hallucination rather than assuming the most expensive model was needed.
Proposal
What the tests found, and what we did not ship
Testing showed that a substantial share of the existing system's errors came from retrieving the wrong source material, not from misclassifying it, which moved the fix from prompt engineering to retrieval engineering. Most of the target attributes could be resolved from structured public data alone, at no model cost.
We proposed the funnel on what the tests showed. One part, an open-vocabulary semantic search layer, was designed, not shipped.