Status: Delivered. Described from our own delivery records. Client identity, brand and commercial terms are withheld. Figures, where given, cover the period stated and nothing beyond it.
Context
A research engagement rather than a production build.
The question was whether business operations on a CRM — the everyday work of finding, updating and acting on customer records — could be driven by prompt rather than by form, and which of the available architectures was actually worth taking further.
The problem
- The claims being made about agentic AI on business systems were mostly untested against real operational data.
- Comparing approaches informally produces an opinion, not a finding: without a fixed task set and a measurement method, whichever architecture is demonstrated last tends to win.
- Data sensitivity ruled out sending records to hosted models, which narrowed the option set to what could run locally.
- The cost of being wrong in production is high, and the cost of being wrong in a lab is not.
What we built
An integration between the operational CRM and locally hosted models
Through a tool-server layer, so an agent could read and act on real records under explicit permissions.
Retrieval-augmented storage over the CRM's own data
So answers were grounded in records rather than generated.
A comparison harness
Capable of running the same task set across retrieval-augmented against non-retrieval approaches; several orchestration pipeline variants; and a range of locally hosted models and vector stores.
How we delivered it
- 1
Frame the hypothesis and the success threshold
Before building — what “good enough to productionise” would mean, agreed in advance.
- 2
Build the integration layer
Against a masked copy of real data in an isolated environment.
- 3
Establish the task set
The operations an agent would need to perform, and the correct answer for each.
- 4
Run the comparisons
Across architectures, models and stores.
- 5
Write up the decision
What held, what didn't, and what would need to be true to take it further. In writing, including the negative results.
How it was phased
Indicative phasing for work of this shape. Actual duration varies with data quality, access and decision speed.
What the client keeps
- The prototype and its source
- The evaluation task set and harness
- Comparison results for every configuration tested
- A written decision record covering what was tried and why it was kept or dropped