How good is agentic AI at analytics, really?

The clearest example is text-to-SQL — the “ask your data a question in plain English” capability under most AI analytics pitches. The same class of models answered about 91% of questions correctly on a clean academic schema, and 17–21% against a real enterprise warehouse with thousands of columns, several SQL dialects, and the messy documentation any mature company actually has1.

The failure modes tell the story: schema-linking errors (the model can't reliably map “revenue” to the right column among thousands), dialect confusion, context overflow. Those are all data-environment problems, not model defects. A dashboard that's wrong 10% of the time, with no flag on which 10%, is worse than no dashboard — someone will make a call on it.

The best case — narrow, standardized, clean.The landmark result put an AI against 20 experienced corporate lawyers spotting issues in NDAs: 94% accuracy against the lawyers' 85% average, in 26 seconds versus 92 minutes5. That result holds up — but NDAs are about the most standardized contract there is, the task was closed-form issue-spotting against a fixed rubric, the inputs were clean, and the vendor funded the study. It is the strongest honest case for contract AI, built on exactly the conditions that rarely survive a real portfolio.

The reality check — open-ended, novel, messy.The first pre-registered, independent evaluation of commercial legal AI tested the retrieval-augmented tools marketed as “hallucination-free”3:

Grounding AI in a trusted document set cut hallucinations versus a raw model — it did not get rid of them. And when the vendors got measured, they disputed the methodology rather than publishing their own independent benchmarks. Vendor clause-extraction figures of 94%+ may hold for the clause types and clean inputs they tested on, but they're self-reported and measured on the vendor's terms, not your portfolio — treat them as claims, not facts.

What a wrong field costs. Poor contract management already erodes about 9% of annual revenue, up to 15% on large, complex portfolios2, and 71% of companies can't reliably locate at least 10% of their own contracts6. Hand that portfolio to an AI extractor and the errors cluster where they hurt most — renewal dates, amendments, party names, carved-out clauses. A blank renewal field gets flagged and checked; a wrong one looks complete, so the team trusts it and acts on it. Wrong data does more damage than missing data.

The link between input quality and AI output has moved from folk wisdom to something you can put a number on. In a controlled example, dropping placeholder zeros into a single column cut a model's predictive fit from R² 0.96 to 0.76 — no error thrown, the output just quietly got worse in step with the input9. That is the dangerous kind of failure: the kind that never raises its hand.

No prompt is clever enough to fix data that's incomplete, inconsistent, or undocumented. Agentic AI has no independent source of truth — only a probabilistic model of language plus whatever data you wire in. The ceiling on any factual task is set by that connected data. Improve it and the ceiling rises; leave it messy and no model, however sharp, gets past the ceiling the data sets.

Lab accuracy is necessary but not enough. About 95% of enterprise generative-AI pilots have shown no measurable P&L impact4; MIT blames a “learning gap” — poor integration, tools that don't adapt — more than raw model quality. Gartner predicts over 40% of agentic-AI projects will be canceled by the end of 20278. These are layers of the same wall: bad data caps accuracy, poor integration caps value even when accuracy is fine, and with no measurement you can't tell which problem you have.

The productivity mirage. In a randomized trial, experienced developers expected AI to make them ~24% faster and felt ~20% faster — but were measured 19% slower, most of it eaten by prompting, reviewing, and fixing the output7. Don't trust self-reported time savings: the verification work AI hands back to the human is real, usually invisible in a demo, and often big enough to wipe out the headline gain — doubly so in legal and contract work, where every output must be checked against a primary source.

Agentic AI is genuinely accurate on contract analytics inside one envelope: standardized instruments, common clause types, clean inputs, a defined rubric. Step outside it — novel terms, mixed legacy portfolios, open-ended judgment, dirty repositories — and accuracy degrades unpredictably, often without a sound. Since data integrity is the binding constraint and model horsepower matters much less, the order of operations is simple: diagnose data readiness, fix the gaps, then automate. Run analytics on an ungoverned repository and what comes out is confident, well-formatted error at scale.

Plotted against those two axes — task type and data condition — contract analytics sorts into a map you can defend to a GC:

The models are moving fast. Most studies here tested 2024 and early-2025 models; agentic scaffolding targets exactly these failure modes. Expect the numbers to improve — the open question is whether they improve enoughto clear the binary reliability bar high-stakes contract work demands. Improvement is near-certain; enough improvement isn't.

It only needs to beat the human baseline.For high-volume, low-stakes review, an AI that's faster and no worse than an average human is a real win. But a well-run legal function isn't benchmarking against the average solo practitioner, and the errors are asymmetric — one hallucinated obligation in a board-level analysis costs more than a hundred correct extractions save.

MIT blames integration, not data.Fair — it means “fix your data” is necessary but not sufficient. Treat data quality as the accuracy ceiling and integration as the value ceiling: two separate constraints, both real.

Any accuracy figure that wasn't measured on a real sample of your own contracts is a demo number dressed up as a deployment number. Before you spend on AI, diagnose whether the estate is even in shape for AI to be accurate — a readiness check is both the honest first move and the step the failure data says decides ROI. Then bake a controlled pilot into every evaluation, and keep a human in the loop on anything that matters.