appresta.iq

Evidence review

How good is agentic AI at analytics, really?

A proof-point review of what autonomous AI actually delivers on business and contract analytics. It separates independent measurement from vendor marketing, and tests one premise: AI can only ever be as good as the data beneath it.

Independent studies weighted over vendor claims · figures current to mid-2026

01 / The headline

Accuracy is real, but it's bimodal and data-bound

There are two accuracy numbers for agentic AI, and they're miles apart. Which one you get depends entirely on the task. Give it a narrow, standardized job over clean inputs and it matches or beats trained professionals. Point it at open-ended reasoning over messy enterprise data and accuracy falls off a cliff, often by 60 to 70 points versus the demo.

What moves that number is mostly the data: how clean, structured, and governed it is, plus the workflow around it. Model quality matters far less than the pitch decks suggest. So the framing that AI can only be as good as the data it's built on turns out to be the whole ballgame, and the evidence backs it hard.

The evidence, in four numbers

The number you're shown, and the one you'll live with

Every vendor demo runs on the vendor's own clean data. These are the independent measurements: the gap between demo and deployment is set by your data.

90%1

Of organizations report having difficulty locating contracts (2021)

78%2

Of organizations don't systematically track contractual obligations (2021)

91% → 21%3

Text-to-SQL accuracy: the best AI agent on real enterprise databases vs. leading methods on a clean academic benchmark

17–33%5

Hallucination rate of legal-research AI whose vendors claimed few or no hallucinations, in the first preregistered evaluation

02 / The cliff

The gap between the benchmark and the boardroom

The clearest example is text-to-SQL: the “ask your data a question in plain English” capability under most AI analytics pitches. Leading GPT-4-based methods answer about 91% of questions correctly on a clean academic benchmark. On real enterprise databases, averaging about 744 columns each, across several SQL dialects and with the messy documentation any mature company actually has, the best AI agent in a peer-reviewed 2025 benchmark solved 21% of tasks3.

The failure modes tell the story. In the authors' analysis of 300 failures, the leading causes were erroneous data analysis (35.5%), wrong schema linking (27.6%: the model can't reliably map “revenue” to the right column among hundreds), and SQL dialect misuse (10.3%). The authors treat this as a real limit of today's models, and public leaderboard scores have risen since. The lesson hasn't changed: the model has to understand your schema, your dialect, and your data before its answer means anything. A dashboard that's wrong 10% of the time, with no flag on which 10%, is worse than no dashboard, because someone will make a call on it.

03 / Contract & legal AI

What the independent evidence says about contract work

The best case: narrow, standardized, clean. The landmark result put an AI against 20 experienced corporate lawyers spotting issues in five standard NDAs, and the AI came out ahead on both accuracy and speed6. But NDAs are about the most standardized contract there is, the task was closed-form issue-spotting, the inputs were clean, and the vendor ran the study, paid the lawyers, and built the answer key partly from the participants' best answers, the AI's included. It is the strongest honest case for contract AI, built on exactly the conditions that rarely survive a real portfolio.

The reality check: open-ended, novel, messy. The first preregistered evaluation of commercial legal AI research tools tested products from LexisNexis and Thomson Reuters whose vendors said retrieval avoided or eliminated hallucinations5:

ToolAccurateHallucinationNote
Lexis+ AI65%~17%Best performer tested; still needs verification
Westlaw AI-Assisted Research42%~33%Longer answers correlated with more errors
Ask Practical Law AI——Incomplete or ungrounded on 60%+ of queries
GPT-4 (general model)—~43%Baseline; the RAG tools beat it, none solved it

Grounding AI in a trusted document set cut hallucinations versus a raw model, but it did not get rid of them. And when the vendors got measured, they disputed the methodology rather than publishing their own independent benchmarks. Vendor clause-extraction figures may hold for the clause types and clean inputs they tested on, but they're self-reported and measured on the vendor's terms, not your portfolio. Treat them as claims, not facts.

What a wrong field costs. Ninety percent of organizations report difficulty locating their contracts1, and 78% say they don't systematically track contractual obligations2. Average contract value erosion, an industry benchmark, is 8.6%4. Hand that portfolio to an AI extractor and the errors cluster where they hurt most: renewal dates, amendments, party names, carved-out clauses. A blank renewal field gets flagged and checked; a wrong one looks complete, so the team trusts it and acts on it. Wrong data does more damage than missing data.

04 / The core question

“As good as the data” is measurable now

The link between input quality and AI output has moved from folk wisdom to something researchers measure. Spider 2.0 is the clearest case: leading methods score 91% on a clean academic database, and the best agent tested solved 21% of tasks on real enterprise databases3. The most dangerous failures are the quiet ones: an answer that looks complete and is wrong never raises its hand.

No prompt is clever enough to fix data that's incomplete, inconsistent, or undocumented. Agentic AI has no independent source of truth, only a probabilistic model of language plus whatever data you wire in. The ceiling on any factual task is set by that connected data. Improve it and the ceiling rises; leave it messy and no model, however sharp, gets past the ceiling the data sets.

05 / Deployment reality

What survives contact with an actual organization

Lab accuracy is necessary but not enough. Gartner predicts over 40% of agentic-AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, or inadequate risk controls7. Those reasons are layers of the same wall: bad data caps accuracy, poor integration caps value even when accuracy is fine, and with no measurement you can't tell which problem you have.

It isn't only AI. In a Gartner survey of 140 legal departments, 20% said their technology system met all of its original goals8. Gartner's explanation is readiness: the data the system needs has to be accessible and high-quality, the process around it standardized and followed, and the people trained and clear on their roles. It's the same order of operations this review arrives at: get the foundations ready, then add the tool.

The productivity mirage. Don't trust self-reported time savings; measure them. The verification work AI hands back to the human is real and usually invisible in a demo, and it counts double in legal and contract work, where every output must be checked against a primary source.

06 / Synthesis

The position, and how it maps to contract analytics

Agentic AI is genuinely accurate on contract analytics inside one envelope: standardized instruments, common clause types, clean inputs, a defined rubric. Step outside it (novel terms, mixed legacy portfolios, open-ended judgment, dirty repositories) and accuracy degrades unpredictably, often without a sound. Since data integrity is the binding constraint and model horsepower matters much less, the order of operations is simple: diagnose data readiness, fix the gaps, then automate. Run analytics on an ungoverned repository and what comes out is confident, well-formatted error at scale.

Plotted against those two axes, task type and data condition, contract analytics sorts into a map you can defend to a GC:

judgmentWhat you ask AI to doextraction
ungovernedData conditionclean, structured
ReliableModerate: verifyUnreliable
Illustrative positioning: a conceptual map, not measured coordinates.
TaskData conditionRealistic reliability
1Issue-spotting on standardized agreements (NDAs, common forms)Clean, standardizedHigh: at or above human
2Extraction of common structured fields (dates, parties, renewal terms)Machine-readable, consistentHigh, with QA sampling
3Clause extraction across a heterogeneous legacy portfolioMixed formats, OCR'd, inconsistentModerate: human in the loop
4Obligation or risk reasoning requiring legal judgmentEven on clean dataLow without verification
5Open-ended portfolio analytics ("aggregate exposure to X?")Ungoverned repositoryUnreliable: data-bound

07 / Where this could be wrong

The honest counter-arguments

The models are moving fast. Most studies here tested 2024 and early-2025 models; agentic scaffolding targets exactly these failure modes. Expect the numbers to improve. The open question is whether they improve enough to clear the binary reliability bar high-stakes contract work demands. Improvement is near-certain; enough improvement isn't.

It only needs to beat the human baseline. For high-volume, low-stakes review, an AI that's faster and no worse than an average human is a real win. But a well-run legal function isn't benchmarking against the average solo practitioner, and the errors are asymmetric: one hallucinated obligation in a board-level analysis costs more than a hundred correct extractions save.

MIT blames integration, not data. Fair. It means “fix your data” is necessary but not sufficient. Treat data quality as the accuracy ceiling and integration as the value ceiling: two separate constraints, both real.

08 / What to do with this

Insist on on-your-data benchmarking

Any accuracy figure that wasn't measured on a real sample of your own contracts is a demo number dressed up as a deployment number. Before you spend on AI, diagnose whether the estate is even in shape for AI to be accurate. A readiness check is both the honest first move and the step the failure data says decides ROI. Then bake a controlled pilot into every evaluation, and keep a human in the loop on anything that matters.

The pilot protocol

What to demand before you believe any headline number

  • 100+ of your own contracts, sampled across types and complexity, not the vendor's clean demo set.
  • Precision and recall reported per clause type, so you see where accuracy holds and where it breaks.
  • Money-and-deadline fields called out separately: renewal dates, amendments, party names, liability caps.
  • A verification budget: measure the human time to check outputs, and count it against the headline time saved.
  • A readiness baseline first: is the estate clean, complete, and structured enough for AI to be accurate at all?

Sources

  1. 1. Ninety percent of organizations report having difficulty locating contracts. EY and Harvard Law School Center on the Legal Profession, contracting report, June 2021: Draws on 1,000 law department and contracting leaders. 2021 data, so it predates current AI tooling. EY sells contract managed services, so the publisher has a commercial interest in the finding.
  2. 2. Seventy-eight percent of organizations say they don't systematically track contractual obligations. EY and Harvard Law School Center on the Legal Profession, contracting report, June 2021: Draws on 1,000 law department and contracting leaders. 2021 data, so it predates current AI tooling. EY sells contract managed services, so the publisher has a commercial interest in the finding.
  3. 3. In a peer-reviewed 2025 benchmark, the best AI agent tested solved 21% of text-to-SQL tasks on real enterprise databases, while GPT-4-based methods score 91% on the classic academic benchmark. Lei et al., Spider 2.0, ICLR 2025 camera-ready version: Database question answering, not contract review. The authors' analysis of 300 failures found erroneous data analysis (35.5%), wrong schema linking (27.6%), and SQL dialect misuse (10.3%). By 2026, vendor agent systems self-report about 76% on Spider 2.0-Lite on the public leaderboard, under a revised setup.
  4. 4. Average contract value erosion is 8.6%. World Commerce & Contracting and Deloitte, ROI of Contract Excellence research, published by Deloitte September 2023: Industry benchmark, not peer-reviewed. WorldCC defines value erosion as the deviation from expected results. Context only: never a savings, margin, or revenue claim. WorldCC's August 2025 white paper repeats the figure.
  5. 5. In the first preregistered evaluation of AI legal-research tools (peer-reviewed, 2025), tools from LexisNexis and Thomson Reuters, whose vendors said retrieval avoided or eliminated hallucinations, still hallucinated on roughly one in six to one in three queries. Magesh et al., Stanford RegLab / HAI, Journal of Empirical Legal Studies (2025): Legal research, not contract extraction. Grounding in a trusted document set reduced hallucinations versus a raw model; it did not eliminate them. A point-in-time test of 2024 product versions. “Hallucinated” includes answers that cite a source that does not support them.
  6. 6. In a 2018 study, an AI outscored 20 experienced lawyers on spotting issues in five standard NDAs (94% vs. their 85% average), in 26 seconds vs. 92 minutes, on the most standardized contract there is. LawGeex (2018), vendor-run study: Study run and published by the vendor (LawGeex), which paid the participating lawyers; the answer key was partly built from participants' best answers, including the AI's. This is AI's home-turf envelope, not a general contract-review result. Never cite the bare 94%.
  7. 7. Gartner predicts over 40% of agentic-AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, or inadequate risk controls. Gartner press release, June 2025: A forecast, not a measured result.
  8. 8. In a Gartner survey of 140 legal departments, 20% said their technology system met all of its original goals. Gartner press release, August 2025: Covers all legal technology, not AI or contract software alone. The link to readiness is Gartner's analysis; the full method is available to Gartner clients only.

Independent, peer-reviewed, and pre-registered sources are weighted above vendor-sponsored ones; vendor figures are labeled as claims. Figures current to mid-2026.