Why this matters
Sullivan & Cromwell just published the most interesting production LLM deployment I've seen in legal tech. Not because Agreement Analyzer automates contract review—plenty of legal AI tools claim that. But because partner John Savva describes an architecture where specialized agents review agreements in parallel, then adversarially challenge each other's conclusions before surfacing findings to human lawyers.
This isn't a chatbot over documents. It's a multi-agent system doing what lawyers actually do: forming hypotheses, stress-testing them, then deciding what survives scrutiny.
The write-up is light on technical details (it's a marketing page, after all), but what Savva reveals about the design is genuinely novel for production legal AI.
The architecture: parallel review plus adversarial validation
Here's how Agreement Analyzer works, per the source:
A lawyer uploads a draft agreement plus their own selected precedents and a "playbook" containing firm positions, client priorities, and market practice information. The tool then "performs multiple review cycles from different perspectives, synthesizes the outputs, helps identify potential issues and returns a memorandum and proposed tracked changes."
The interesting part is what Savva describes next: "an application in which agents develop conclusions and other agents challenge them in an adversarial process."
Specialized agents review the agreement in parallel for:
- Market deviations and risk allocation
- Interactions between clauses
- Ambiguities that could impact litigation
Then other agents challenge those findings. The final output distinguishes:
- Findings that survived adversarial challenge
- Findings that were dismissed
- Unverified issues retained with qualifications
Lawyers can then "inspect the issues, their supporting explanations, and the relevant precedents, then decide which findings warrant changes in the final markup."
Why adversarial agent review is interesting
Most legal AI I've seen follows a simpler pattern: single-pass analysis with maybe some retrieval augmentation. You ask questions, the model searches precedents, you get answers.
Adversarial validation is different. It mirrors how good lawyers actually work: you don't just identify issues, you stress-test whether they're real. Does this clause really deviate from market? Could opposing counsel argue the ambiguity cuts the other way? Is the risk material or theoretical?
Savva's framing—that some findings are "dismissed" while others are "retained with qualifications"—suggests the system is actually modeling epistemic uncertainty. Not just flagging everything that looks weird, but ranking findings by how well they hold up under challenge.
That's a harder problem than retrieval, and it requires models with stronger reasoning capabilities. Which brings us to the timing.
The technology evolution: from Codex to GPT-o3 Astra
Savva originally built Agreement Analyzer as "a single-user proof of concept built with Codex." Then he "collaborated with OpenAI on improving the quality of outputs, enhancing efficiency through parallel processing and transforming the application into a tool that could be used by multiple lawyers at the same time."
The write-up specifically calls out ChatGPT Enterprise as part of the current integration, alongside "Sullivan & Cromwell's workflows, curated firm precedents, negotiating playbooks."
And here's where the reasoning capabilities matter. Savva says: "Astra's really moved things forward in terms of the quality of the reasoning that's available, and that has been the bottleneck in the past."
(For the uninitiated: GPT-o3 Astra is what OpenAI calls the latest version of their reasoning-focused models, following the o1 and o3 series.)
The bottleneck wasn't retrieval. It wasn't speed. It was reasoning quality—the ability to evaluate whether a contract provision actually creates risk, whether a finding holds up under challenge, whether an ambiguity matters.
Transparency: making sources and uncertainty visible
Savva emphasizes something that often gets handwaved in legal AI pitches: "If an AI tool makes a claim about market practice, you need to be able to see the source of that information."
This matters because legal work has really high stakes for hallucination. If the model says "this deviates from market standard" but can't point to the precedents, lawyers can't use it.
Agreement Analyzer surfaces "the issues, their supporting explanations, and the relevant precedents" so lawyers can inspect the reasoning. The adversarial validation layer adds another form of transparency: you can see not just what the system found, but what survived challenge.
That's closer to how research memos actually work in BigLaw. You don't just list issues—you distinguish strong concerns from edge cases, flag where the law is unclear, show your work.
The shift from administrative automation to analytical work
Savva's framing of the evolution is sharp: "AI was initially used most for repetitive, administrative tasks like drafting email responses or checking calendars... Now, we're seeing many more opportunities for use in challenging legal tasks such as deep analysis of complex agreements."
This tracks with what I'm seeing across legal AI. Early tools focused on low-stakes tasks where mistakes were cheap. The shift to analytical work—agreement review, research, risk analysis—requires models that can reason, not just retrieve.
Savva's explicit: "It's not about replacing lawyers. It's about providing a tool for lawyers to do deeper analyses and find more authority and precedents to provide an even higher quality product for their clients."
That framing is both politically savvy and technically honest. Agreement Analyzer doesn't make final decisions. It does parallel multi-perspective analysis, adversarial validation, then surfaces findings with provenance so lawyers can decide.
What's actually novel here
Let me separate hype from substance.
Not novel: Using LLMs to review contracts. Lots of legal AI does this.
Not novel: Retrieval over precedents. That's table stakes.
Actually interesting: Adversarial agent architecture where findings are challenged before surfacing to humans.
Actually interesting: Explicit modeling of uncertainty (dismissed vs. verified vs. qualified findings).
Actually interesting: Integration of firm workflows, playbooks, and precedents into a multi-agent review process.
Actually interesting: A major law firm publicly discussing how reasoning model improvements (specifically o3 Astra) unblocked analytical legal AI.
The adversarial validation pattern feels like it generalizes beyond legal. Anywhere you need high-confidence analysis—due diligence, compliance review, technical audits—you probably want specialized agents challenging each other's findings rather than a single-pass system.
The production question I still have
Savaa says Agreement Analyzer "performs multiple review cycles from different perspectives" and uses "parallel processing" for efficiency. But he doesn't specify how many cycles, how long it takes, or what the cost structure looks like.
If you're running specialized agents in parallel, then adversarial challenge rounds, then synthesis, you're potentially looking at dozens of LLM calls per review. With reasoning models that's not cheap.
For Sullivan & Cromwell clients that's probably fine—legal fees dwarf API costs. But it's an open question whether this pattern scales to mid-market or in-house legal teams. Adversarial validation is expensive.
Still, if the reasoning quality actually justifies the cost—if it catches material issues that single-pass review misses—then it's worth it. Legal mistakes are way more expensive than LLM inference.
What Savva is exploring next
The write-up ends with Savva noting the firm "is continuing to explore tools shaped around specific legal tasks" and that "stronger analytical capabilities create an opportunity to support more of the research and writing process."
That direction makes sense. If adversarial agent review works for agreement analysis, the pattern should work for research memos, due diligence, regulatory analysis—anywhere lawyers need to form conclusions and stress-test them.
The key unlock, per Savva, is reasoning quality. "Astra's really moved things forward in terms of the quality of the reasoning that's available, and that has been the bottleneck in the past."
That's the bet: as reasoning models improve, they unlock analytical legal work that was previously too risky to automate. Not because retrieval was hard, but because evaluation was hard.
Why I find this exciting
Most enterprise LLM deployments I read about are either trivial (chatbots over docs) or vaporware ("AI will revolutionize everything"). Agreement Analyzer sits in a rare middle ground: a real production system doing legitimately hard analytical work with a novel architecture.
The adversarial validation pattern is genuinely interesting. The transparency around sources and uncertainty is how you actually build trust. And the explicit focus on reasoning quality as the bottleneck—not retrieval, not speed—feels like the right frame.
I'd love to see more technical details: how many agent rounds, what the prompts look like, how they handle disagreements between agents, what the error modes are. But even the high-level architecture is worth paying attention to.
If you're building LLM systems for high-stakes analytical work, Sullivan & Cromwell's approach is a better model than most of what's being hyped in legal AI.