The Setup: CFOs Want Answers, OpenAI Has a Scorecard
Sarah Friar, OpenAI's CFO, just published a manifesto on measuring AI ROI that reads like a blend of McKinsey deck and product launch. The core pitch: forget seats and licenses, measure "Useful Intelligence per Dollar" through four questions—how much useful work gets done, what successful tasks actually cost, how dependable AI is, and whether value compounds at scale.
The piece is polished, coherent, and absolutely a sales document. It name-drops GPT-5.6 (with three tiers: Sol, Terra, Luna), quotes benchmarks showing 54% fewer output tokens than competitors on coding tasks, and frames ChatGPT Work as the inevitable evolution of enterprise AI. This isn't analysis—it's positioning.
But here's the thing: the questions Friar poses are exactly the ones enterprises should be asking. The methodology is half-baked and self-serving, but the underlying framework matters. Let's pull it apart.
The Four Questions: Right Framing, Wrong Implementation
1. How Much Useful Work Gets Done?
Friar's first question is whether AI completes work that matters: customer issues resolved, code shipped, contracts reviewed. The shift from "tokens generated" to "outcomes delivered" is directionally correct. Tokens are an input metric; completed work is an output metric. You don't pay employees by the word—why measure AI that way?
The problem is "useful work" is deliberately vague. Friar gives examples—"a customer issue resolved," "a code change that passes its tests"—but conveniently skips the hard part: defining quality. A code change that passes tests but introduces tech debt? A customer issue "resolved" by a chatbot that frustrated the user into silence? These count as wins in Friar's framework but are failures in reality.
The finance team example is telling: ChatGPT Work handles data reconciliation, freeing humans for "judgment, creativity, and expertise." That's a real productivity gain if the reconciliation is error-free and the human trusts it enough not to re-check. Friar assumes dependability (question 3) but doesn't connect it back to useful work (question 1). The metrics are siloed.
2. What Does a Successful Task Actually Cost?
This is where the sales pitch gets loud. Friar argues that cost-per-token is misleading—what matters is cost-per-successful-outcome. A frontier model might cost more per token but succeed in one pass, while a cheaper model requires retries and human review.
Fine. That's economically sound. But then she pivots immediately to GPT-5.6's three-tier pricing (Sol, Terra, Luna) and a benchmark showing Sol uses 54% fewer output tokens than "another leading model" on the DeepSWE v1.1 coding benchmark, achieving 72.7% success versus Claude Fable 5's 69.9% at 36.2% lower estimated API cost.
Let's be clear: this is a product launch wrapped in methodology. The "estimated API cost" claim depends on OpenAI's pricing, which can change tomorrow. The benchmark is cherry-picked—coding tasks favor reasoning-heavy models. And "another leading model" is unnamed competitive shade.
The underlying point—that total task cost matters more than per-token cost—is valid. But measuring it requires tracking retries, human review time, rework, and failure modes. Friar's formula ("Add the full cost of completing the work. Count the tasks that met the required quality bar. Divide") is conceptually right but operationally handwaved. Who defines the quality bar? How do you instrument retry loops? What's the overhead of tracking all this?
3. How Often Does AI Get the Work Right?
Dependability is Friar's third pillar, and it's the most honest section. She breaks outcomes into three categories: ready to use, needs correction, needs escalation. This is useful! It maps directly to automation maturity—draft assistance versus autonomous action.
But then the corporate speak kicks in. "Dependability also requires clear boundaries"—what data the system can access, what actions it can take, when humans review. These are table stakes for production AI, not differentiators. The plug for ChatGPT Work's "security, privacy, compliance, and workspace-management foundation" is pure enterprise sales language.
The real insight buried here: dependability is economically valuable because it reduces review, correction, and retry costs. That's the link between questions 2 and 3. A 90%-accurate model that requires 10% human review is cheaper than an 80%-accurate model that requires 30% review, even if the latter has lower per-token costs. Friar gets this but doesn't quantify it.
4. Does Each AI Dollar Buy More Work as Usage Grows?
The fourth question is about scaling economics: does ROI improve as usage grows? Friar's answer is a hymn to compute efficiency—better models, smarter inference, purpose-built hardware, higher utilization. "The gains compound."
This is where the piece fully shifts from methodology to manifesto. She's describing OpenAI's flywheel: better infrastructure → better models → better products → more revenue → more compute investment. It's a theory of how OpenAI wins, not a framework for how customers measure ROI.
The claim that "when one layer improves, every product and customer can benefit" only works if you're locked into OpenAI's stack. If you're using their API, sure, you get model improvements automatically. But the "shared intelligence platform" pitch (ChatGPT, Codex, API, enterprise deployment) is vendor lock-in dressed up as ecosystem value.
The Missing Pieces: What the Scorecard Doesn't Measure
Friar's framework ignores three critical dimensions:
Opportunity cost. What else could that compute budget buy? What work isn't getting done because resources are allocated to AI experimentation? CFOs care about portfolio optimization, not just AI ROI in isolation.
Failure modes. The scorecard measures successful tasks but not the cost of failures. A hallucinated contract clause, a buggy code suggestion that ships to production, a customer service response that violates policy—these have asymmetric downside. One bad outcome can wipe out the value of a hundred good ones.
Organizational overhead. Implementing AI isn't just API costs and model licenses. It's prompt engineering, workflow redesign, change management, governance, monitoring, and incident response. Friar's "cost per successful task" formula gestures at "employee time" but doesn't account for the standing army required to operationalize AI at scale.
What CFOs Should Actually Do
If you're a CFO reading Friar's post, here's the translation:
- Ignore the product pitch. The
GPT-5.6benchmarks are marketing. Evaluate models empirically on your tasks. - Steal the questions. How much useful work, at what cost, with what dependability, and does it scale? These are the right axes.
- Build your own instrumentation. OpenAI won't give you the tooling to measure retry loops, human review time, or failure costs. You need to instrument that yourself.
- Start small and measure everything. Pick one workflow, define "done," track the three outcomes (ready to use, needs correction, needs escalation), and calculate total task cost including human time.
- Don't trust vendor-provided ROI. The economics only work if you can measure them independently.
The Verdict: Right Questions, Wrong Messenger
Friar's scorecard is corporate hopium—a sales pitch that feels like strategic guidance. But strip away the product plugs and benchmark flexing, and the underlying framework is sound. CFOs should shift from adoption metrics to outcome metrics. Cost-per-successful-task is more meaningful than cost-per-token. Dependability does have economic value.
The problem is that OpenAI has every incentive to make the scorecard easy to game in their favor. A framework that emphasizes frontier model capabilities and de-emphasizes failure costs naturally tilts toward expensive, high-capability models. The compute efficiency narrative serves OpenAI's moat, not customer ROI.
If you're implementing AI, use Friar's four questions as a starting point—but build your own instrumentation, define your own quality bars, and measure the costs she conveniently omits. The scorecard is right. The score is up to you.