JENS MARIONS

Thinking

Where Is the ROI?

Why the economic value of artificial intelligence may be harder to measure than the technology itself.

Jens Marions 6 min read

Every leadership team that has invested in generative AI eventually arrives at the same question. The licences are bought, the pilots have run, usage is climbing. Then someone in the room — often the CFO — asks what it has returned.

In most organisations, the honest answer is that nobody quite knows. That is usually treated as a measurement problem. I think it is more often an organisational one.

An old paradox in a new form

In 1987, the economist Robert Solow, reviewing a book about the future of manufacturing, observed that “you can see the computer age everywhere but in the productivity statistics.”1 Businesses had been investing in computers for years. The aggregate numbers had barely moved.

The remark gave a name to the productivity paradox. Productivity growth in the United States did eventually pick up in the second half of the 1990s. By then, firm-level research had found that the returns to computers were real but uneven — and that they depended heavily on what else a firm changed alongside the technology.2

The parallel with AI is obvious, and it invites a comforting conclusion: be patient, the gains will come. History supports a more demanding one. The gains did come, but not to everyone, and not from the technology alone.

What electricity teaches

The most useful parallel is older than the computer. In 1990, the economic historian Paul David compared the computer to the electric dynamo.3 Electricity was a transformative technology, yet its effect on factory productivity took decades to appear.

The delay had little to do with the technology. Early electrified factories used electric power much as they had used steam: one large central source driving a system of shafts and belts that ran through the building. The power source was new. The factory was not. Its layout, its workflow and the building itself were still designed around a single drive shaft.

The substantial gains arrived when manufacturers gave individual machines their own motors. Production could then be arranged around the flow of work rather than the line of a shaft — in single-storey plants, with reorganised production and different ways of supervising it. The value came from the redesign the technology permitted, not from the technology itself.

AI is closer to the dynamo than to a new application. It is a general-purpose technology: one that reaches into many tasks across many functions, and whose value depends on complementary changes around it.4 Most organisations are still at the central-shaft stage. We have connected a new source of power to the existing building.

The J-curve

Erik Brynjolfsson, Daniel Rock and Chad Syverson have given this pattern a shape.5 General-purpose technologies require large complementary investments — new processes, new skills, new business models — and much of that investment is intangible. It rarely appears as an asset in company accounts or national statistics, but it absorbs money and management attention.

The result is a J-curve. Early on, organisations are investing in things that measurement does not capture, so productivity looks weaker than it really is. Later, as those investments pay off, it can look stronger than it really is.

For a leadership team, the implication is uncomfortable. The period in which you are asked to justify AI spending is likely to be the period in which the numbers look worst. And a flat line on its own cannot tell you whether you are at the bottom of the J or simply on a flat line.

Adoption is not an outcome

Most AI dashboards measure adoption: licences assigned, active users, prompts per week, satisfaction scores. These are useful. They tell you whether people are using the tool. They do not tell you whether the organisation is better off.

It helps to make the chain between the two explicit:

  1. Capability. The tool is available, secure and works.
  2. Usage. People use it.
  3. Task productivity. Individual tasks get faster or better.
  4. Workflow performance. End-to-end processes improve: throughput, cycle time, quality, cost.
  5. Financial outcome. The improvement appears in revenue, margin, cost base or risk.

Value leaks at every link. The weakest is usually the one between task productivity and workflow performance. Half an hour saved drafting a document does not shorten a process whose real constraint is a weekly approval meeting. And time saved across thousands of people tends to disperse quietly into other work. Time saved is not money saved. It becomes money only when someone decides what happens to it.

The task-level evidence is real. One of the most careful field studies so far followed customer-support agents given a generative AI assistant. On average they resolved around 14% more issues per hour, and the largest gains went to the least experienced agents.6 That is a meaningful result. It is also a measurement of tasks in one well-instrumented workflow. Turning gains like that into a financial result still requires decisions about staffing, service levels and how work is routed — decisions the tool cannot make.

Why the technology arrives first

There is a structural reason organisations buy AI before they change around it. Buying is a procurement decision. It has a budget owner, a supplier and a deadline. Redesigning work is a leadership decision that cuts across functions, roles and decision rights. It is slower, more political and much harder to delegate.

So technology investment tends to precede organisational change, often by years. Brynjolfsson and Lorin Hitt reached a similar conclusion about computers: the firms that gained most were those that paired the technology with changes to work practices, skills and organisation.2

This is also why AI value cases written by technology functions are rarely convincing. A technology function can own the deployment. It cannot commit the business to running differently.

The value case belongs to the business

If the value depends on redesign, the value case must be owned by the people who control the redesign: the leaders of the functions where the work happens.

In practice, that means the leader of customer service — not the CIO — owns the case for AI in customer service, including the operational changes it assumes. The plan says what happens to freed capacity: slower hiring, higher service levels, work brought in-house or a smaller cost base. And the function reports against workflow and financial measures, not adoption.

Technology and finance still have essential roles. Technology makes the capability reliable, secure and well governed. Finance keeps the baseline honest and the attribution sceptical. But accountability for value sits where accountability for results already sits.

Assessing an AI-driven operating improvement

For a leader reviewing an AI proposal, a handful of questions do most of the work.

  • Which workflow, end to end? Name the process, its owner, and where it starts and ends. “Productivity” in general is not a workflow.
  • What is the baseline? Throughput, cycle time, error rate and unit cost before the change. Without a baseline, every later claim is an anecdote.
  • What else has to change? Roles, approvals, decision rights, controls, data. If the answer is “nothing”, expect modest gains.
  • Where does freed capacity go? A named decision, owned by someone with the authority to make it.
  • What does the J-curve look like here? When will costs peak, which leading indicators show the work is on track, and at what point would you stop?
  • What is the counterfactual? What would have happened anyway — through other improvements, seasonality, or people simply working harder while a pilot is being watched.

None of these questions is about the model. All of them are about the organisation.

The question behind the question

“Where is the ROI?” is a fair question, and leadership teams should keep asking it. But it is usually asked of the technology, when it belongs to the organisation around it.

The historical record suggests that the return on a general-purpose technology is largely the return on the organisational change it makes possible. Organisations that add AI to existing structures will capture some of that return. Those that redesign work around it — deliberately, with clear ownership and honest measurement — are likely to capture far more.

That is the difference between being AI-enabled and being AI-native. It may also be where the ROI has been all along.

Notes

  1. Robert M. Solow, “We’d Better Watch Out,” The New York Times Book Review, 1987 — a review of Stephen S. Cohen and John Zysman, Manufacturing Matters: The Myth of the Post-Industrial Economy. ↩

  2. Erik Brynjolfsson and Lorin M. Hitt, “Beyond Computation: Information Technology, Organizational Transformation and Business Performance,” Journal of Economic Perspectives 14, no. 4 (2000): 23–48. doi.org/10.1257/jep.14.4.23 ↩ ↩2

  3. Paul A. David, “The Dynamo and the Computer: An Historical Perspective on the Modern Productivity Paradox,” American Economic Review 80, no. 2 (1990): 355–361. ↩

  4. The concept is usually traced to Timothy F. Bresnahan and Manuel Trajtenberg, “General Purpose Technologies: ‘Engines of Growth’?,” Journal of Econometrics 65, no. 1 (1995): 83–108. ↩

  5. Erik Brynjolfsson, Daniel Rock and Chad Syverson, “The Productivity J-Curve: How Intangibles Complement General Purpose Technologies,” American Economic Journal: Macroeconomics 13, no. 1 (2021): 333–372. doi.org/10.1257/mac.20180386 ↩

  6. Erik Brynjolfsson, Danielle Li and Lindsey R. Raymond, “Generative AI at Work,” NBER Working Paper 31161 (2023). nber.org/papers/w31161 ↩