Can LLMs Actually Reason? Dan Roth on Agents, Retrieval, and Evaluation

Oracle Chief AI Scientist Dan Roth joins Appen’s Jeanine Sinanan-Singh to examine what changes when AI researchers stop treating the model as the entire system.

Is an LLM reasoning when it produces a correct chain of thought? Or are we assigning a property to the model that actually belongs to a much larger system?

That distinction runs through this episode of The Data Layer.

Dan Roth, Chief AI Scientist at Oracle and Distinguished Professor at the University of Pennsylvania, joins Jeanine Sinanan-Singh, Director of GenAI Research at Appen, and host Jenkins, VP of Sales & Marketing at Appen for a technical discussion spanning agent architectures, formal reasoning, retrieval, external knowledge, realistic evaluation environments, runtime monitoring, and AI reliability.

A recurring idea emerges throughout the conversation: many of the hardest problems in AI are no longer model-only problems.

They are system problems.

Can LLMs actually reason?

Roth draws a sharp distinction between producing a sequence of tokens that arrives at the right answer and reasoning in the formal sense.

His definition is intentionally stricter: a reasoning problem has an answer that can be derived or proved through a formal system.

Under that definition, an LLM alone cannot solve every class of reasoning problem.

But an AI system can.

The distinction matters because an LLM can interpret a problem, translate it into the right representation, and delegate parts of the computation to specialized solvers. Planning, constraint satisfaction, formal verification, and other computational problems may therefore be better understood as capabilities of the complete system rather than properties of one model.

For AI researchers, the question shifts from:

“Can this model reason?”

to:

“Can this system reliably determine when reasoning is required, invoke the right mechanism, and verify the result?”

Why are agents becoming systems rather than models?

The current generation of AI agents increasingly separates the model from the larger architecture around it.

That architecture can include models, tools, retrieval systems, specialized solvers, planning mechanisms, memory, permissions, and orchestration.

Roth identifies several reasons this decomposition matters.

Some problems require computational capabilities that an LLM alone does not provide. Different tasks may require different contexts. Privacy and governance requirements may prevent every component from having access to the same information. And despite increasingly large context windows, simply putting everything into one context does not necessarily produce the strongest system.

The interesting research problem is therefore not just how capable the underlying model is.

It is how responsibilities should be distributed across the system.

Why isn’t retrieval solved?

Retrieval becomes especially important once the information an AI system needs is not contained in its model weights.

And much of the information enterprises care about will remain external: internal databases, private documents, email, medical information, financial data, operational systems, and other proprietary knowledge.

Roth argues that retrieval remains difficult because the way a user expresses an information need often differs from the way the relevant information is represented in the underlying source.

Enterprise data makes this worse.

Tables can have cryptic column names. Documents may contain important information buried in figures or tables. Relevant evidence may be split across multiple sources. Different sources may contradict each other.

One approach Roth discusses is semantic enrichment: adding descriptions, summaries, possible questions, metadata, and other semantic information to data before runtime so that the retrieval system has a better representation to search.

That also raises an important systems question:

What work should happen offline, and what work should an agent perform at runtime?

Doing everything at runtime is possible in some cases, but it can make an agent slower, more expensive, and harder to make reliable.

How can researchers test whether a model is reasoning?

Passing a reasoning benchmark is not the same as proving that a model is reasoning.

Jeanine Sinanan-Singh describes Appen’s approach as closer to falsification.

Researchers can perturb names, numbers, entities, or other features of a problem and observe whether the model still succeeds. These tests can expose brittle pattern matching.

But passing them still does not establish that the underlying behavior is genuine reasoning.

One direction Appen is exploring involves human-authored evaluation tasks where distractors cannot be identified through a simple surface pattern. Evaluators must first understand and solve the problem to determine which evidence is relevant.

This creates a harder test of whether a model can generalize beyond the structures it has learned to imitate.

How realistic should agent evaluations be?

Agent evaluations face another problem: the environment itself becomes part of the test.

A perfectly faithful copy of every production environment would be prohibitively expensive.

A perfectly clean benchmark, however, can remove exactly the conditions that cause agents to fail.

Real enterprise environments contain inconsistent schemas, legacy systems, conflicting information, poorly named fields, missing context, heterogeneous sources, and unexpected dependencies.

The goal is therefore not necessarily a one-to-one simulation of production.

It is to build enough diversity and realistic messiness into the evaluation environment that researchers can observe how the system behaves when its assumptions stop holding.

Why does “visibility of failure” matter?

One of Roth’s most important points concerns failures that users cannot see.

If a model gives the wrong answer to a question where you already know the answer, the failure is obvious.

But consider an agent performing financial analysis, querying multiple internal systems, or synthesizing information that the user cannot independently verify.

The same failure may be invisible.

This changes the evaluation problem.

Offline benchmarks remain useful, but deployed systems may also require runtime monitoring, verification, and mechanisms capable of detecting failures while the system is operating.

The research question becomes larger than accuracy:

How do we build systems that know when their own output may not be trustworthy?

Is agent evaluation becoming harness evaluation?

As agentic systems incorporate more components, evaluating only the final model output may tell researchers less about why a system succeeded or failed.

• Routing decisions matter.
• Tool selection matters.
• Retrieval matters.
• Context construction matters.
• Model selection matters.
• Verification matters.

Jeanine argues that agent evaluation is likely to move increasingly toward the harness itself: evaluating how the system orchestrates models and tools, not simply benchmarking the underlying model.

That transition could substantially change how AI teams think about evaluation infrastructure.

What comes after bigger models?

Roth expects raw model size to become less central over time.

Instead, system builders may increasingly optimize consistency, reliability, cost, storage, energy use, and the ability to detect failures.

That points toward a different definition of progress.

Not simply a model that can answer more questions.

A system that can determine what it knows, retrieve what it does not, delegate tasks appropriately, operate over messy real-world information, verify intermediate decisions, and make its failures visible.

For researchers working on agentic AI, the model is increasingly only one part of the research problem.

Watch the full conversation with Dan Roth, Jeanine Sinanan-Singh, and Brian Jenkins above.

Explore more AI research and evaluation work from Appen: https://x.com/AppenResearch