The Future of AI Agents: Long Context, Benchmarks, and Real-World Deployment | The Data Layer Ep. 2

AI agents are being asked to do more than answer questions. They're expected to plan, execute, and operate with increasing autonomy - across coding, enterprise workflows, and eventually robotics. But according to Alex Whedon, co-founder and CTO of Subquadratic, and Sergio Bruccoleri, VP of Delivery at Appen, most agents today are held back by a problem that has little to do with intelligence: context management.

The two joined Appen's Karla Heredia on episode 2 of The Data Layer to discuss where agentic AI is heading, why current benchmarks fall short, and what it will take to build agents that can reason - not just retrieve - across millions of tokens.

Why Transformers Hit a Wall

Whedon traces the industry's current bottleneck back to a foundational tradeoff made in 2017. Transformer architecture became the industry standard but came with quadratic compute complexity, meaning that doubling input size quadruples compute requirements. In practice, that means models get less efficient, not more, as they're given more data to work with - a scaling problem the industry has lived with for nearly a decade.

Subquadratic's answer is a sparse attention approach designed to bring compute scaling back to linear, so cost per token stays flat as inputs grow, rather than compounding.

Agents Are Spending Most of Their Effort on Bookkeeping, Not Reasoning

One of the more striking findings discussed on the episode came from benchmarking a frontier model on TAU-Bench Pro. The model needed dozens of steps to complete tasks, and the overwhelming majority of those steps were spent re-reading and managing context rather than executing or reviewing work. Only a small fraction of the total steps involved actual task completion.

The implication: agents aren't necessarily failing because they lack reasoning ability. They're failing - or at least running slower and more expensively than they should - because they can't hold enough context to avoid constantly re-orienting themselves.

Long-Horizon Memory Is the Next Frontier

Beyond raw context size, Whedon pointed to long-horizon memory - the ability for an agent to retain useful information not just within a session, but across days or weeks - as critical for the next wave of agentic tools. This matters even more as agentic systems move beyond coding assistants (where engineers are accustomed to manually maintaining context files) into general workforce productivity tools, where users are far less willing to do that kind of manual upkeep.

The same principle extends to robotics. Whedon argued that general-purpose robotics will require dramatically more memory compression than today's demos suggest, since real-world tasks - like navigating a house or handling laundry - require remembering far more than the 10-15 second windows most current robotic demonstrations rely on.

How Do You Benchmark an Agent, Not Just a Model?

Sergio Bruccoleri outlined how evaluation needs to evolve alongside the models themselves. Testing agentic fitness means going beyond single-turn accuracy to assess whether a system can complete full workflows: choosing the right tools, making sound decisions across multiple steps, recovering from failures, respecting permissions, and knowing when to escalate to a human.

For long-context models specifically, that evaluation gets harder. It requires environments with genuinely large volumes of realistic content, plus expert-validated ground truth for multi-hop reasoning tasks - a combination that's difficult and expensive to produce well.

What the Benchmarks Show - and What They Don't

Appen independently benchmarked Subquadratic's model up to 12 million tokens, focused on a single-needle retrieval task adapted from the RULER benchmark suite, achieving 98% retrieval accuracy at that scale.

But both guests were careful to draw a distinction between retrieval and reasoning. Whedon noted that most existing long-context benchmarks - including RULER and the long-context reasoning benchmark from Artificial Analysis - top out well below the million-token range, and none yet reliably measure reasoning performance at multi-million-token scale. Building that benchmark, he said, is largely greenfield work the industry still needs to do.

Building Differently With Long Context

According to Whedon, removing context constraints changes how systems get built, not just how fast they run. He connected this to the "bitter lesson" - the long-standing observation that generalizable, less human-curated approaches tend to outperform heavily hand-engineered ones as scale increases.

In practical terms, that means enterprises may no longer need to chunk hundred-page financial documents into abstractions before analysis, or aggressively engineer retrieval pipelines for knowledge bases. With enough context capacity, more of that curation becomes unnecessary - though it also means enterprises can start extracting value from messy, unstructured data earlier in their AI adoption process, rather than needing expensive data transformation projects first.

The Underrated Opportunity: Tabular Data

Both guests pointed to structured, tabular data - spreadsheets, databases, legacy systems - as an underexplored opportunity relative to text-heavy use cases like RAG. Bruccoleri noted that the vast majority of enterprise data exists in tabular form, yet the industry has historically focused more on document-based retrieval. Text-to-SQL, in particular, remains a challenging but high-value use case, especially for enterprises with large, complex, multi-schema databases.

What's Next for Subquadratic

Looking ahead, Whedon said Subquadratic's near-term focus is enterprise commercialization, working with design partners who are already spending heavily on token usage and have use cases that would benefit significantly from long-context capabilities. Longer term, the company is exploring long-horizon agentic capabilities, selective vertical specialization based on client demand, and continued work on sample and memory efficiency beyond its current sparse attention model.

Key Takeaways

• Quadratic compute scaling is a structural limitation of transformer architecture that long-context-focused companies like Subquadratic are working to solve with linear-scaling alternatives.
• Many AI agents spend the bulk of their effort managing context rather than reasoning or executing - a problem long context aims to directly address.
• Long-horizon memory, spanning weeks rather than sessions, is emerging as a critical need for agents used outside of coding contexts.
• Current long-context benchmarks are still immature, particularly when it comes to measuring reasoning (versus retrieval) at multi-million-token scale.
• Independent, third-party benchmarking - as opposed to self-reported model claims - is increasingly important for enterprises evaluating new model architectures.
• Tabular and structured data represent a significant, underexplored opportunity relative to the industry's current focus on text and RAG-based approaches.

Listen to the full episode of The Data Layer, hosted by Appen, for the complete conversation with Alex Whedon and Sergio Bruccoleri.