I did not build the ICLR 2026 analysis by sitting down with 9,000 papers and reading them like a normal person. That would have been noble, impossible, and not very useful. The point of this project was to test whether a conference at this scale can be turned into usable executive knowledge through an agentic pipeline: scraping, parsing, topic clustering, local model analysis, NotebookLM synthesis, Claude Code and Codex automation, knowledge-graph extraction, agentic orchestration, and then human editorial judgment at the end.

The Short Version

• The pipeline has four layers: ingestion and normalization, bulk analysis, narrative synthesis, and human editorial judgment. Each one can fail in a way the next cannot repair.

• Roughly 80 to 90 percent of the volume work ran locally, on my laptops, with Qwen and other smaller models. A frontier-only workflow would have been too expensive and too slow to iterate at this scale.

• The remaining 10 to 20 percent still needed cloud models or human supervision. This was not a fully autonomous pipeline, and claiming otherwise would make it less credible.

• The output is not just prose. It is source maps, evidence buckets, and thesis memos — structured enough to interrogate, and to reuse.

• The honest caveat: the method is fragile and tool-dependent. NotebookLM needs steering, transcripts need QA, local models need verification, and topic clusters need human curation.

Ingestion: The Unglamorous Foundation

The pipeline began with ingestion. Conference material had to be collected, converted, and normalized: abstracts, paper text where available, topic reports, transcripts, slides, audio-derived podcast material, and analysis reports. That work is not glamorous, but it is the foundation. A language model cannot reason over a conference if the source material is scattered, mislabeled, or trapped in formats it cannot reliably consume.

The transcripts in particular needed quality control. Some original OpenAI podcast transcripts were truncated or suspicious, so five topics were rerun with chunk validation. The local-model transcripts were useful as cross-checks, but the source authority map marks which transcript is authoritative for each topic. This is the kind of work a polished summary hides. It should not be hidden. If the input transcript is incomplete, the resulting claim can be wrong before the model starts reasoning.

Analysis: Local Models Did the Heavy Lifting

The next layer was analysis. I used a mix of local models and coding agents to organize the corpus into topic areas, generate reports, compare evidence, and produce summaries. Roughly 80 to 90 percent of this volume work ran locally on my laptops - Beauty and the Beasts - with Qwen and other smaller models doing the heavy repetitive passes. That mattered because a frontier-only workflow would have been too expensive and too slow to iterate at this scale.

The cloud models and coding agents were still necessary. Claude Code and Codex helped automate file processing, registry work, markdown conversion, source mapping, and planning. Frontier models were useful for hard synthesis and review. The honest split is that local systems handled the bulk volume, while 10 to 20 percent still required stronger cloud compute or more deliberate human supervision. This was not a fully autonomous pipeline, and claiming otherwise would make it less credible.

Synthesis: NotebookLM Was Useful, Not an Oracle

NotebookLM played a different role. It was useful for narrative synthesis: podcasts, topic walkthroughs, and repeated listening. It helped me hear the conference back to myself. But it was not the oracle. NotebookLM can miss important points, choose the wrong emphasis, and produce titles that skew pessimistic or clickbait. I changed titles because the generated versions often reached for drama instead of signal. That editorial override is not decoration; it is part of the filter.

What the Pipeline Produced

It produced executive summaries, topic reports, source maps, thesis memos, series architecture, and the six posts in this series. More importantly, it produced a way to interrogate the conference. I could ask which claims were first-party thesis, which were conference evidence, which came from podcast synthesis, and which were public company claims. That separation is the difference between "a model summarized ICLR" and "a source-grounded analysis exists."

The process still has no perfect name. "Conference distillation" is close. "Machine-readable knowledge extraction" is closer. The important distinction is that the output is not only for human reading. The goal is to create structured material that can be reused: by people writing essays, by agents building software, by knowledge graphs, by future conference comparisons, and by systems that need to reason over the research record.

The Caveat, and Why the Method Still Matters

That ambition creates a danger: overclaiming. I have not solved machine-readable knowledge extraction. The pipeline is fragile, tool-dependent, and full of editorial judgment. NotebookLM needs steering. Transcripts need QA. Local models need verification. Cloud models can still hallucinate. Topic clusters need human curation. The series titles and thesis architecture were not outsourced to a podcast generator because the whole point is to preserve judgment, not erase it.

Still, the method matters. A conference of this size is no longer readable by ordinary human attention alone. If we want executives, researchers, and builders to make decisions from the research frontier, we need pipelines that turn giant corpora into inspectable claims. That means not only summarizing papers, but preserving source authority, tracking uncertainty, separating evidence buckets, and leaving enough structure for later reuse.

The best way to read this series is therefore with two layers in mind. The first layer is the argument about ICLR 2026: filtering, data curation, verification, local models, and process-grounded AI. The second layer is the method that made the argument possible. If the method feels a little unfinished, that is because the field is unfinished. We are learning how to read conferences with machines while still keeping humans responsible for what the reading means.

The previous posts leave us with a practical business question. If cheap passive data is becoming less sufficient, and if agent learning increasingly depends on executable worlds, then who is going to manufacture the practice?

That is the training data business. Not the old business of scraping more text, labeling more examples, or buying a larger corpus. The new business is creating verified tasks in worlds where an agent can act, observe, fail, recover, and learn. The scarce object is not a document. It is a well-formed challenge.

The Short Version

If you only read one section, read this one.

• Most tasks are worthless for training. A task the model already solves is regression data. A task it cannot approach gives almost no signal, because pass/fail on something never passed is empty. The valuable task is the one the model just misses.

• That "just-missed band" is the real bottleneck. ICLR 2026 work says it directly: RLVR depends on well-crafted tasks with ground truth, and the difficulty of generated agentic tasks is hard to control. The frontier model may be expensive, but the frontier task generator is scarce.

• Owning a real environment is not enough — you need a difficulty dial. If a task is too easy, can you lengthen the reasoning path without changing the domain? Too hard, can you remove exactly one hidden dependency? Can you tell a missing fact from a bad tool call from a misread business rule?

• Ontology is a complexity dial. That is our synthesis. Once entities, relationships, constraints, permissions, state transitions, and exceptions are explicit, task generation becomes controlled perturbation of a formal world rather than prompt writing — and the same structure lets you verify final state, not just output text.

• The moat is not the PDF, the meeting note, or the raw log. It is the ability to turn domain structure into a stream of verified, calibrated tasks — and to know when a model is exactly one abstraction away from learning.

What's in This Post

Each section opens with the short version, then a deeper dive with the paper-level evidence. Read the short versions for the argument; stay for the deep dives when you want the receipts.

1. The Just-Missed Band — why most tasks produce no learning signal.

2. Environments Need a Difficulty Dial — where "environment" becomes an engineering discipline.

3. Ontology Is a Complexity Dial — how to make task difficulty adjustable, and verifiable.

4. What This Changes on Your Roadmap — the questions to ask instead of "what data do we have?"

The Just-Missed Band

The short version: For reinforcement learning, not every challenge is useful. A task the model already solves is mostly regression data — it confirms yesterday's model still works. A task the model cannot approach is also weak: if the only reward is pass or fail and the model never gets close, the learning signal is almost empty. The valuable task is the one the model just misses: hard enough to expose a real capability gap, structured enough that the model can make partial progress, trigger useful feedback, receive partial credit, and try again.

Deeper dive: the conference evidence that this band is scarce

You can sometimes brute-force your way past a too-hard task with enormous sampling budgets, but that is not a business model most companies can afford.

Several ICLR 2026 papers suggest that this band is becoming a bottleneck. Don't Just Fine-tune the Agent, Tune the Environment1 starts from the problem directly: LLM-agent development is hampered by scarce high-quality training data, synthetic SFT data can overfit, and standard RL struggles with cold start and instability in complex multi-turn tool use. Its answer is not simply "more data." It is Environment Tuning1: a structured curriculum, environment augmentation, and fine-grained progress rewards.

Supervised Reinforcement Learning2 makes the same point from another angle. RL with verifiable rewards can fail when correct solutions are rarely sampled even after many attempts. The paper's response is smoother step-wise supervision, giving the model richer signals even when all rollouts are wrong. In other words, "wrong" is too coarse a category. The training system needs to know how the model was wrong.

Search Self-Play3 puts the business problem in one sentence: RLVR depends on well-crafted task queries and ground-truth answers, while the difficulty of generated agentic tasks is hard to control. That is the strategic bottleneck. The frontier model may be expensive, but the frontier task generator is scarce.

Environments Need a Difficulty Dial

The short version: Real environments help, but they do not solve the problem by themselves. You can buy real data, collect logs, or build a cloud lab. But once you have it: what is the difficulty dial? If the task is too easy, can you lengthen the reasoning path without changing the domain? If it is too hard, can you remove exactly one hidden dependency? If the agent fails, can you tell whether the failure came from a missing fact, a wrong traversal, a bad tool call, a state update error, or a misunderstanding of the business rule? This is where "environment" stops being a generic word and becomes an engineering discipline.

Deeper dive: four dials from ICLR 2026

AgentGym-RL4 treats agent training as long-horizon decision making across realistic environments. Its staged interaction approach begins with short-horizon interaction and progressively expands the interaction length so agents can stabilize before deeper exploration. That is a complexity dial over time and horizon.

CodeGym5 transforms static coding problems into interactive, verifiable, controllable multi-turn tool-use environments. That is a complexity dial over tools, executable steps, and workflow shape.

BIRD-INTERACT6 shows the same idea in database work. It couples databases with a hierarchical knowledge base, metadata, a user simulator, CRUD tasks, execution feedback, and executable tests. That is much closer to enterprise reality than a single text-to-SQL prompt. It is also a reminder that the world is not just the database. The world is the database plus the business semantics plus the user plus the allowed actions plus the success condition.

OpenApps7 takes the UI-agent version of the same problem. Instead of testing an agent on one fixed clone of an app, it generates thousands of configurable app variations across messenger, calendar, maps, and related interfaces. Reliability changes dramatically across those variations. That matters because a fixed environment can hide fragility. A configurable environment can reveal it.

These papers point toward a different view of data. Data quality is not only whether the tokens are clean. It is whether the task produces useful learning pressure. Scaling Laws Revisited8 makes the broader scaling point explicit by adding a data-quality parameter to the Chinchilla-style view of model size and dataset volume. Why Less is More (Sometimes)9 studies curation by difficulty and correctness, showing conditions where smaller curated datasets can outperform full datasets. The point is not that scale stopped mattering. The point is that scale without quality, difficulty, and verifier design is an incomplete strategy.

Ontology Is a Complexity Dial

The short version: That is our synthesis. An ontology is not the only way to model a world, but for enterprise worlds it is one of the cleanest ways to make task difficulty adjustable. A semantic model gives you entities, relationships, constraints, permissions, state transitions, exceptions, and business rules. Once those are explicit, task generation becomes less like prompt writing and more like controlled perturbation of a formal world — and the same structure lets the training system grade final state against the rules of the environment, not just check whether an output string looks right.

Deeper dive: turning the dial in both directions

You can make a task easier by shortening a traversal, removing an exception, reducing the number of entities, revealing an intermediate state, narrowing the action set, or making the success condition more local. You can make it harder by adding a dependency, introducing a conflicting constraint, requiring reconciliation across two systems, hiding a needed fact behind a tool call, adding temporal state, or asking for a final action that must satisfy several business rules at once.

That is the difference between a static collection of examples and a training-data machine. A static collection has fixed coverage; a machine can generate another task at the right level.

The ontology also helps with the verification problem raised in the previous post. If the world is formal enough, the training system can check not just whether an output string looks right, but whether the final state satisfies the rules of the environment. In principle, when connected to executable tests or state checks, it can grade intermediate steps, expose partial progress, and distinguish a semantic failure from a formatting failure. The richer the world model, the more precise the feedback can become.

What This Changes on Your Roadmap

The short version: At first glance, "making tasks" sounds like routine production work. In reality it is the work of encoding a domain so that an agent can practice inside it. Most companies do not need to own generic generation. They need to own the worlds where generic models become useful. So the roadmap question shifts: not "what data do we have?" but "what world can we simulate, what actions are allowed, what state changes, what does success mean, and how do we generate the next task the model just misses?"

Deeper dive: what the moat actually is, with an example

The moat is not the PDF, the meeting note, or the raw log. The moat is the ability to turn domain structure into a stream of verified, calibrated tasks. It is the ability to know when a model already understands a workflow, when it is outside the relevant capability range, and when it is exactly one abstraction away from learning.

A tax business setup makes the pattern concrete. Taxes are full of entities, rules, exceptions, thresholds, documents, jurisdictions, and stateful decisions. That makes them painful for static prompting and natural for an executable world. The interesting question is not whether a model can answer one tax question. It is whether we can build a dial that creates the next thousand tax tasks at the right difficulty.

That is the new training data business. Not more data for its own sake: better worlds, better tasks, and better feedback.

Source Notes

The previous post argued that the next scarce asset is not generic text or generic generation. It is the ability to build worlds. This post is about what happens once those worlds exist: agents need to practice in them, fail in them, and be graded by them.

The Short Version

If you only read one section, read this one.

• A gym is a world you can run. The agent acts, the world changes, and a reward or verifier says whether the action helped. In modern agent systems the gym is a terminal, a browser, a database, a workflow, a simulated lab, or a container full of tools.

• The grader is not a scoring script — it is part of the world. Benchmark design is now partly security work: a good task has to resist agents that hard-code answers, delete failing tests, or satisfy the grader while violating the intent.

• The answer is not the artifact; the trace is the artifact. A correct final answer can come from broken reasoning, so a cluster of ICLR 2026 papers moves reward inside the trace — step-level logical supervision, Lean as a process oracle, step-wise expert rewards, and detectors for reward hacking hidden behind benign-looking chains of thought.

• The next strategic skill is specifying correctness. Requirements, constraints, invariants, permissions, completion criteria, failure conditions. In a world of learning agents, whoever can define what counts as correct may have more leverage than whoever writes the code that attempts the task.

• Enterprises need a different verification language than Lean. A business is data and rules — customers, contracts, approvals, exceptions — not a theorem. Executable ontologies are one plausible candidate: a gym with semantics and a verifier with memory.

What's in This Post

Each section opens with the short version, then a deeper dive with the paper-level evidence. Read the short versions for the argument; stay for the deep dives when you want the receipts.

1. Gyms, Rollouts, and Graders — the vocabulary, and what a modern agent task actually contains.

2. The Grader Is Part of the World — why fair grading turned into security work.

3. The Answer Is Not the Artifact — the process-supervision cluster at ICLR 2026.

4. Specifying Correctness — Lean, and the skill that outlasts writing the code.

5. The Enterprise Turn: Executable Ontologies — why business worlds need a different language.

6. The Bottom Line — RL needs gyms, gyms need graders, graders need specifications.

Gyms, Rollouts, and Graders

The short version: A gym is a world you can run: the agent takes an action, the world changes, the agent observes the result, and a reward or verifier says whether the action helped. The sequence of thoughts, tool calls, observations, file edits, queries, errors, and recoveries is the rollout, or trajectory — the thing the model actually did, not the story it tells afterward. Modern agent tasks package this as a container, an instruction, tests, and a reference solution.

Deeper dive: what a Harbor-style task contains

In simple games this structure is obvious. In modern agent systems, the gym may be a terminal, a browser, a database, a calendar-style workflow, a simulated lab, or a Docker container full of tools and services.

Terminal-Bench1, an ICLR 2026 paper, describes tasks with a containerized environment, an instruction, tests, and a reference solution. The paper uses the Harbor task format and harness. In the Harbor documentation2, the same idea appears as instruction.md, task.toml, an environment/ directory, a solution/ path, and tests/. The agent receives the task, acts inside the environment, modifies files or services if needed, and leaves behind a final state. The grader then checks whether that final state satisfies the task.

This design is straightforward in outline until you try to make it fair.

The Grader Is Part of the World

The short version: The grader must know whether the agent solved the task or merely passed the tests. A task is not good if the agent can inspect hidden future commits, exploit a fixture, hard-code the answer, delete a failing test, or produce a file that satisfies the grader while violating the real intent. Terminal-Bench devotes real machinery to specificity, solvability, and anti-cheating checks. That is the first ICLR lesson for enterprise AI: a grader is not a scoring script. It is part of the world.

Deeper dive: benchmark design as security work

Terminal-Bench1 asks whether a task is specific, solvable, and resistant to cheating, and the paper describes integrity and anti-cheating checks as a first-class part of task construction. The reason is straightforward once you take agents seriously as optimizers: any gap between "what the grader measures" and "what the task intends" is an exploitable surface, and a capable agent will find it faster than a human task author will notice it.

The Answer Is Not the Artifact

The short version: Even if the final answer is correct, the reasoning may be wrong. A program can pass a test because two bugs cancel. A model can produce the right SQL result after misunderstanding the schema. An agent can repair a service by disabling a feature. A robot can hit a goal state through an unsafe path. In each case the final state is real, but it does not tell us whether the process should be learned, repeated, or trusted. A cluster of ICLR 2026 papers responds by moving reward inside the trace.

Deeper dive: the process-supervision cluster

LogicReward3 says this directly. Existing training methods often depend on outcome-based feedback, and that can produce correct answers with flawed reasoning. LogicReward3's response is to enforce step-level logical correctness with a theorem prover. It converts natural-language reasoning into formal structure, softens the brittleness of exact formalization with a method it calls Autoformalization with Soft Unification, and then gives the model reward at the level of logical steps rather than only the final answer.

Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning4 is the optimistic side of the story. It argues that RLVR can encourage correct reasoning even when rewards are based only on answer correctness, and it introduces CoT-Pass@K to account for both final answers and intermediate reasoning. This matters: RLVR is powerful precisely because it turns verifiable tasks into training signal.

But other papers show why the final answer is not enough. Process-Verified Reinforcement Learning for Theorem Proving via Lean5 uses Lean itself as a symbolic process oracle. Lean does not merely say whether the proof eventually worked. Its elaboration can mark locally sound tactics and the earliest failing step, giving dense, verifier-grounded credit signals rooted in type theory. That is a particularly informative grader: it knows the target, knows the formal rules, and can localize failure.

Supervised Reinforcement Learning6 makes the credit-assignment problem even clearer. It points out that RLVR can fail when correct solutions are rarely sampled, then proposes smoother step-wise rewards based on expert actions. The important line for business is that richer learning signals can help even when all rollouts are incorrect. A rollout that gets 98 percent of the way there should not be indistinguishable from nonsense. If we cannot give partial credit, we waste training compute and lose diagnostic information.

TRACE addresses the failure mode. In Is it Thinking or Cheating?7, the authors study implicit reward hacking: the model exploits a loophole in the reward function while its chain-of-thought appears benign. TRACE detects this by truncating reasoning and checking how quickly a model can still obtain reward. If a shortcut gives the reward with surprisingly little reasoning effort, the model may be gaming the world rather than solving the task.

From Verifiable Dot to Reward Chain8 gives the broader framing. RLVR works well in math and code when a final answer can be checked, but single-dot supervision becomes inefficient and reward-hacking-prone when tasks are open-ended. Their proposed move is a reward chain: more ordered, structured feedback than one bit at the end.

This is the ICLR 2026 verification story in one sentence: the answer is not the artifact; the trace is the artifact.

Specifying Correctness

The short version: Lean is the sharpest example of a machine-checkable world because mathematics has unusually clean rules — a tactic either elaborates or it does not. The broader lesson is not that Lean should replace mainstream programming. It is that knowledge work can move from prose that sounds right to artifacts machines can check, and that the next strategic skill may be the ability to specify correctness: requirements, constraints, invariants, permissions, completion criteria, and failure conditions.

Deeper dive: from prose to checkable artifacts

A proof assistant can say what follows from what. This is why the recent book The Proof in the Code9 is relevant here. Kevin Hartnett traces the birth and rise of Lean as a proof assistant, but the broader relevance is not only mathematical culture. It is the idea that knowledge work can move from prose that sounds right to artifacts that machines can check.

Python and Java still matter. SQL still matters. But in a world of learning agents, the person who can define what counts as correct may have more leverage than the person who only writes the code that attempts the task.

The Enterprise Turn: Executable Ontologies

The short version: Lean is beautiful for mathematical and logical worlds, but an enterprise is not a theorem. It is data: customers, suppliers, products, contracts, inventory, invoices, policies, approvals, events, exceptions, forecasts, and decisions. That world's language has to compute over data and express rules, not just proofs — closer to SQL, but with semantics, constraints, and executable business logic. An executable ontology fits that description: a gym with semantics and a verifier with memory.

Deeper dive: the conference evidence, and a procurement example

BIRD-INTERACT10 is the conference evidence that this enterprise-adjacent world is already visible. It takes text-to-SQL out of the static prompt setting and puts it into a dynamic environment with databases, hierarchical knowledge bases, metadata, a user simulator, CRUD tasks, execution feedback, and executable tests. That is much closer to a real business assistant. The agent must ask questions, inspect state, recover from mistakes, and satisfy operational requirements.

CodeGym11 and AgentGym-RL12 show the same pattern from different directions. CodeGym11 converts static coding problems into interactive, verifiable, controllable multi-turn tool-use environments. AgentGym-RL12 treats multi-turn interaction across diverse environments as the substrate for training LLM agents. The pattern is not "more prompts." The pattern is executable worlds with verifiers.

This is where ontologies re-enter the story. An executable ontology is a model of the entities, relationships, rules, constraints, and actions of a domain that a machine can run against. The previous post framed digital twins, ontologies, and gyms as three dialects of the same idea. One way to put the synthesis is this: an executable ontology can function as a gym with semantics and a verifier with memory.

That view is already being built on: executable ontologies for decision worlds, where the agent must reason over data, rules, relationships, and constraints rather than only generate text. The point is not to replace Lean. It is to recognize that enterprises need a different verification language: one that can express the correctness conditions of a business process, compute over live data, and explain why an action is allowed, useful, or wrong.

Imagine training an agent to manage procurement. A Harbor-style task can give it a container and a grader. A Lean-style proof can teach us the value of formal verification. But the procurement world itself needs suppliers, contracts, budgets, approval chains, substitutions, delivery risks, tariff rules, and historical outcomes. That is ontology territory. The grader should not merely ask whether the final purchase order exists. It should know whether the agent followed policy, respected budget authority, handled exceptions, chose valid substitutions, and produced an auditable decision trace.

The Bottom Line

That is the practical synthesis. RL needs gyms. Gyms need graders. Graders need specifications. Specifications need languages. For math, Lean is one answer. For code, tests and type systems are part of the answer. For enterprise AI, executable ontologies are one plausible candidate because the enterprise world is made of data and rules.

The future of agent training will not be only better models. It will be better worlds that can say, with precision, what happened inside the rollout.

Source Notes

Harbor-style tasks, rollouts, and grading

• Terminal-Bench PDF, for Harbor task format, task formulation, final-state tests, specificity, solvability, integrity, and anti-cheating audit details

RLVR, process verification, and reward hacking

• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward, ICLR 2026 Poster

Background sources requested for this topic

The previous post ended with a question: if interaction data is the scarce resource, how do we create the worlds that produce it?

This is where ICLR 2026 becomes much more practical than the phrase "AI research conference" suggests. The strategic question is not whether models can generate more candidates. They can. It is whether those candidates can survive a world. Can the molecule be synthesized? Can the antibody bind? Can the robot manipulate the object? Can the database assistant recover after an execution error? Can the calendar agent respect a policy, a user's preference, and a conflicting meeting constraint?

The model does not need only a bigger imagination. It needs a world that can push back.

The Short Version

If you only read one section, read this one.

• World-building, not scraping, is the new bottleneck. Generating candidates is cheap; narrowing them to a set that can be manufactured, tested, measured, and fed back is not. The field is building simulated laboratory worlds before claiming autonomous discovery — and agents given real scientific software still reach only 15% success.

• Physical labs don't generalize. You can rent a biology lab. You cannot cheaply create a new company, a port, a supply chain shock, a regulatory crisis, or zero gravity on demand. The real problem is broader: making domains executable.

• Three ways to do it. (1) Encode knowledge into the world so models don't rediscover physics from scratch. (2) Curate the data rather than piling up tokens — quality is now a variable in the scaling equation, not background. (3) Build executable models of the domain: digital twins, ontologies, and gyms.

• Those three words describe one thing: a world you can run. Enterprises say digital twin, knowledge-representation people say ontology, RL people say gym. The "game" is now a calendar, a SQL database, a browser, a lab bench, a robot workcell, or a procurement process.

• This reframes the moat. Generic generation keeps getting cheaper. The durable capability is knowing your domain well enough to build the world in which intelligence can practice.

What's in This Post

Each numbered direction opens with the short version, then a deeper dive with the paper-level evidence. Read the short versions for the argument; stay for the deep dives when you want the receipts.

1. Why Physical Labs Are Not Enough — where automated science stops generalizing.

2. Encode Knowledge Into the World — symmetry, boundary conditions, and world models that know things.

3. Curate the Data, Not Just the Dataset — quality as a first-class scaling variable.

4. Build Digital Twins, Ontologies, and Gyms — the environment turn, and what it means for enterprises.

Why Physical Labs Are Not Enough

The short version: AI4Science makes the bottleneck visible. The automated lab — model proposes, robot executes, instruments measure, measurement becomes new data — is real enough to shape the research agenda, but ICLR's clearest signal is more careful: the field is building simulated laboratory worlds first. And physical labs only cover part of the need. We cannot cheaply instantiate a company, a port, a supply chain shock, or a regulatory crisis whenever we need training data. The world-building problem is bigger than robotics.

Deeper dive: simulated labs and the 15% ceiling

AutoBio1 is a simulation and benchmark for robotic automation in digital biology laboratories. It digitizes lab instruments, adds specialized physics plugins, renders dynamic instrument interfaces and transparent materials, and evaluates biologically grounded manipulation tasks. The headline is not "robots can now do all biology." It is that scientific automation needs a structured, high-precision world before the model's actions mean anything.

ScienceBoard2 makes the same point from the software side of science. It gives agents realistic scientific workflows with professional software across domains such as biochemistry, astronomy, and geoinformatics. The benchmark has 169 human-curated and validated tasks, and current agents reach only 15 percent overall success. That number is useful because it keeps demos in perspective. An agent that can describe science is not yet an agent that can do science inside a real workflow.

Even when a world is physically possible, collecting enough safe, diverse, repeatable interaction data may be too slow or too expensive. There are three main ways to make domains executable instead.

1. Encode Knowledge Into the World

The short version: Reduce the data you need by putting more of the domain into the model, simulator, or environment itself. Physics is the cleanest case because physics has laws: if a system must conserve mass, obey a boundary condition, or respect a symmetry, the model should not rediscover that from scratch. ICLR evidence is pointed — equivariant models that leverage task symmetry scale better, and for neural operators, small amounts of boundary-diverse data beat large amounts of same-family data. If the missing variable defines the world, volume inside the wrong world does not fix the problem.

Deeper dive: symmetry, boundaries, and world models

In Scaling Laws and Symmetry3, the authors study neural force fields for interatomic potentials and show that architectures with more symmetry expressivity have different scaling exponents. Equivariant models that leverage task symmetry scale better than non-equivariant models. The operational lesson is important: as systems scale, we should not simply leave fundamental inductive biases such as symmetry for the model to discover.

Boundary conditions tell the same story in a different dialect. Neural Operators Fail to Generalize Across Boundary Condition Families4 shows that a neural operator trained on one boundary-condition family can look good in distribution and then fail badly when the boundary family changes. In the paper's experiments, adding much more same-family training data helps less than adding small amounts of boundary-diverse data.

Robotics gives the embodied version of this lesson. WorldGym5 treats a world model as an environment for policy evaluation because real robot testing is costly and handcrafted simulators are hard to improve. Ctrl-World6 makes the same economic point: rigorous robot evaluation requires many real rollouts, and systematic improvement requires corrective expert data. Their proposed alternative is a controllable world model in which policies can roll out in imagination space, with synthetic successful trajectories used to improve the policy.

This is not a claim that simulation has solved robotics. WorldGym5 explicitly notes that highly realistic object interaction remains hard. The point is more useful: when real interaction is expensive, the world has to carry domain knowledge. It must encode geometry, contact, action, time, failure, and reward in a way that produces useful pressure on the policy.

That is the first direction: build worlds that know things.

2. Curate the Data, Not Just the Dataset

The short version: Stop treating "data" as a scalar. Chinchilla-style scaling laws captured a real relationship, but they treated quality as background. It isn't anymore — recent work adds an explicit data-quality parameter to the scaling equation, and higher-quality data can reduce model size and compute requirements. For enterprises this matters because your useful data is not the web. It is policy, exceptions, procedures, approvals, contracts, and outcomes: compressed world knowledge. But quality is not universal, so curation has to be tied to the world being built.

Deeper dive: quality as a scaling variable

Scaling Laws Revisited7 makes this explicit. The paper extends the Chinchilla framework with a dimensionless data-quality parameter, Q, so loss can be modeled as a function of model size, data volume, and quality. In its experiments, higher-quality data can reduce model size and compute requirements. That does not mean "small data beats big data" as a slogan. It means the unit is not just the token. The unit is useful signal.

Why Less is More (Sometimes)8 attacks the same issue from theory. It asks when it is better to use less data and gives conditions under which curated subsets can outperform full datasets. The important business lesson is not that curation always wins. It is that the answer depends on data quality, task difficulty, noise, and the selection strategy. Once those variables matter, the company that knows which examples belong in the world has an advantage over the company that only owns a larger pile.

This matters for enterprise AI because most useful enterprise data is not "the web." It is policy, exceptions, procedures, approvals, customer histories, product hierarchies, supplier constraints, contract terms, taxonomies, SLAs, decisions, and outcomes. Much of it is boring in exactly the way valuable data is boring: it encodes how the organization actually works. A manual, a textbook, a protocol, or a policy document is not merely text. It is compressed world knowledge.

The catch is that quality is not universal. A clean textbook may be high-quality data for a tutor and weak data for an agent that must operate a warehouse. A database schema may be high-quality data for an analytics agent and nearly useless for a procurement negotiation unless it is connected to suppliers, terms, approval rules, and exception handling.

That is the second direction: build worlds out of the right data, not just more data.

3. Build Digital Twins, Ontologies, and Gyms

The short version: This is the direction enterprises should watch most closely: build executable models of the domain. Digital twin, ontology, and gym come from different traditions but point at the same thing — a world you can run. The RL community has had this intuition for years; what is new is the pattern moving into software, science, databases, and enterprise workflows. ICLR 2026 shows the environment turn clearly, including a result worth internalizing: agent reliability can vary by more than 50 percent across variations of the same app. A useful world is not one pretty clone. It is a generator of realistic variations that expose brittle behavior.

Deeper dive: the environment turn

Enterprises like "digital twin" because it sounds like a model of a product, factory, supply chain, port, grid, or company. Database and knowledge-representation people like "ontology" because it means entities, relationships, constraints, rules, and semantics. Reinforcement-learning people like "gym" because it means an environment where an agent can act, observe, receive reward, fail, reset, and try again. These are not identical traditions, but strategically they point at the same thing.

A game environment makes the intuition obvious. An agent plays Pong, takes actions, observes pixels or state, receives reward, and learns. The "game" is now a calendar, a messenger, a SQL database, a browser, a lab bench, a robot workcell, or a procurement process.

Several ICLR 2026 papers show this environment turn. AgentGym-RL9 is an open-source framework for training LLM agents across diverse, realistic multi-turn environments, and the paper argues that agents need expanded external interaction rather than internal reasoning alone. CodeGym10 converts static coding problems into interactive, verifiable, controllable tool-use environments. Environment Tuning11 goes further and says: do not only fine-tune the agent; tune the environment, using curriculum, environment augmentation, and progress rewards so agents can learn from problem instances without pre-collected expert trajectories.

BIRD-INTERACT12 is a strong enterprise-adjacent example. It turns text-to-SQL from a single prompt into a dynamic world: database, hierarchical knowledge base, metadata, user simulator, clarification, execution errors, CRUD tasks, and executable tests. This looks much closer to production work than a static benchmark because real data assistants have to ask questions, recover from mistakes, and satisfy changing requirements.

OpenApps13 shows how cheap these worlds can become when the domain is modeled correctly. It provides configurable messenger, calendar, maps, and other app environments that run on a single CPU and can generate thousands of versions. The paper finds that agent reliability can vary by more than 50 percent across app variations.

WALT14 offers another clue. Instead of forcing a web agent to reason through every click and layout change, it reverse-engineers latent website functionality into deterministic callable tools such as search, filter, create, edit, and delete. That is a world abstraction. It takes messy surface interaction and exposes stable operations.

Deeper dive: what this means for your company

This is why the word ontology matters for enterprise readers. A good ontology is not just a diagram. It is a portable, structured model of what exists, how things relate, what actions are possible, what constraints must hold, and what counts as success. If it is executable, it can become a gym. If it mirrors a company or process, it can become a digital twin.

The enterprise version should be concrete. As an illustrative pattern, you do not need to train an agent directly on a live Google Calendar or Slack workspace to teach scheduling behavior. You can model users, calendars, rooms, policies, conflicts, permissions, messages, deadlines, and success conditions. You can reset the world, create adversarial cases, simulate ambiguity, and verify outcomes. You do not need to touch the real procurement system to train negotiation or approval behavior. You can model suppliers, catalogs, contracts, budget constraints, approval chains, exceptions, and audit rules.

That is not a toy version of the company. If the ontology captures the right entities, relationships, constraints, and verifiers, it is the version of the company that an agent can safely learn against.

The Bottom Line

In our view, this reframes the moat. The moat is less about generic generation, which large labs are likely to keep making cheaper, and more about the ability to build useful worlds: scientific worlds with physics and measurement, robotic worlds with contact and action, enterprise worlds with rules and consequences, and software worlds with tools and tests.

In the old data regime, the question was "how much can we scrape?" In the new regime, the question is "what worlds can we make executable?"

That question is harder, but it is also more strategic. It turns AI from a model-buying exercise into an organizational capability: knowing the domain well enough to build the world in which intelligence can practice.

Source Notes

The first era of modern AI scaling was built on cheap passive data. We turned the web, code repositories, math solutions, and worked explanations into training data, and the model learned to predict, complete, translate, summarize, answer, and eventually reason in sequences. That era produced an enormous leap.

But the next bottleneck is not another trillion scraped tokens. The next bottleneck is interaction data: data from models acting in external worlds, receiving feedback, recovering from mistakes, and learning which actions actually change the state of a system.

This is why the scaling-laws conversation has to change. The old question was how loss improves as model size, compute, and token count increase. That question still matters. But the strategic question is now different: how do we create enough high-quality worlds for models to act in?

The Short Version

If you only read one section, read this one.

• Cheap passive data built the first scaling era; interaction data is the next bottleneck. The valuable corpus is no longer text about the world — it is traces of models acting inside worlds that push back.

• Software makes the gap concrete. Writing code is text generation; development is an action loop. Frontier agents score below 65% on Terminal-Bench's hard terminal tasks, and a model at 74.4% on SWE-bench Verified completes only 11.0% of FeatureBench's feature-level tasks.

• The gap repeats in every world. Web data science: 15% agent success where humans reach 90%. Messy tool use: no model above 15% across 57 LLMs. Interactive database work: GPT-5 under 17%. Robotics and science are harder still — six drug-design models generated roughly 120,000 molecules and exactly one showed even very weak activity.

• Synthetic data only helps when verified. Model-collapse results show that retraining on unverified self-generated data can degrade models; the moat is not "generate synthetic data" but verified world generation.

• The new scaling question is conditional: more useful data, from richer worlds, with better verifiers, stronger provenance, and tighter feedback loops — not just more parameters, compute, and tokens.

What's in This Post

Each section opens with the short version, then a deeper dive with the paper-level evidence. Read the short versions for the argument; stay for the deep dives when you want the receipts.

1. Code Is Not Development — the cleanest demonstration that passive data ran out.

2. The Biggest Data Gap — digital worlds: web tasks, tools, and databases.

3. Physical Worlds Are Harder — robotics as a data-world construction problem.

4. Scientific Worlds Are Stranger — why generated candidates are not results.

5. Is the Future Dull? — the new data economy, and the synthetic-data caveat.

6. The New Scaling Question — what replaces "more tokens."

Code Is Not Development

The short version: Writing code is text generation. Development is an action loop: terminal, files, dependencies, tests, errors, logs, retries. New benchmarks finally measure the difference, and the numbers are stark — frontier models and agents score below 65% on Terminal-Bench's hard terminal tasks, and a model reported at 74.4% on SWE-bench Verified succeeds on only 11.0% of FeatureBench's end-to-end feature tasks. The easy corpus was code as text. The valuable corpus is now code as interaction.

Deeper dive: the anatomy of an action loop

That distinction sounds obvious to engineers, but it has not always been obvious in AI benchmarks. A model that writes a plausible patch has not necessarily done development. A development agent has to survive contact with the environment.

Terminal-Bench1 was a quiet but consequential shift because it made this distinction concrete. At the simplest level, a Terminal-Bench-style setup has four pieces:

1. An LLM that proposes actions.

2. A terminal agent or harness that executes those actions and returns observations.

3. A sandboxed environment, typically container-like, where the work can run safely.

4. A verifier, usually tests, that decides whether the task succeeded.

That is a different kind of data. It is not just "here is a problem and here is the answer." It is an action loop. The model proposes. The world responds. The model adapts. The verifier judges.

Terminal-Bench1 makes this explicit: 89 hard tasks in command-line environments, each with a unique environment, a human-written solution, and tests. The reported result is not "frontier models are solved." It is the opposite: frontier models and agents score below 65%.

FeatureBench2 sharpens the point. It evaluates end-to-end feature development across 200 tasks and 3,825 executable environments from 24 repositories. The striking number is the gap: a model reported at 74.4% on SWE-bench Verified succeeds on only 11.0% of FeatureBench2 tasks. That is the difference between generating code and doing development.

The Biggest Data Gap

The short version: The major next data gap in AI is not another archive of web pages. It is traces of generative models interacting with worlds. Across web data science, messy real-world tool use, and interactive database work, agents that look strong on static benchmarks collapse in interactive ones. That pattern matters for enterprises because businesses are not prompt-answer machines — they are worlds, with rules, exceptions, roles, approvals, state changes, and external systems. Operating inside them requires world data.

Deeper dive: three digital worlds, one pattern

WebDS3 tests agents on 870 web-based data science tasks across 29 websites. These tasks require finding data, navigating heterogeneous sources, cleaning, analyzing, and summarizing. One reported browser agent completes 80% of tasks on WebVoyager but only 15% on WebDS3, while humans reach about 90%.

WildToolBench4 shows the same story for tools. It is built around messy user behavior: compositional tasks, implicit intent spread across turns, and instruction transitions where the user changes direction or clarifies. Across 57 LLMs, no model exceeds 15% accuracy.

BIRD-INTERACT5 brings the point into databases and enterprise-adjacent workflows. Static text-to-SQL is not enough when real database work requires clarification, execution feedback, CRUD operations, metadata, knowledge bases, user simulation, and executable tests. The full benchmark contains 600 tasks with up to 11,796 dynamic interactions. GPT-5 completes 8.67% in the conversational setting and 17.00% in the more open-ended agentic setting.

Those benchmarked settings are narrower than a full enterprise, but the pattern matters for enterprises because supply chains, sales operations, marketing workflows, procurement, finance, compliance, support, and database operations are governed by rules, exceptions, roles, approvals, state changes, and external systems. A model that can write a paragraph about procurement is not yet an agent that can operate inside procurement.

Physical Worlds Are Harder

The short version: The terminal is comparatively cheap to reset. The browser adds state and ambiguity. Robotics makes the feedback loop materially more expensive: geometry, friction, contact, latency, perception, hardware failures. ICLR 2026 shows robotics becoming a data-world construction problem — learned world models standing in as environments, video foundation models becoming policies, and imagined trajectories becoming training data.

Deeper dive: world models as environments

WorldGym6 treats a learned world model as an environment for robot policy evaluation. The motivation is blunt: real-world testing is costly, and handcrafted simulators require manual effort to improve realism. WorldGym6 uses an action-conditioned video generation model as a proxy environment for robot policies, then compares policy success in that generated world with real-world success.

Cosmos Policy7 pushes from the other direction. It adapts a video foundation model into a robot policy that can generate actions, future states, and values. The point is not just to make robot videos look good. The model uses future-state and value predictions for planning action trajectories.

Ctrl-World8 makes the synthetic-data loop even more explicit. It trains a controllable multi-view world model on the DROID dataset, with 95,000 trajectories across 564 scenes, and uses imagined successful trajectories for supervised fine-tuning. The paper reports a 44.7% policy-success improvement.

That is the pattern: world model, rollout, action, observation, verifier, correction. Robotics makes it visible because the real world is stubborn.

Scientific Worlds Are Stranger

The short version: A generative model can propose molecules, proteins, materials, hypotheses, equations, and experimental plans. But a generated candidate is not a result — it has to survive the world it claims to describe. The cleanest warning from ICLR 2026: six public structure-based drug-design models generated roughly 120,000 molecules, and exactly one purchased compound showed even very weak activity. Agents given real scientific software reach only 15% success. These are not failures of sentence fluency; they are failures of world binding. AI for science is a world-building problem: simulators, verifiers, experimental interfaces, expert-curated tasks, and representations that respect the domain.

Deeper dive: chemistry, biology, physics, and the verification bottleneck

In chemistry, the GEM workshop talk on experimental assessments of structure-based drug design9 is the cleanest warning: six public structure-based drug-design models generated roughly 120,000 molecules, about 99% were removed by pre-filtering, 47 compounds were bought, and only one showed very weak inhibition or binding at 100 micromolar. The useful artifact is not the generated ligand alone; it is the ligand after screening, purchase, assay, and measurement. The same point appears in paper form through SYNC10, which treats synthesizability as part of structure-based drug design, and 3DCS11, which stresses that molecular representations must be sensitive to 3D conformation rather than only valid as graphs or strings.

In biology, the lesson is constraint. Constrained Diffusion for Protein Design12 builds hard structural constraints into protein generation, because rejection after unconstrained sampling is expensive and often wasteful. Genomics adds a different version of the same warning: ContrastiveBiVI13 focuses on perturbation effects in transcriptional dynamics, while Taking The Easy Way Out14 shows how single-cell foundation models can learn local correlation shortcuts instead of biological mechanism when the masking objective ignores gene co-regulation.

Physics and PDEs make the failure mode even sharper. One Operator to Rule Them All?15 argues that neural PDE solvers can behave like boundary-indexed families rather than a single boundary-agnostic operator. Residual-Spectrum Diagnostics16 makes a related evaluation point: low solution error does not necessarily mean the prediction satisfies the governing equation. And text-trained LLMs extrapolating PDE dynamics17 show that even when models mimic spatiotemporal patterns, multi-step rollouts accumulate error over the horizon.

ScienceBoard18 is a useful bridge because it evaluates multimodal autonomous agents in realistic scientific workflows. It gives agents professional scientific software and 169 human-curated tasks across areas such as biochemistry, astronomy, and geoinformatics. Despite strong backbones, evaluated agents reach only 15% overall success. The same verification problem appears in Silent Squad19, where multimodal agents use real scientific software, and in DSGym20, which reports that many existing data-science benchmark tasks can be shortcut without using the actual data files.

Scientific hypothesis generation has the same asymmetry at a larger scale. The Verification Bottleneck21 frames the problem directly: AI can generate plausible claims faster than scientific institutions can verify them. Work on controlled synthetic causal worlds22 adds a narrower but important point: plausible causal stories are not enough when hidden confounding and mechanism composition matter.

Physics benchmarks show another side of the gap. CMT-Benchmark23 tests expert-level condensed matter theory, including quantum many-body and classical statistical mechanics. Across 17 models, average performance is reported as 11.4% plus or minus 2.1%, with GPT-5 solving 30%. CMPhysBench24 contains more than 520 graduate-level condensed matter physics questions and reports the best model at 29% accuracy.

The model can produce plausible symbols, but the scientific world has constraints: conservation laws, symmetries, boundary conditions, non-commuting operators, experimental noise, physical feasibility, and domain-specific notions of correctness.

Is the Future Dull?

The short version: No. The end of cheap data is not the end of progress; it is the beginning of a more interesting data economy. Building interactive worlds used to require tedious expert labor. Modern coding assistants can multiply expert throughput — drafting tasks, scaffolding tests, building simulators and harnesses — as long as experts stay in the loop. But the caveat is real: unverified synthetic data can collapse models. The moat is not "generate synthetic data." The moat is verified world generation.

Deeper dive: the economics, and the collapse caveat

In the old regime, creating interactive data for a specialized world required tedious expert labor. Someone had to define the scenarios, build the environment, write the tests, produce solutions, validate edge cases, and coordinate between domain experts and engineers. That was slow, expensive, and hard to scale.

Modern coding assistants may change the economics. They can draw on broad technical corpora to draft tasks, generate candidate solutions, scaffold tests, create simulators, write harnesses, and help experts iterate. They are not reliable enough to replace experts. But they are useful enough to multiply expert throughput when experts remain in the loop.

This matters because the most valuable synthetic data is not arbitrary text. It is verified interaction data. It is a scenario plus a world plus a solution trace plus an evaluator. It is a task where the model must act and the environment can push back.

There is a caveat. Synthetic data can hurt. ICLR 2026 has multiple model-collapse papers warning that retraining on self-generated data can degrade models unless the data is mixed, verified, or anchored by real signal. "Escaping Model Collapse via Synthetic Data Verification25" shows that an external verifier, human or stronger model, can prevent collapse in the studied setting, but imperfect verification can plateau or reverse gains. Work on optimal mixing ratios26 similarly argues that real data remains necessary even when synthetic data helps.

The New Scaling Question

The old scaling law was built around more: more parameters, more compute, more tokens.

The emerging scaling question is conditional: more useful data, from richer worlds, with better verifiers, stronger provenance, and tighter feedback loops.

That is why data-quality scaling papers matter. "Scaling Laws Revisited27" explicitly adds data quality to the scaling equation. "Why Less is More (Sometimes)28" gives conditions under which curated subsets can beat full datasets. Common Corpus29 shows that even pretraining data is becoming a provenance and licensing problem, not merely a scraping problem.

The strategic implication is clarifying. Generic generation is likely to keep becoming cheaper, but durable value sits upstream and downstream of generation: tools that create worlds, environments that return feedback, verifiers that distinguish success from plausible but incorrect outputs, and curated traces that bind models to specific realities.

For software, that world is the terminal, the repository, the test suite, the UI, and the deployment environment. For robotics, it is the physical scene and the world model that can safely approximate it. For science, it is the simulator, lab, equation, assay, microscope, telescope, or quantum/chemical system. For the enterprise, it is the process graph: roles, rules, approvals, schemas, exceptions, contracts, and operational state.

We are no longer only in the business of harvesting data. We are in the business of making worlds legible to models.

The next question is the hard one: how do we create these worlds?

Source Notes

Reasoning, data curation, and scaling:

• OpenThoughts, "Data Recipes for Reasoning Models"

• "Softmax Transformers are Turing-Complete"

Every year, AI now has a public rhythm. ICLR opens the year in the spring. ICML carries the summer. NeurIPS closes the year in December.

Fifteen years ago, most executives, investors, and product leaders could safely ignore this calendar. The conferences were academic meetings, the papers needed a PhD colleague to translate, and the market delay could be years. That bargain is gone. In 2026, the work presented at these venues arrives already wrapped in code, benchmarks, datasets, evaluation harnesses, agent environments, leaderboards, safety tests, and workflow abstractions. The lag between "research result" and "thing a developer can run, copy, benchmark against, or get pressured by" is now measured in months, sometimes weeks.

This post makes the case for why that matters to you, using two trends that visibly changed shape in the six months between NeurIPS 2025 and ICLR 2026.

The Short Version

If you only read one section, read this one.

• The trilogy is an early-warning layer. ICLR, ICML, and NeurIPS are where the field shows its working memory: what researchers think is hard, which benchmarks are losing credibility, which abstractions are becoming standard, and which product categories will soon feel obvious.

• Scale killed paper-by-paper reading. ICLR 2026 accepted 5,355 papers, slightly more than the NeurIPS 2025 main track. The useful question is no longer "which paper is important?" but "which weak signals are becoming clusters?"

• Trend 1: the LLM monolith is breaking into layers. Attention variants, KV-cache policy, memory, routing, controllers, and adaptive compute are becoming separable systems layers. At NeurIPS 2025 you had to assemble this thesis from scattered fragments; by ICLR 2026 it was a dense, named research program.

• Trend 2: evaluation is moving from static benchmarks to real tasks. Executable environments, trajectories, tools, logs, and human baselines are replacing one-prompt-one-score tests. The new benchmarks are humbling: a frontier model that scores 74.4% on SWE-bench Verified succeeds on only 11% of FeatureBench tasks.

• The answer is to read conferences as data. Extract papers, claims, benchmarks, and systems into knowledge graphs, compare signals across venues, and watch clusters form before they reach vendor roadmaps.

What's in This Post

Each section below opens with the short version, then offers a deeper dive with the paper-level evidence. Read the short versions for the argument; stay for the deep dives when you want the receipts.

1. The Scale Has Changed — the trilogy by the numbers, and why ICLR now rivals NeurIPS.

2. From Reading Papers to Reading Clusters — why the old way of following research broke.

3. Trend 1: The Architectural Monolith Breaks Into Systems Layers — the model is becoming one layer in a stack.

4. Trend 2: Evaluation Moves From Static Benchmarks to Real Tasks — benchmarks are becoming executable environments, and agents are failing them.

5. Our Answer: Analyze the Conferences as Data — how we turn conference corpora into roadmap input.

6. The Bottom Line — what to do about it.

The Scale Has Changed

The short version: ICLR, the youngest of the three conferences, now accepts slightly more main-conference papers than NeurIPS. These are no longer small specialist venues. They are enormous technical markets of ideas, tools, benchmarks, and talent — and their scale is exactly why they need a new way of being read.

Deeper dive: the numbers

The history is worth being precise about. NeurIPS was founded in 1987. ICML's named-conference lineage starts in 19931, building on earlier International Workshops on Machine Learning. ICLR is much younger; the first ICLR was held in 20132.

So the "trilogy" order is calendar order, not age order: ICLR, then ICML, then NeurIPS. But the calendar order also used to carry a rough intuition about scale. ICLR was the younger spring conference. NeurIPS was the giant year-end finale. That intuition is now outdated.

NeurIPS 2025 reported3 21,575 valid main-track submissions and 5,290 accepted papers, a 24.52% acceptance rate, supported by 20,518 reviewers, 1,663 area chairs, and 199 senior area chairs. ICLR 2026 reported4 19,525 valid submissions, 76,139 reviews, 18,054 reviewers, and 5,355 accepted papers, a 27.4% acceptance rate. By accepted main-conference papers, ICLR 2026 slightly exceeded the NeurIPS 2025 main track.

That does not mean ICLR is larger in every sense. Attendance data still shows NeurIPS as the larger gathering in 2025: the AI Index / Our World in Data series5 reports 26,382 attendees for NeurIPS 2025, compared with 11,039 for ICLR 2025 and 8,000 for ICML 2025, with the usual caveat that conference attendance can mix in-person, online, and hybrid participation depending on the year.

From Reading Papers to Reading Clusters

The short version: The old way to read conference research was paper by paper. That worked when the output was small and the market delay was long. It breaks when one conference accepts more than five thousand papers and the delay compresses from years to months. The skill that matters now is spotting which weak signals are becoming clusters — and the two trends below show how fast that can happen.

Between NeurIPS 2025 and ICLR 2026, two examples make the change visible. They are not the only ones; similar six-month shifts appear across data curation, scientific discovery, safety, verification, and small-model deployment. But these two are especially clear because they moved from scattered evidence to dense, explicit, named research programs. A useful way to read them: the first is about model/runtime decomposition, the second about evaluation realism.

Trend 1: The Architectural Monolith Breaks Into Systems Layers

The short version: The transformer is being decomposed into separable layers — attention mechanisms, state-space routing, KV-cache management, memory, model routers, dynamic depth, and test-time compute policies. The model is no longer the whole product; it is becoming one layer in a stack that also includes cache policy, memory, routing, inference budgets, tool interfaces, verifiers, and runtime policy.

The six-month compression is the story. At NeurIPS 2025 the stack thesis existed, but you had to assemble it yourself from a paper here, a workshop slide there. By ICLR 2026 the pieces were dense enough that the stack surfaced on its own. If your company builds on or buys LLM infrastructure, this is where differentiation is moving.

Deeper dive: the paper trail from NeurIPS to ICLR

At NeurIPS 2025, the LLM-stack thesis was already present, but it was not yet sitting in one obvious place. Routing Mamba6 treated sequence modeling as a mixture of state-space dynamics and sparse expert routing. Gated Attention7 made the attention block itself more controllable by adding gates to softmax attention. ChunkKV8, KVzip9, and RetrievalAttention10 made KV state a memory-hierarchy problem: compress it, evict it, retrieve it, or move indexes off the critical GPU path.

Twilight11 treated attention budget as something to allocate adaptively. REFRAG12 exploited attention sparsity to accelerate RAG decoding, while AdmTree13 compressed long context through adaptive semantic trees. A-Mem14 made agent memory a dynamic, evolving subsystem and compared against MemGPT alongside other memory baselines. KVCOMM15 pushed the KV cache into multi-agent communication, where shared context becomes a systems object rather than a hidden implementation detail.

The workshop evidence points in the same direction. In the Lock-LLM workshop16, one slide summarized the scaling shift as "multi-dimensional": architecture was moving from dense to sparse, capabilities toward long context and multimodality, and inference toward test-time scaling.17

Those papers and workshops did not all announce the same thesis. That is the point. A human analyst had to connect state-space routing, gated attention, KV eviction and retrieval, semantic context compression, agent memory, attention sparsity, context reuse, and test-time compute into one pattern: the model was becoming one layer in a stack.

By ICLR 2026, the signal had become much denser. Mamba-318 was an oral paper built around an "inference-first" state-space design. From Collapse to Control19 analyzed long-context failure in hybrid Mamba-Transformer models and proposed a bridge between Transformer position scaling and Mamba state dynamics. Scaling Laws Meet Model Architecture20 moved scaling-law discussion beyond parameters and data into architecture choices such as hidden size, the MLP-to-attention ratio, and grouped-query attention.

The attention and KV layer was no longer a side topic. Draft-based Approximate Inference21 used small draft models to guide KV dropping and prompt compression. FusedKV22, FreeKV23, KVTC24, and ICaRus25 treated KV state as something to reconstruct, retrieve, compress, store, share, or reuse across models. IceCache26 combined semantic token clustering with PagedAttention, while FASA27 and ThinKV28 made sparse attention and cache compression adaptive to frequency, queries, or reasoning state. vAttention29 and Race Attention30 pushed sparse and linear attention toward deployable regimes with verification or kernel-level evidence.

The routing and control layer was visible too. Universal Model Routing31 selected among models dynamically. LD-MoLE32 made LoRA expert allocation token-dependent and layer-wise. DND33 selectively reprocessed critical tokens through dynamic depth. Strategic Scaling of Test-Time Compute34 treated inference as an allocation problem, giving more compute to harder queries and less to easier ones. vCache35 added verified prompt caching, and the MemAgents workshop made the vocabulary explicit with sessions on controllable parametric memory36 and data-centric memory for LLM agents37.38

Read together, this is no longer a story about one bigger model or one undifferentiated transformer call. It is a story about specialized layers: architecture, cache, memory, routing, inference budgets, tool interfaces, verifiers, and runtime policy.

Trend 2: Evaluation Moves From Static Benchmarks to Real Tasks

The short version: The old evaluation pattern was one prompt, one answer, one score. That is increasingly inadequate — for agents especially, but also for coding, web data science, tool use, and long-horizon planning. The new benchmarks look like real work: executable environments, changing state, trajectories, tools, logs, human baselines, and partial failures.

The new benchmarks are also humbling. Frontier models stay below 65% on Terminal-Bench's hard terminal tasks. A model at 74.4% on SWE-bench Verified succeeds on only 11.0% of FeatureBench's feature-level tasks. A browser agent that scores 80% on WebVoyager falls to 15% on WebDS, where humans reach almost 90%. If you are deciding whether to trust a benchmark claim or move an agent workflow into production, this gap between old and new benchmarks is material input to the decision.

Deeper dive: from scattered signs to a named transition

At NeurIPS 2025, the evidence for this benchmark-realism turn was present, but distributed. OpenCUA39 treated computer-use agents as a scalable open-source training and evaluation problem. TheAgentCompany40 framed workplace simulation as an agent benchmark. AgentIF41 tested instruction following in tool-constrained agentic scenarios.

The spoken conference material pointed in the same direction. The NeurIPS social on evaluating agentic systems42 argued that static and single-turn metrics fail for autonomous planning, tool use, and long-horizon behavior. In AI4NextG, the agent was described as a task-level system with persistent goals, memory, real-time feedback, tool and action selection, and continuous replanning.43 The same source captured the practical QA problem in blunt terms: once agentic AI is "not deterministic," the verification process has to change.44

Again, this was not absent at NeurIPS. It was scattered. The papers, social session, and workshop evidence were pointing beyond "can the model answer?" toward "can the system act?" But the benchmark infrastructure was still forming.

By ICLR 2026, the trend was much harder to miss. The conference program itself was naming the transition. The Agents in the Wild workshop45 scheduled invited talks titled From LLMs to Agents: The Evaluation Challenge46 and Compound AI Systems: Design Patterns for Secure Multi-Agent Deployments47. The DATA-FM workshop carried The Art and Science of Benchmarking Agents48. Our local ICLR transcript pass summarized the same shift: evaluation was moving from static, single-turn benchmarks toward multi-step, multi-agent systems that use tools and engineering environments, while the difficulty of evaluating agents was growing alongside the ease of building them.49

The broader ICLR source pack made clear why static benchmarks are failing. Its cross-topic synthesis describes a shift from "model as product" to "AI as composed, governed system," and the executive summary argues that progress and risk now sit at the system level, not only at the model level.50 The reasoning track made a parallel evaluation point: chain-of-thought and longer inference were being reframed as controlled systems involving data, RL/RLVR, verifiers, scaffolds, memory, retrieval, tools, and cost-aware test-time compute.51 Agentic Context Engineering52 and the MemAgents workshop are useful here because they show why realistic evaluation has to include changing context and memory, not just final answers to static prompts.53

The paper evidence became more concrete too. Terminal-Bench54 evaluated agents on hard terminal tasks in isolated environments, each with a human-written solution and tests, and still found frontier models below 65%. FeatureBench55 moved software evaluation from patch-level work toward feature-level implementation across 200 tasks and 3,825 runtime environments, reporting that Claude 4.5 Opus, at 74.4% on SWE-bench Verified, succeeded on only 11.0% of FeatureBench tasks. WebDS56 made the same realism point for web data science: Browser Use fell from 80% on WebVoyager to 15% on WebDS, while humans reached almost 90%.

Computer Agent Arena57 broadened the scope to computer-use agents in dynamic, human-centric settings. TRACE58 pushed toward benchmarks that evolve through recorded and validated trajectories. Holistic Agent Leaderboard59 standardized agent evaluation across hundreds of VMs and inspected logs to find failure modes that raw success rates miss. MCP-SafetyBench60 showed why this is also a safety and governance problem: once agents use real tool servers, evaluation has to cover actions, permissions, and attack surfaces, not just text outputs.

In December, the field had strong but discontinuous signs that simple static benchmarks were becoming inadequate: agents needed memory, tools, persistent goals, replanning, non-deterministic QA, and more realistic tasks. By April, the signs had become papers, workshops, talks, leaderboards, harnesses, and failure taxonomies. The conference was not merely saying "agents are coming." It was telling us where agents break, which benchmarks are too easy, what new harnesses are being built, and which product claims will need stronger evidence.

This is exactly why conference analysis has become business analysis. If your company is deciding whether to deploy coding agents, use local models, trust benchmark claims, expose tools through an MCP-like interface, or move agent workflows into production, these papers are not abstract. They are material input to roadmap discussions.

Our Answer: Analyze the Conferences as Data

The answer is not for every executive to read five thousand papers. That is not a viable operating model.

We have been building a more systematic way to read these conferences. We treat the conference as a corpus. We extract papers, claims, benchmarks, methods, datasets, systems, and evaluation results. We organize them into knowledge graphs. We compare signals across conferences. We ask which ideas are isolated, which ones are becoming clusters, and which ones are crossing from research vocabulary into product vocabulary.

That is how the two trends above become useful before they are obvious. At NeurIPS 2025, cache policy, architecture, routing, computer-use agents, workplace simulation, and agentic evaluation looked like related but separate fragments. By ICLR 2026, they had hardened into two clearer signals: the model/runtime is decomposing into layers, and evaluation is moving from static tests to realistic complex tasks with environments, tools, verifiers, logs, memory, process supervision, and safety boundaries. The field had moved from "can the model answer?" to "can the system act reliably?"

That is the reason to care about ICLR. Not because every paper matters. Most individual papers will not matter to your business. But the clusters matter. The direction of benchmark design matters. The failure modes matter. The abstractions that repeat across hundreds of papers matter.

If you wait until those abstractions are packaged into vendor roadmaps, you have given up some early signal.

The Bottom Line

You cannot be in the AI business and ignore ICLR, ICML, and NeurIPS.

You do not need to treat the conferences as infallible. You do not need to accept every benchmark, chase every model, or pretend that academic incentives perfectly match product reality. But you do need a disciplined way to watch them.

The frontier now moves through papers, code, benchmarks, data, and systems at the same time. The companies that learn to read that motion early will make better product bets, ask better vendor questions, hire against the right capabilities, and avoid being surprised when "research" turns into "everyone expects this feature" six months later.

ICLR is no longer just a conference for researchers. It is a map of what the AI market is about to argue over next.

Source Notes

Conference scale and history:

• NeurIPS 2026 conference page

• ICML 2025 call for papers

• CS Conf Stats, ICML 2025