Anthropic's multi-agent research system produces a 90.2% improvement on breadth-first research tasks, and it burns 15 times more tokens doing it. That single vendor-stated pair of numbers is the modular vs monolithic debate in 2026, compressed. Modular agent architecture buys quality with tokens. Monolithic agent design buys efficiency with rigidity.
The interesting question is no longer which side wins. It's where you draw the boundary, because the data shows modularity inside the model makes inference cheaper while modularity outside the model makes tasks more expensive.
TL;DR
- Modular agent architecture externalizes tools, memory, and sub-agents behind open protocols (MCP for tools, A2A for peers); monolithic design keeps everything in one loop.
- Vendor-reported numbers: multi-agent decomposition gains 90.2% on complex research but costs 15x the tokens (Anthropic); DeepMind reports up to 17x for some flows.
- MoE and hybrid models (Jamba 1.6 Mini, DeepSeek-V3, Mixtral 8x22B) prove modularity inside the model cuts cost per token by 4-5x versus dense peers.
- The 2026 production consensus is hybrid: monolithic within an agent, modular between agents and tools.
What is modular agent architecture?
A modular agent decomposes its capabilities, including tool access, memory, planning, and sub-agent collaboration, into interoperable services reachable through open protocols, while a monolithic agent collapses all of those concerns into a single tightly coupled loop. The academic literature names the latter explicitly: arXiv 2502.00409 calls the single-loop baseline a "monolithic static architecture" and contrasts it with routing-based modular designs.
There's a clean operational test, and it's worth memorizing. If you can swap the LLM without rewriting the tools, swap a tool server without retraining the model, and add a sub-agent without rewriting the orchestrator, your system is modular.
If any of those moves requires a coordinated rewrite, you've built a monolith in that concern.
The 12-factor-agents principles, distilled from production LLM deployments, operationalize this: own your prompts, own your context window, own your control flow, and expose small, well-typed tools. They're the most-cited litmus test in 2025-2026 engineering practice.
A 2026 reference architecture decomposes the agent platform into seven concerns: model serving, routing, tool gateway, memory store, observability, identity, and policy enforcement. A 2026 taxonomy of agentic AI architectures catalogs more than 20 enterprise production systems using MCP as the tool-isolation layer.
The protocol substrate: MCP and A2A
Two protocols now define what "modular" means in production: MCP for connecting models to tools, and A2A for connecting agents to each other. The Model Context Protocol is Anthropic's open standard, JSON-RPC 2.0 over stdio, HTTP, or SSE, with servers advertising capabilities and hosts discovering them dynamically. Major runtimes including Quarkus LangChain4j and Microsoft Semantic Kernel ship MCP support.
The Agent2Agent protocol solves the problem MCP doesn't: cross-vendor agent collaboration. Google announced A2A in April 2025 with 50+ technology partners, and the project moved to the Linux Foundation in June 2025. Agent Cards handle capability advertisement; push notifications handle long-running tasks.
NVIDIA's AI-Q Blueprint positions the two at different layers, MCP for tools and resources, A2A for peer agents, with OpenTelemetry traces spanning the hops. That's the de facto modular reference stack in 2026.
Performance benchmarks: where each architecture actually wins
The benchmark record splits cleanly: modularity inside the model wins on cost per token, while modularity between agents wins on task quality and loses badly on cost per task. Start with the model level, where the numbers are public list prices.
AI21's Jamba 1.6 is the canonical example of modular design inside the model: Transformer attention blocks interleaved with Mamba state-space blocks at a 1:7 ratio, plus a mixture-of-experts layer activating only a subset of experts per token. Jamba 1.6 Mini runs 12B active parameters out of 52B total and prices at $0.20 per million input tokens and $0.40 output, roughly 4-5x cheaper than comparable dense models. AI21's docs cite up to 2.5x faster inference than dense peers of the same total size (vendor-stated).
DeepSeek-V3 pushes the same logic further: 671B total parameters, only 37B active per token, with inference cost around 1/30th of comparable dense frontier models according to DeepLearning.AI's The Batch. Mixtral 8x22B activates ~39B of 141B via top-2 routing, per Mistral's model card.
The Joint MoE Scaling Laws paper gives this a theoretical floor: at fixed FLOPs, MoE designs outperform dense designs, and the gap widens as compute grows. Sparse modularity inside the model isn't a trick. It's the scaling-efficient regime.
At the agent level, the picture inverts. On SWE-bench Verified, frontier 2025 scores in the 60-80% range came from modular stacks: long-context model plus multi-tool harness plus verification loop. WebArena and GAIA leaderboards show the same pattern, with planner-plus-browser-plus-verifier decompositions beating monolithic chat on long-horizon tasks.
But quality isn't free.
Modular systems produce dramatically better output on complex tasks, and they pay for it in tokens, latency, and orchestration complexity. The architecture decision is a budget decision.
Why do multi-agent systems cost 15x more tokens?
Orchestration overhead grows superlinearly with agent count and message volume, so every sub-agent you add multiplies redundant context, inter-agent messages, and duplicated reasoning. Anthropic's own report puts its research orchestrator at 15x the token consumption of a single-agent baseline; DeepMind's 2025 multi-agent evaluation reports up to 17x for some flows. Both figures are vendor-internal and specific to their evaluated workloads, so treat them as indicative, not universal.
The failure modes are architectural, not implementation bugs. A survey of LLM-based multi-agent systems catalogs cascading hallucination and deadlocked negotiation as structural risks. Andrej Karpathy's widely cited 8-agent parallel test found token cost, not quality, was the dominant failure mode, and his broader "Software 3.0" argument holds that the LLM is the new computer and state belongs in the context window.
That's a serious case for monolithic-style designs on simple tasks.
There's also a ceiling on tool sprawl: MCP Atlas leaderboard analysis shows agents handle dozens of MCP tools well, but selection accuracy degrades measurably past a few hundred.
And the most provocative 2026 result cuts against multi-agent orthodoxy entirely. The Latent Agents work (ACL 2026) shows that training a single model to internalize what a multi-agent system produces externally can match performance with 93% fewer tokens, though only on a narrow task distribution.
Today's modular pipeline is tomorrow's training data for a monolith.
When should you choose monolithic agent design?
Choose monolithic when the task fits one reasoning trace and one tool call, when latency or energy budgets are tight, or when debugging a single trace matters more than swapping vendors. Edge and on-device deployment is the clearest case: orchestration hops are impractical on a phone, which is why Google AI Edge ships small on-device models and Liquid AI's LFM2.5-8B-A1B keeps active parameters near 1B.
Monoliths also win on cold-start latency (one process), debugging (one trace), and end-to-end gradient quality when you control training. Adept's original ACT-1 action transformer co-trained planning and execution in one loop, something no protocol-mediated stack can replicate.
The monolith's costs are equally concrete: long-context traces accumulate errors superlinearly per a 2025 survey of agentic architectures, behavior changes require retraining instead of swapping a tool server, and compliance certification (SOC 2, HIPAA) must be redone on every model update because nothing is isolated. McKinsey's healthcare-AI analysis makes the regulated-industry case for modularity on exactly those grounds.
The 2026 decision matrix
| Workload | Architecture | Why |
|---|---|---|
| Single-shot Q&A, short chat | Monolithic | Orchestration overhead exceeds benefit |
| Edge / mobile / embedded | Monolithic or small-active MoE | Latency and energy constraints |
| Code generation (SWE-bench-style) | Modular (LLM + harness + verifier) | Frontier scores require tool integration |
| Enterprise support & RAG | Modular (retriever + policy + escalation) | Compliance, audit, component swapping |
| Open-ended research | Modular multi-agent (A2A) | 90.2% quality gain justifies token cost |
| Regulated industries | Modular with model isolation | Re-certify the model without rewriting the system |
| Real-time robotics | Hybrid: monolithic policy, modular perception | Tight control loop can't tolerate hops |
The startup census backs the matrix. Sierra ($10B valuation, September 2025), Decagon (profiled by Microsoft for Startups), Letta with its memory-as-a-service MemGPT lineage, and Cohere with its "ROI not AGI" positioning at a $7B valuation all sell modular agent platforms with hot-swappable model cores. The enterprise floor is modular because procurement demands it, even when the brain inside is a monolith.
What this means for you
Stop framing this as a binary. The production consensus, visible in Red Hat's Kagenti ADK writeup, EY's enterprise agentic AI OS case study, and Yann LeCun's AMI Labs "Joint Architecture" program, is monolithic within an agent, modular between agents and tools.
Three rules to apply Monday morning. First, default to a single LLM call until the task demands two or more tool families; the Karpathy test says most tasks don't.
Second, when you do go modular, put the boundaries on protocols (MCP for tools, A2A for peers), not on framework internals, so the model stays swappable. Third, budget tokens like money: a 15x multiplier is fine for a research task worth hours of human time and ruinous for a chat reply.
And instrument everything. Per-component activation rates, cross-hop traces, and cost per task are the only honest way to know which side of the boundary is earning its keep.
Sources
- 12-factor-agents (humanlayer), production principles for modular agent design
- Google: Announcing the Agent2Agent Protocol, A2A launch with 50+ partners
- arXiv 2502.00409: Routing Strategies in LLM-Based Systems, names the "monolithic static architecture" baseline
- arXiv 2601.12560: Agentic AI Architectures, survey of 20+ MCP-based enterprise architectures
- AI21: Jamba hybrid Transformer-Mamba model, modular design inside the model
- Jamba 1.6 launch pricing, $0.20/$0.40 per 1M tokens
- The Batch: DeepSeek-V3 cost efficiency, ~1/30th dense inference cost
- Mistral: Mixtral 8x22B model card, MoE routing details
- Joint MoE Scaling Laws, MoE beats dense at fixed FLOPs
- SWE-bench Leaderboards, agent software-engineering benchmark
- NVIDIA AI-Q Blueprint architecture, MCP + A2A reference stack
- McKinsey: Healthcare AI's modular evolution, regulated-industry modularity case
- EY: Enterprise-scale agentic AI OS, production hybrid posture
- Fortune: Cohere's "ROI not AGI" positioning, enterprise modular platform strategy
- AMI Labs (Wikipedia), LeCun's Joint Architecture research program