ReAct, the architecture behind most of today's agent loops, burns roughly 10,000 tokens per HotpotQA question. ReWOO, which plans all its retrieval steps up front instead of interleaving them, hits slightly better accuracy at about 2,000 tokens.
That single comparison explains most of what's happening in AI agent architecture right now: multi-hop reasoning works, but nobody can afford to run it on every query.
So the real question for 2026 isn't "reasoning vs retrieval." It's how to route between them without giving up the accuracy gains that made multi-hop worth building in the first place.
TL;DR
- Multi-hop reasoning lifts retrieval quality by up to 21 points and downstream QA by up to 15 points over single-shot baselines, per the IRCoT paper.
- Single-hop retrieval costs roughly one embedding call plus one LLM forward pass. A multi-hop agent multiplies that by every step, and active-retrieval methods like FLARE can trigger 10 to 30 retrievals per answer.
- No major vendor ships single-hop-only or multi-hop-only in a flagship product. The production default is a hybrid router: cheap single-hop first, escalation when confidence is low.
- Evaluation has moved from single-turn RAG metrics to per-step trajectory scoring with RAGAS, DeepEval, Phoenix, and LangSmith.
What's the actual difference between single-hop retrieval and multi-hop reasoning?
Single-hop retrieval decouples retrieval from reasoning; multi-hop reasoning interleaves them. In single-hop, the query is embedded once, top-k chunks come back from a vector index, and the LLM answers in one pass. In multi-hop, the model decides what to retrieve next based on what the previous step revealed, looping until it has enough evidence.
Single-hop is the architecture every "chat with your docs" tutorial ships. It's cheap, fast, and genuinely fine for direct lookups, which is why Google Cloud and Databricks still document it as the baseline RAG pattern.
Multi-hop has four canonical instantiations, all worth knowing by name. ReAct interleaves reasoning traces with tool calls (Yao et al.). IRCoT triggers a retrieval after each chain-of-thought sentence (Trivedi et al.). Self-RAG trains the model to emit reflection tokens deciding per-sentence whether to retrieve (Asai et al.). FLARE re-retrieves whenever the model's next-sentence confidence drops (Jiang et al.).
"Agentic RAG" is the marketing umbrella stretched over all of this. Strip the branding and the question is always the same: does retrieval happen once, or inside a loop?
The numbers: what multi-hop buys you, and what it costs
Multi-hop delivers measurable accuracy gains, but the token bill scales with every step. IRCoT improved retrieval by up to 21 points and QA by up to 15 points across HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. ReAct improved absolute success rates by 34% on ALFWorld and 10% on WebShop against imitation and RL baselines, per its authors.
The cost side is brutal. The ReWOO comparison on HotpotQA puts it in one chart:
That's a 5x token gap for a 1.6-point accuracy difference, and ReAct is the cheap end of multi-hop. Tree-of-Thoughts has been reported at 10 to 100x the token cost of plain chain-of-thought on Game of 24. FLARE on a long-form answer can fire dozens of confidence-triggered retrievals.
Embeddings, by contrast, are nearly free. OpenAI's text-embedding-3-small runs $0.02 per million tokens with 50 to 200 ms of added latency. The expensive part of every extra hop is the additional LLM forward pass, not the search. This is why AI inference costs, not retrieval costs, decide the architecture.
One corrective worth internalizing: an Allen Institute analysis found that many HotpotQA questions labeled multi-hop are actually answerable single-hop, because compositional phrasing doesn't always require chained evidence. Benchmark wins overstate how much of your real traffic needs an agent loop.
Why does single-hop retrieval fail?
Single-hop fails because stuffing top-k chunks into a prompt collides with how LLMs actually read context. Liu et al.'s "Lost in the Middle" showed models perform best when evidence sits at the start or end of the context and degrade sharply when it's buried mid-prompt. GPT-3.5-Turbo with the relevant document in the middle of its context scored below its own closed-book baseline of 56.1%.
Read that again. Retrieval that lands evidence in the wrong position can be worse than no retrieval at all.
Multi-hop sidesteps this structurally. Each step works with a small, focused context where the relevant evidence is freshly anchored at the top. That, more than any reasoning magic, is why interleaved retrieval beats one big context dump on hard questions.
Single-hop also has no verification mechanism. If the retriever misses, the model hallucinates, and nothing in the pipeline notices. HopRAG (February 2025) attacks the retrieval-miss problem directly by replacing semantic similarity with logical graph traversal.
The architecture decision in 2026 is not single-hop versus multi-hop. It's how cheaply your router can tell which queries actually need the expensive path.
How are production teams scaling AI agents in 2026?
Every serious deployment has converged on the hybrid router: single-hop as the cheap default, multi-hop as the escalation. Adaptive-RAG made it explicit with a classifier routing queries to no-retrieval, single-hop, or multi-hop by complexity. LlamaIndex ships it first-party as the RouterQueryEngine, and Cohere's Agentic RAG docs treat routing, parallel, sequential, and multi-faceted queries as the four canonical patterns.
The vendor landscape confirms it across domains. In customer service, Intercom's Fin, Decagon, and Sierra all describe multi-step tool use with single-hop knowledge-base lookup as the first stage. Gartner predicts agentic AI will autonomously resolve 80% of common customer-service issues by 2029, and a 2026 Gartner survey found 91% of customer-service leaders under pressure to ship AI this year.
In data analysis, Hebbia is the flagship multi-hop case: OpenAI's case study describes automating roughly 90% of finance and legal document work by chaining retrieval and reasoning across 10-Ks, contracts, and decks. Glean runs the textbook hybrid, single-hop search across SaaS apps with multi-hop traversal on top.
Content generation has drifted toward FLARE-style active retrieval. Writer, Jasper, and Typeface all ship agentic workflows in 2025-2026; Microsoft's coverage of Typeface frames it explicitly as agentic marketing with multi-step planning. And Andrej Karpathy, in his widely covered April 2026 "LLM Wiki" gist (per secondary coverage; the exact wording varies by source), dismissed naive RAG as a quick and dirty hack, arguing for long-running agents that curate a knowledge base with retrieval as one tool among many.
The telling detail: nobody ships single-hop-only anymore, and the most aggressive multi-hop vendors all keep a single-hop first stage. The argument is over.
When does multi-hop reasoning actually pay off?
Multi-hop wins when the cost of a wrong answer exceeds the cost of extra inference; single-hop wins when latency or query volume dominates. Voice agents and sub-second copilots have no room for a seven-step loop. Compliance, legal, and financial workflows happily trade a ten-second wait for grounded citations, because one hallucinated clause costs more than a million extra retrievals.
Here's the decision table production teams have settled on:
| Situation | Use | Why |
|---|---|---|
| Direct lookup, FAQ-style traffic | Single-hop | One embed + one pass; accuracy gain from hops is marginal |
| Sub-second latency budget (voice, copilots) | Single-hop, escalate async | Multi-hop can't fit the budget |
| Evidence chained across documents | Multi-hop (ReAct, IRCoT) | Each step depends on the prior step's findings |
| Long-form generation with shifting evidence needs | FLARE-style active retrieval | Re-retrieve only at low-confidence spans |
| Most queries need no retrieval at all | Self-RAG / adaptive | Pay for retrieval only when it helps |
| Entity-rich corpus, semantic search failing | GraphRAG / HopRAG | Graph traversal beats similarity |
| Mixed easy and hard traffic | Hybrid router | The 2026 default |
A third position deserves mention: for small corpora (one 10-K, one codebase), long-context models with 1M to 2M token windows often match multi-hop accuracy at lower latency by skipping retrieval entirely. That advantage evaporates on enterprise-scale corpora, where Lost in the Middle reasserts itself.
Multi-hop has its own failure modes. Error propagation is the big one: a wrong intermediate fact derails everything downstream, which is exactly what Self-RAG's reflection tokens and CRAG's corrective retrieval grading exist to catch. And unconstrained agent loops on adversarial queries can run arbitrarily long, so step caps aren't optional.
Evaluating both: the tooling has caught up
Single-turn RAG metrics can't score a multi-hop trajectory, and the 2025-2026 toolchain reflects that. RAGAS remains the open-source standard for reference-free faithfulness, answer relevancy, and context precision/recall. DeepEval added agentic metrics (task completion, tool correctness) through its 2025 releases, positioning itself as Pytest for LLMs.
For multi-hop specifically, tracing matters more than scoring. Arize Phoenix exposes the full agent trajectory and integrates RAGAS for per-step evaluation; LangSmith does the equivalent for LangChain and LangGraph shops. Most production teams run two or more together: RAGAS offline, DeepEval in CI, Phoenix or LangSmith on live traces.
What this means for you
Key takeaways:
- Don't pick a side. Build the router. Single-hop handles the cheap majority of traffic; multi-hop handles the queries where being wrong is expensive.
- Budget for the 5x. If your agent loop averages five steps, your inference cost per hard query is roughly 5x your single-hop cost. Price the escalation tier before you ship it.
- Cap your loops. Set hard step limits on ReAct-style agents; adversarial or ambiguous queries will otherwise spend without bound.
- Audit your "multi-hop" traffic. Per the Allen Institute finding, a chunk of queries that look multi-hop are answerable in one shot. A complexity classifier in front of the agent pays for itself fast.
- Instrument trajectories, not just answers. If you can't see which hop failed, you can't fix error propagation. Phoenix or LangSmith tracing plus RAGAS per-step scoring is the current baseline.
The scalability answer, then, is neither. Multi-hop reasoning doesn't scale economically on its own, and single-hop retrieval doesn't scale in accuracy as corpora and stakes grow. What scales is the router that knows the difference, and the next wave (Self-RAG, Adaptive-RAG, Auto-RAG) is already pushing that routing decision inside the model itself.
Sources
- ReAct: Synergizing Reasoning and Acting in Language Models, the foundational interleaved reasoning-and-acting paper, with ALFWorld and WebShop results.
- Interleaving Retrieval with Chain-of-Thought Reasoning (IRCoT), the +21 retrieval / +15 QA multi-hop result.
- Self-RAG: Learning to Retrieve, Generate, and Critique, reflection tokens for adaptive per-sentence retrieval.
- Active Retrieval Augmented Generation (FLARE), confidence-triggered re-retrieval for long-form generation.
- Lost in the Middle: How Language Models Use Long Contexts, why position of evidence in context drives single-hop failure.
- Compositional Questions Do Not Necessitate Multi-hop Reasoning, Allen Institute's corrective on HotpotQA.
- Gartner: agentic AI to resolve 80% of common customer-service issues by 2029, the demand-side forecast.
- Gartner survey: 91% of customer-service leaders under AI pressure in 2026, adoption pressure data.
- OpenAI case study: Hebbia, multi-hop document analysis automating ~90% of finance and legal work.
- Cohere: Building Agentic RAG, first-party routing, parallel, sequential, and multi-faceted query patterns.
- LlamaIndex RouterQueryEngine, the hybrid router as a first-party primitive.
- LangChain multi-agent documentation, agent orchestration patterns for long-horizon tasks.
- RAGAS metrics documentation, reference-free RAG evaluation standard.
- DeepEval 2025 changelog, agentic and per-step evaluation metrics.
- HopRAG: Multi-Hop Reasoning for Logic-Aware RAG, graph traversal as a response to retrieval misses.
- Microsoft on Typeface's agentic marketing, content-generation vendor positioning.