Latency Tradeoffs of Synchronous vs. Asynchronous Tool Calls in Agents

Synchronous versus asynchronous tool calling in AI agents is a structural decision set by how tool calls depend on each other, and getting it wrong has a measurable cost: agents that stall when they should run in parallel, or agents that corrupt their own task state when they rush ahead of a result they needed first.

Start with the mechanism. Under synchronous execution, an LLM's decoding halts the moment it issues a function call, and it stays halted until that call returns. A runtime built on synchronous handling serializes every piece of tool work onto a single execution path, one call after another, whether or not those calls had any reason to wait on each other. The same runtime, rebuilt with async handling, can serve many concurrent requests because it releases the execution thread the instant a call gets dispatched. It starts the task, moves on to other work while that task is still running, and comes back to handle the result when it arrives.

The gap matters because of its scale, not because this is an old debate about API calls. LLM inference is a slower category of I/O than almost anything agents have routinely called before. A model generating a multi-step reasoning chain can cost seconds, sometimes many of them. That difference raises the stakes on the sync/async choice considerably, because every blocked thread is now blocked for longer, and every wrong choice compounds faster across concurrent sessions.

None of this settles the question by itself, though. The real question an engineering team has to answer is whether these tool calls depend on each other. That single question, answered honestly for each pair of calls in an agent's workflow, determines which execution model is correct. Everything else follows from it.

The dependency graph between tool calls and the correct execution model

Mapping the dependency graph between tool calls is the first design step for any agent workflow, not a detail to clean up after the fact. Whether two calls can run in parallel comes down to one fact: does the output of one feed the input of the other? If yes, there's a dependency edge between them. If no, there isn't. That's the entire test.

Picture it as two shapes. The first is a fan: five calls branching out from a single starting point, none of them touching each other, all of them converging back at the end. The second is a chain: call A's result becomes call B's input, which becomes call C's input, link after link. Those two shapes call for opposite treatment.

Independent calls, the fan shape, are safe to dispatch in parallel. A data pipeline pulling metrics from five monitoring sources is the clean example: nothing about source two depends on what source one returned, so running them one after another multiplies total latency by five for no benefit.

Dependent calls, the chain shape, must be serialized at each boundary where one result feeds the next step, no matter what else is running concurrently around them. Collapse that chain, let the agent continue before the log query actually returns, and the damage is a wrong root cause and a wrong patch, built on a result that was never really there.

UFO, built as a tool-connected agent harness for teams, handles this by making developers annotate resource interactions before a call gets dispatched. That turns what used to be an implicit assumption about ordering into something inspectable, something the whole team can look at and check, which closes off a common way these systems fail quietly. The dependency graph becomes a shared artifact the runtime can enforce, instead of a thing a single engineer holds in their head.

What sync and async execution each cost when misapplied

Get the sync/async choice wrong in one direction, and the cost is latency. Get it wrong in the other direction, and the cost is correctness. Those are not the same kind of problem, and conflating them is where a lot of agent designs go sideways.

Running independent calls synchronously when they should have been parallelized wastes time in a straightforward, linear way. The symptom a user sees is the spinner. Tool use has to happen in sequence, so the agent's wait time becomes visible, and the whole interaction loses its sense of responsiveness.

Parallelizing dependent calls when they should have been serialized is the more dangerous mistake, because it doesn't fail loudly. Specifically, models sometimes continue a dependent task before the result it actually needed has come back. That's not a crash. The agent proceeds on a stale value, or no value at all, and the result reads as complete and plausible even though it is wrong underneath. Catching that in production takes real output validation, not just a timeout handler or an exception trap, because nothing throws an error. The pipeline just finishes, confidently, with the wrong answer.

That asymmetry should shape how a team weighs the risk. Platforms like UFO that run agents across shared team workspaces feel this tradeoff sharply: a runtime that serializes independent calls burns wall-clock time on every concurrent session, while one that lets async continuation happen too early risks a silent state corruption that's considerably harder to catch than any thrown exception.

AsyncFC: how an execution-layer shim achieves parallelism without retraining the model

A framework called AsyncFC, developed at UC Berkeley, makes a specific and useful claim: decoupling an LLM's decoding from the execution of its function calls is a problem that can be solved entirely at the execution layer, without touching the model itself. That matters for any team running an agent on an off-the-shelf model, because it means meaningful parallelism can be retrofitted onto that agent today, without fine-tuning and without rewriting the functions it calls.

The mechanism is a placeholder. The moment AsyncFC dispatches a function call, it hands the model a symbolic future back in place of the real result, letting decoding keep going. From the model's point of view, the call-and-return pattern it was trained on still holds. Nothing about the protocol changes, and no function implementation needs to be rewritten to support it.

That one mechanism opens up two distinct kinds of concurrency. The second is dependency-safe parallelism between functions: calls with no dependency edge between them get dispatched at the same time, governed by whatever resource interaction annotations the developer supplied.

A finding that matters beyond AsyncFC itself: the results show that LLMs already have a native capacity to reason over these symbolic futures, these placeholders standing in for results that haven't arrived yet. Nobody had to teach the model this skill through additional training. A runtime willing to hand the model a placeholder instead of forcing it to wait could draw on a capacity that was already there.

The framework doesn't remove the need to map dependencies correctly. It executes that map faster and with less wasted time, which is a different thing, and a more honest one.

The accuracy penalty async execution imposes on dependent-call chains

Async execution is not uniformly better than sync, and the place it degrades performance is specific enough to name directly: dependent-call chains, where timing actually carries meaning.

The multi-task benchmark isolates this cleanly. It evaluates agents under realistic tool response latency across several tasks running at once, and it finds that models with a better ability to track task switching and dependency state outperform models without that ability. Models that handle this well are doing something specific, tracking what depends on what, that weaker models aren't.

The failure pattern itself is consistent: a model keeps going on a task before the tool result it needs has actually come back, operates on a value that's missing or stale, and produces output that looks structurally complete while being wrong. And there's a deeper reason this happens. Frontier LLMs operate turn by turn, with no built-in sense of elapsed time and no native tracking of what's still in flight. That turn-based design leaves them oblivious to the passage of time by default; messages arriving out of order can genuinely confuse a model, even when the prompt has been written carefully to warn it about async behavior.

This leaves an open question that AsyncFC's results don't close. AsyncFC works around the model's blind spot toward in-flight state. It doesn't resolve that blind spot at the model level.

On workflows built from dependency chains, it's a correctness risk unless the dependency boundaries are explicitly enforced by the runtime.

The event-driven finite-state machine as the runtime architecture for async agents

An agent that's correctly async needs more than async syntax layered over a synchronous call loop. It needs an execution architecture that tracks the state of every in-flight tool call explicitly, as its own first-class concern.

Research from Salesforce AI Research on asynchronous tool usage proposes an event-driven finite-state machine for agent execution. The FSM keeps track of which tool calls are still pending, handles results that come back out of order, and routes each result to the right place in the agent's reasoning without blocking the main execution path while it waits.

The architecture borrows directly from real-time operating systems, and the analogy holds up well: an RTOS has always had to manage multiple concurrent processes finishing at unpredictable times, and that's structurally the same problem as managing multiple concurrent tool calls inside an LLM agent.

What this buys over a simple async wrapper comes down to a few concrete capabilities. Out-of-order completions get handled correctly: the FSM knows which state a given result belongs to, even if that result arrives after calls dispatched later have already returned.

How persistent agent memory interacts with the sync/async decision

Memory management looks like a separate concern from tool calling, but the same sync/async tradeoff runs straight through it. An agent that stuffs its full conversation history into every synchronous call is paying a latency tax that compounds with scale, even if it's invisible in a quick single-turn test.

LLMs carry no memory between calls on their own. The fix is structured retrieval: surfacing only the memories relevant to the prompt in front of the agent right now, not the entire accumulated history.

A workable version of this splits memory into two tiers. The first tier is the context window itself, treated as working memory: the current turn, a scratchpad, and a small set of memories retrieved specifically for this prompt. The second tier is an external persistent store, whether that's a SQL database, a vector index, or a managed service, holding everything the agent has accumulated and feeding tier one on demand. That feeding happens through parallel retrieval, not by replaying history sequentially.

Memory consolidation, the work of merging and deduplicating what's accumulated over time, is itself an async workload, and treating it as one matters. It can run during the agent's idle time rather than sitting on the live request path, so its cost never touches the latency a user actually experiences.

The connection back to tool calling is direct: a fetch from the memory tier is structurally the same thing as an async tool call. It's a dispatch, a wait, and a return of a result the agent needs before it can reason correctly about the current turn. The same dependency-graph discipline that governs tool calls applies here without modification.

Inter-agent delegation as async tool calling at scale: what A2A standardizes

When one agent hands a subtask to another agent, that handoff is an async tool call aimed at a peer runtime. Everything built up in the earlier sections, dependency graphs, failure modes, state tracking, applies at this scale too, and it now has a shared standard behind it.

The Agent2Agent Protocol reached version 1.0 on March 12, 2026, backed by more than 150 organizations, including AWS, Cisco, Google, IBM, Microsoft, Salesforce, SAP, and ServiceNow. It standardizes how one agent hands work to another, which turns async inter-agent delegation into an interoperable pattern any team can build against, rather than something locked inside one vendor's framework.

The structural parallel to single-agent tool calling is exact. A parent agent dispatching work to a child agent and continuing its own work in the meantime follows the same fan-out pattern described earlier for independent tool calls, operating across a process or network boundary. And the failure modes carry over without change: a parent agent that moves forward before a dependent child agent's result has actually returned will produce the same kind of silent correctness error as a single agent that jumps ahead of a tool call it needed first. The dependency discipline holds across a network call just as it does across a function call.

What A2A adds on top of the single-agent case is standardized handoff semantics: a shared way for agents built on different runtimes, from different vendors, to agree on how a handoff works and what a result means once it comes back. That lets a team enforce dependency boundaries across an entire multi-agent system, not just inside one framework it happens to control. At the scale a production agent platform runs at, time spent blocked on something that should have run in parallel or been respected as a dependency is what makes the whole system feel fast or feel broken in a new way.