AI Agent Orchestration at Scale: How to Size a System for 10,000 Concurrent Workflows

Sizing AI agent orchestration for 10,000 concurrent workflows comes down to three numbers: how many workflows are in flight (arrival rate × average duration), how many of those need compute right now, and how many tokens per minute they send to your LLM provider. The token limit usually binds first, so size for it before you add workers.

A single agent is easy to run. At 10,000 workflows in flight, some are waiting on the model, some are calling MCP servers or databases, and some have been paused for a human reviewer since yesterday. This article works the arithmetic through on one illustrative workload so you can swap in your own numbers. For the rest of the production picture, see our guide to running AI agents in production.

Why AI agent orchestration breaks differently from web scaling

A web service handles short, stateless requests. At 500 requests a second taking 200 milliseconds each, you need about 100 requests' worth of capacity, and any replica can take the next one.

Agent workflows break that model in three ways.

• Durations vary enormously. A model call takes seconds. A human approval can take two days. The long runs dominate how many workflows are open at once.
• Most in-flight time is waiting. A 90-minute workflow might spend one minute calling models and tools and 89 minutes waiting for a person, a timer or an external event.
• Someone else sets your ceiling. LLM providers enforce request and token limits, databases have connection pools, and SaaS APIs throttle. If 10,000 workflows each fire five tool calls at once, downstream systems face 50,000 requests while your cluster sits well below its own limit.

The arithmetic below also assumes workers are interchangeable, which only holds if workflow progress lives outside worker memory. If it doesn't, every restart loses work.

Sizing concurrency: the Little's law calculation

Little's law says that in a stable system, the average number of items inside equals the arrival rate times the average time each item spends inside:

L = λ × W
concurrent workflows = arrival rate × average duration

It holds for any duration distribution, including a mix of 30-second and two-day runs, as long as you use the mean duration and a period where arrivals and completions roughly balance.

Here is an illustrative workload. None of these figures come from a real system; replace them with numbers from your own traces.

Step 1: workflows in flight. 2 runs a second × 5,400 seconds = 10,800 concurrent workflows.

Step 2: workflows that need compute. Active time per run is 8 × 5 seconds + 20 seconds = 60 seconds. Apply Little's law to the active part only: 2 × 60 = 120 runs executing at any moment, or 1.1% of the fleet.

Step 3: size for the peak. If peak arrivals are three times the average (6 a second), active runs rise to 6 × 60 = 360. At 25 activities per worker, 360 ÷ 25 = 14.4, so 15 workers, plus a few to survive losing a node. Call it 18.

Step 4: compare with holding compute while waiting. If every open workflow kept a thread or container slot while it waited, peak capacity would have to cover 6 × 5,400 = 32,400 workflows, or 32,400 ÷ 25 = 1,296 workers. That is about 86 times the 15 the active load needs (1,296 ÷ 15).

Your LLM rate limit is the real ceiling

OpenAI measures requests and tokens per minute, among others, at the organization and project level, and varies them by model. Anthropic measures requests, input tokens and output tokens per minute for each model class and returns a 429 error with a retry-after header when you exceed one. Whichever limit you hit first stops you.

requests per minute = runs per second × 60 × model calls per run
tokens per minute   = requests per minute × tokens per call

Continue the illustrative workload with 6,000 input tokens per call (system prompt, tool definitions and history) and 400 output tokens:

• Requests: 2 × 60 × 8 = 960 a minute
• Input tokens: 960 × 6,000 = 5,760,000 a minute
• Output tokens: 960 × 400 = 384,000 a minute

Now suppose your account allows 3,000,000 input tokens a minute (a made-up figure; read yours from your provider's console). You are 1.92 times over it.

Each run uses 8 × 6,000 = 48,000 input tokens, so that limit supports 3,000,000 ÷ 48,000 = 62.5 runs a minute, or about 1.04 runs a second. At 2 runs a second, roughly half your model calls come back as 429s and the 18 workers sit in backoff. More workers only add retries against the same limit.

Two effects push the real number higher. Agents append history on every turn, so later calls carry more tokens than the first; use the measured average across all calls. And a 10% retry rate adds 10% to token load.

The levers, in the order they usually pay off:

• Cache the repeated prefix. Anthropic's docs state that for most Claude models only uncached input tokens count toward the input limit. If 70% of each call's input is a cached system prompt and tool definitions, counted input drops to 1,800 tokens a call: 960 × 1,800 = 1,728,000 a minute, under the illustrative 3,000,000.
• Set output caps to what you need. OpenAI's guide says the limit is calculated as the maximum of max_tokens and the estimated tokens in your request, so an oversized cap spends quota. Anthropic's docs say max_tokens does not count toward its output limit. Read your provider's rules.
• Make fewer calls per run. Route classification and extraction to a smaller model. Anthropic applies limits separately for each model, so this also spreads load.
• Smooth bursts. Anthropic notes that a per-minute limit may be enforced over shorter intervals, so a spike of resumed workflows can trigger 429s even when the minute's total fits.

For how these token volumes turn into a bill, see the economics of running AI agents at scale.

Scale runnable work, not waiting work

The sizing math only works if the 10,680 waiting workflows cost storage and nothing else. That needs three things from your orchestration layer: workflow state, including completed steps and results, is persisted outside the worker process; a waiting workflow releases its thread, connection and memory and resumes when its timer fires or its event arrives; and any worker can pick up the resumed workflow from that state.

Waiting workflows often turn back into running ones all at once. Approval requests sent at 5 p.m. get answered at 9 a.m. A thousand workflows that hit a 429 and back off for exactly 30 seconds retry in the same second. Add random jitter to backoff delays and scheduled wake-ups, and include the morning resume spike when you size for the peak. The same goes for rate-limit backoff: a worker that sleeps inside an activity holds a slot that an active run needs, so hand the wait back to the orchestrator.

Backpressure and bounded fan-out

Multi-agent orchestration multiplies fan-out: a planner that spawns four sub-agents, each making eight model calls, turns one run into 32 calls. Unbounded fan-out converts a normal arrival rate into a spike at every dependency.

Bound it at three levels:

• Per workflow. Fifty documents become five batches of 10, each finishing before the next starts.
• Per worker pool. Cap concurrent activities per worker (the 25 in the sizing table) so one pod can't open 500 connections.
• Globally. Put one shared limiter in front of each model and each rate-limited API, set to about 80% of the provider limit. Eighteen pods that each enforce the full limit locally will send 18 times the limit.

A minimal per-workflow bound in Python:

import asyncio
 
MAX_PARALLEL = 10  # per workflow; tune with the global limiter in mind
 
async def review_documents(docs, review_one):
    sem = asyncio.Semaphore(MAX_PARALLEL)
 
    async def bounded(doc):
        async with sem:
            return await review_one(doc)
 
    return await asyncio.gather(*(bounded(d) for d in docs))

When a 429 comes back, honor the retry-after header if present, and otherwise use exponential backoff, which OpenAI's rate-limit guide recommends, with random jitter added. Work that can't run yet should stay pending in the orchestrator.

Autoscaling signals that matter

Agent workers mostly wait on network calls, so CPU stays low even when the system is saturated. These signals tell you whether another worker will help.

Scale out on queue depth only when the 429 rate and dependency latency are healthy. When oldest-workflow age climbs, start with the step whose oldest workflow is aging fastest and check whether it's waiting on a dependency or on a worker.

A reference architecture

An agent fleet sized with the numbers above has five layers.

• Intake. A queue that absorbs arrival spikes.
• Orchestration runtime. Starts workflows, records every completed step, holds waiting workflows as state, and hands runnable steps to workers. The 10,800 live here.
• Activity workers. Stateless processes that run model calls, tool calls and business logic, scaled on runnable queue depth. The agent framework (LangGraph, CrewAI, OpenAI Agents or your own loop) runs here and decides how the agent reasons.
• Shared limiters. One per model and per rate-limited API, set below the provider limit.
• Dependencies. Models, MCP servers, databases and SaaS APIs, with capacity you don't control.

The same layering holds for enterprise AI agents built by many teams: one orchestration layer under many frameworks. The AI orchestration page describes how Catalyst fills the runtime layer.

Durable timers in Catalyst: waiting without holding compute

The 86× gap in the sizing example depends on waiting workflows not occupying workers. Diagrid Catalyst runs Dapr Workflows, with orchestration in the Catalyst data plane and workflow history stored outside application memory. Durable timers and external events are how a workflow waits.

The Dapr workflow documentation states that workflows can be unloaded from memory while waiting for a durable timer, that timers can run for minutes, days or years, and that timer duration has no maximum. A workflow waiting for a reviewer exists only as a record in the state store.

This agent step waits up to 24 hours for a human decision, adapted from the Catalyst workflow patterns:

from datetime import timedelta
import dapr.ext.workflow as wf
 
def agent_review_workflow(ctx: wf.DaprWorkflowContext, task: dict):
    draft = yield ctx.call_activity(run_agent, input=task)
    yield ctx.call_activity(request_review, input=draft)
 
    decision = ctx.wait_for_external_event("review_decision")
    timeout = ctx.create_timer(timedelta(hours=24))
    winner = yield wf.when_any([decision, timeout])
 
    if winner == timeout:
        return (yield ctx.call_activity(escalate, input=draft))
    return (yield ctx.call_activity(apply_decision, input=decision.get_result()))

In the illustrative fleet, that is 10,680 state records and 120 active runs sharing the worker pool. The same timer handles rate-limit backoff: when an activity reports a 429, the workflow creates a timer for the retry-after interval instead of sleeping inside a worker.

Two caveats. Activities run with at-least-once semantics, so steps with side effects still need idempotency keys. And some platforms count waiting workflows against a concurrency allowance; our durable execution pricing comparison shows how each bills for waiting. Catalyst's other production features are covered in the production guide.

To size your own fleet, pull a week of traces and write down five numbers: arrival rate at average and peak, mean end-to-end duration, active seconds per run, model calls per run, and average input and output tokens per call. Run them through the formulas above, compare the token result with the limits in your provider console, and set your shared limiter at 80% of the lowest one.

Frequently asked questions