AI infrastructure 3 min readPlan serving capacity
Your local AI is fast. What happens when everyone asks at once?
A fast reply for one person is not a capacity plan for a team. Follow the wait from each request’s arrival to its finished answer.
The short answer
A single-user demo cannot tell you how long everyone waits. Compare first output, completion and failure for each request under the intended overlapping workload.
On this page
A busy machine can still leave someone waiting
Imagine a local document assistant that answers one colleague promptly. Now several colleagues submit documents together. Text keeps appearing on one screen while another person sees nothing.
This fictional scene separates machine output from a person’s experience. A busy service can be productive overall while one request waits for capacity.
Original conceptual example / Not measured results
Measure the wait too.
Three requests arrive together Illustrative timing · no measured scale
- Request A
Starts promptly; the answer still takes time to finish.
- Request B
Waits before starting, then generates alongside A.
- Request C
Waits longer even while the service is producing output.
The machine is busy. One person is still waiting. Measure each request from arrival to first output and to completion.
One possible serving pattern. Schedulers may batch or interleave work differently. Queueing is only part of time to first output.
About this visual
An independently authored explanatory model. It shows the condition to inspect, not measured performance or a complete implementation. Read the source and limits below.
What changed
The vLLM team’s September 22 vllm-metal report describes serving overlapping requests on Apple Silicon. The system schedules work and budgets memory for execution and cached state. Requests that do not fit in its cache pool wait for capacity. Reported performance belongs to the configurations in the post, not every Mac or workload. [1]
Measure two waits, not one speed
Time to first token runs from the request’s arrival to the first piece of its reply. Time to completion runs from arrival to the finished answer. Queueing and input processing can both contribute to the first wait. Neither measure is the same as total output per second across users.
Our interpretation: decide which wait matters for the task. A drafting assistant may benefit from showing progress; a downstream job may need the complete answer. Record both, along with requests that fail or never finish.
For builders Methods, implementation details and source evidence
Describe the workload before the benchmark
Record model and serving versions, hardware, input lengths, output limits and arrival pattern. Repeat the same work sequentially and with controlled overlapping arrivals. Record every request’s arrival, first output, completion and failure or cancellation.
Separate runs with reusable cached input from runs without it. Report individual waits or their distribution, not just an average that hides slow cases. A request that never completes must remain in the results.
What we don’t know
We have not run vllm-metal or measured a machine for this edition. No throughput, memory-capacity or purchasing claim follows from the example. Supported models and release behavior may change. A synthetic workload does not establish performance on private production documents.
Try this Open the full exercise and its limits
Try this
On an authorized test system, use the same small set of public-text requests in two rounds: sequentially and in a modest overlapping batch. Respect resource limits and keep output limits fixed.
Record waiting, completion, errors and cancellations. Check whether the interface accurately reports queued work. The expected evidence is a documented difference—or absence of difference—under the chosen load. This is a proposed diagnostic, not a completed benchmark.
What to watch next
For each fast local-AI demo, ask: how many people, which input lengths, and which waiting time? A repeatable workload description is more useful than an isolated speed number.
One AI decision, made clearer.
Get a short email with one visual explanation and one practical check. Follow the full article when you want the evidence.
Send me the briefingOpens our email signup page. Confirm by email to receive the briefing. Free; unsubscribe anytime.
Preview a short email →Follow the evidence
Sources & further reading
- vLLM: Concurrent Serving on Apple Silicon ↗
September 22, 2026. Maintainer release report; results apply to its stated setups. Checked September 27, 2026.
Back to cited passage ↑
A correction or a different perspective? Write to us.