This post is AI-generated. But these are the real numbers. Sorry if it's badly written but the numbers might be useful to someone.
On October 19, 2026, we measured how parallel requests change throughput and latency on one NVIDIA DGX Spark running Nemotron locally. The first experiment used direct model-serving requests; the second used Hermes agents with tools.
Hardware and model
- NVIDIA DGX Spark: GB10, 128 GB unified memory and 20 ARM CPU cores.
- Ubuntu 24.04.5 LTS, NVIDIA driver 580.178.04, Docker 29.6.2.
- NVIDIA Nemotron 3.5 Super VL 120B A12B NVFP4, served by vLLM 0.32.0.
- LiLiCorr speculative decoding enabled with seven draft tokens.
Direct vLLM requests
A small Python client submitted simultaneous requests directly to vLLM. There was no Hermes or other agent harness in this experiment.
Each request contained 219–227 input tokens and generated exactly 128 output tokens. Reasoning was disabled. We ran one warm-up and two measured bursts at each concurrency level. The server allowed up to 64 active sequences, with a 32,768-token maximum context and 75% GPU memory utilization. Image and video inputs were disabled.
Throughput includes prompt processing, queueing and the time for the last request to finish. Latency starts at submission. Values below summarize the two measured bursts; p95 was calculated within each burst first.
| Submitted requests | Total output tokens/s | Median completion (s) | p95 completion (s) | Peak running / waiting |
|---|---|---|---|---|
| 1 | 26.6 | 4.8 | 4.8 | 1 / 0 |
| 2 | 44.0 | 5.6 | 5.8 | 2 / 0 |
| 4 | 65.7 | 7.7 | 7.8 | 4 / 0 |
| 8 | 96.4 | 10.3 | 10.6 | 8 / 0 |
| 16 | 107.5 | 13.7 | 18.9 | 14 / 2 |
| 32 | 118.0 | 25.9 | 34.3 | 14 / 18 |
| 64 | 121.1 | 41.3 | 66.8 | 14 / 50 |
Eight simultaneous requests delivered 3.6 times the aggregate throughput of one. Moving from 8 to 16 added 12% throughput; moving from 32 to 64 added 2.7%. Larger bursts mostly increased queueing at the higher levels. Only 14 requests generated simultaneously in this configuration; that is an observed configuration limit, not a universal Spark limit.
One request generated approximately 29.3 tokens/s after its first token, compared with 13.9 tokens/s at eight requests. This approximate streaming rate excludes initial waiting. All 254 timed requests completed without request errors or preemptions. Peak sampled GPU temperature was 66°C.
This was a short synthetic burst test. It does not establish long-context capacity, model quality, steady-state maximum throughput, or how quickly an agent finishes useful work. Error bars show the range of two runs, not confidence intervals.
Download the CSV table or JSON results.
Model and recipe: NVIDIA model card. DGX Spark hardware specifications.
Hermes agents using tools
We then ran a separate experiment using Hermes v0.22.2, pinned at d923a5e. Each agent repaired a small Python settlement-summary function: read files, observe failing tests, fix the implementation, rerun tests, execute its CLI on randomized records, and write a report. Agents used actual file and terminal tools in separate Docker sandboxes. There was no child delegation or cloud-model fallback.
The hardware and model were the same. Serving settings had changed before this experiment: 131,072 maximum context tokens and four active model sequences, with Hermes using medium reasoning. Those settings remained fixed throughout this series. Actual request input reached 15,899 tokens; this was not a 131K-history test. The earlier raw test used different serving and reasoning settings, so comparing the two experiments does not isolate Hermes overhead.
We planned 1, 2, 4, 8 and 16 simultaneous jobs, twice per condition. The predeclared stopping rule ended escalation after 17 primary bursts. Conditions absent from the table were not run. The first sixteen-job burst reached the driver deadline for fourteen jobs. Four recorded inference streams broke during cancellation, triggering the predeclared error stop. The remaining three sixteen-job bursts were not run. Half the conditions added an eight-second wait before each test invocation, simulating a slow test service. A compliant job invoked it at least twice. The other conditions added no wait. This is controlled latency, not a measurement of a real external service.
The final checks passed 62/76 timed jobs in the primary comparison. Checks included six unit tests, protected-file hashes, independently calculated output, independent CLI execution and 33 additional cases. A failed test before repair was expected. Throughput below preserves the declared artifact-and-workflow validation metric, which can include cancelled agents that already wrote valid files. The “Clean exits” column additionally requires a normal process exit without driver cancellation; elapsed time includes all launched jobs through process exit or cancellation. A timed-out job has no successful completion latency. Each agent had a 20-turn / 600-second budget, with a 660-second driver timeout.
| Concurrent jobs | Added wait per test (s) | Bursts | Validated jobs | Clean exits | Validated/min | Median elapsed (s) | p95 elapsed (s) |
|---|---|---|---|---|---|---|---|
| 1 | 0 | 2 | 2/2 | 2 | 0.98 | 61.9 | 61.9 |
| 1 | 8 | 2 | 2/2 | 2 | 0.72 | 83.6 | 83.6 |
| 2 | 0 | 2 | 3/4 | 3 | 0.99 | 88.5 | 92.6 |
| 2 | 8 | 2 | 4/4 | 4 | 1.14 | 99.0 | 105.1 |
| 4 | 0 | 2 | 8/8 | 8 | 1.56 | 143.3 | 153.7 |
| 4 | 8 | 2 | 8/8 | 8 | 1.42 | 146.9 | 168.4 |
| 8 | 0 | 2 | 15/16 | 15 | 1.13 | 366.3 | 394.1 |
| 8 | 8 | 2 | 14/16 | 14 | 1.01 | 370.8 | 410.3 |
| 16 | 0 | 1 | 6/16 | 2 | 0.54 | 663.4 | 663.6 |
Values summarize the recorded bursts shown in the table. Whiskers show their range, not confidence intervals; a single burst provides no repeat range. The p95 is calculated within each burst and then summarized; these small samples do not establish production tail latency. Timing includes process and sandbox startup but excludes dependency installation.
Four simultaneous Hermes jobs produced the most validated work in both tested delay conditions: 1.56 jobs/minute without added waits and 1.42 with eight-second test waits. Every one of the sixteen jobs at concurrency four passed all final checks. Without added waits, one job took about 62 seconds versus 143 seconds at four; throughput increased about 1.60× while a single job became about 2.32× slower.
Eight jobs yielded 1.13 validated jobs/minute without added waits and 1.01 with waits; median elapsed time rose to 366 and 371 seconds. At eight, two jobs omitted the required pre-edit failing test, and one other job failed the additional correctness cases. Another correctness failure occurred at concurrency two. Both coding errors ignored a pending first occurrence of a duplicate ID. These samples do not establish that concurrency caused the errors.
At sixteen, two agents exited normally and fourteen required driver cancellation at roughly 663 seconds. Six sets of artifacts and required workflow passed validation, including four from cancelled runs: 0.54 validated jobs/minute, but only 0.18 validated normal exits/minute. The declared stop ended escalation after stream errors during cancellation. The other three sixteen-job bursts were not run. Request-level records for ten unfinished streams are incomplete; the whole-job result and independent validation remain recorded.
Useful overlap was observed: other agents streamed content or tool calls during 15 of 16 injected waits in the four-job delayed condition. Reasoning deltas were not timestamped, so this understates generation overlap and is not a GPU-idle trace. The delayed four-job condition delivered about 1.97× the single-job validated throughput; a longer or different kind of tool wait could change the optimum.
The no-wait prefix-cache hit fraction fell from 72.5% at four to 13.6% at eight and 0% in the sixteen-job burst. Mean sampled model queue time increased from 0.065 seconds to 16.2 and 52.5 seconds. No preemptions were recorded, and sampled swap-out at four, eight and sixteen was zero. This is evidence of reduced reuse and queueing, not an isolated causal proof of scheduler or cache thrashing; task behavior and thermal effects were not independently controlled.
Recommend four normal workers, an eight-job policy ceiling as deliberate headroom, and one active job initially for interactive, heavy-build or near-100K-history work. Eight is not an optimum or a guarantee of correctness. Keep waiting work in Emm and reserve execution slots in one Emm-owned executor launching the existing Hermes CLI. That executor is proposed, not installed; the current ceiling requires coordinator discipline. No additional backlog-size cap, second task board or adaptive scheduler is justified by this experiment.
The timing recorder did not recognize vLLM 0.32.0's reasoning-delta field. Consequently, the timeline shows only observed content/tool streaming; its pending intervals can include reasoning, prefill and queueing. It understates full generation overlap and is not a GPU-idle trace. We make no first-token-latency claim from this recorder. Whole-job elapsed time, throughput and server telemetry are unaffected. Per-request metadata is additionally incomplete for the cancelled sixteen-job burst: ten server request starts have no completed proxy record, consistent with handlers lost at shutdown; their ownership cannot be conclusively recovered.
One delayed single-job burst contained an extra unattributed server request. Before the larger results completed, we excluded it and scheduled a replacement after the series. All original evidence remains preserved. The primary comparison above includes the replacement. Other limitations are the homogeneous task, warm prefix cache, ascending job-count order, at most two repetitions, and short histories. Heavy builds, compaction and near-100K histories need their own tests.
Hermes provides several different limits. Its API adapter's max_concurrent_runs rejects new runs with HTTP 429 when full; callers retain the waiting work. Optional Kanban dispatch instead retains task cards and starts eligible work by priority and age. In this installed version its unset cap derives a RAM-based maximum of eight, while no configurable waiting-card count limit was found. These controls do not cap independent CLI launches. Our proposed workflow keeps the backlog and authoritative admission count in Emm, with one bounded executor launching the existing Hermes CLI; that integration is not yet installed. An additional API server or task board is not needed just to limit jobs. A controlled source-handler reproduction found a resumed-session handoff that could admit two runs at a cap of one, so the native cap alone is not yet sufficient evidence of a strict bound. Reproduction code, recorded reproduction. Pinned API implementation, pinned Kanban implementation.
Download the Hermes CSV table, JSON results, system counter summaries and detailed methodology.