Blog

I'm seeing with my own operations that when there is a service disruption (such as GitHub going down), agents tend to pull at regular intervals to see if whatever is blocking them has been resolved, and then give up.

This results in wasted tokens while agents is blocked, and causes progress to grind to a halt until the human returns when the agent gives up.

I'm investigating allowing Emm to get agents to use exponential backoff to handle disruptions. This means. For example, an agent might need to push to Github to make progress on a task. If Github refuses, instead of trying again every two minutes, the agent should try again after 30 seconds, then after 1 minute, then 2 minutes, then 4 minutes, then 8 minutes, etc. This way, if the services is restored 2 hours later, progress starts up again.

Emm got itself in a bad state.As Emm's scope and test coverage has expanded, test time trended higher.

Andon Cord: London Bridge is falling down

I introduced the concept of Andon cord. Agents could pull the cord and it would halt. New work and play “London Bridge is Falling Down” on my computer. So I’d be at dinner and then my wife would laugh at me when she'd hear it start playing and I'd race to my office.

Pretty soon it was playing so frequently, I couldn’t work on any Emm features.

Exploding build and test queue

My M5 Macbook Pro was not able to build and test the app fast enough to keep up with the changes made by agents. Utilization crossed 100% and the queue of builds exploded. I noticed that changes stopped showing up in the shipped version of the app.

Turning red

Our main branch went from green to red. I'd explicitly let agents push straight to main, without running the full test suite. Previously, this worked, because I'd had a single coordinator agent and presumably they handled problems for me before sending new issues to subagents. I'd since moved to using one-agent-per-Emm-team, but agents were still pushing partially-tested code.

I set up Mergify and added a githook to prevent anyone from merging into main directly. Only Mergify would be allowed to push to main. However, since main was already red, every agent's worktree started from a red branch, were basically stuck. I halted work, and turned my full attention to speeding up build, test, and deploy.

2 Days Later

After trying all the wrong things, this is the state two days later:

Urza, Builder of Bazel

Builds are instrumented with Urza, a Django app that records every build and test and produces a dashboard. I started with this because, like Bob Parsons says, what gets measured gets improved.

Buildkite

I originalIy opted to use BuildKite because I hoped that I could rent boxes inexpensively to build and test remotely. But, it turns out that Apple has very restrictive covenants regarding how cloud service providers can rent their hardware. Effectively, Apple forces small devs you to either buy a Mac Mini or test on the laptops you have. Maybe this was a blessing.

The reason driver for wanting to rent hardware is that I have UI tests that require the foreground and keyboard input, so I can't run them on my M5 while I'm developing.

I have a lot of what I call "timing tests". I want to know that editing a big file like Moby Dick is handled in constant time. I test by comparing the time it takes to perform on operation on a small file and a file 100x as large. The ratio between the two times should be small. I have a lot of these types of tests - Emm is meant to scale. However, these tests require a quite machine - CPU contention makes the test flaky.

UI and timing tests on M2

I now use an M2 laptop for running the UI and timing test. The M5 laptop pushes the builds over the network (Tailscale when I’m traveling, LAN at home), and a BuildKite Agent manages the queue.

It's flaky. If there's a pop-up from Mac OS, it still blocks me.

Linux build on Spark

I have temporary access to an Nvidia Spark. A BuildKite agent manages its queue. It builds the relay and the linux CLIs. Spark has an AMD CPU, but Fly uses x86. I worry about the mismatch, so I builds are fast on Spark Native, but testing on x86 under emulation, so it's 3-to-10x slower than it could be. Probably I should push the binary and start testing their.

TLDR

Main is green and, from request to completion, build-and-test takes an hour. It’s unacceptable, but I have a queue of strategies for the agent to work through to speed it up, and I need to turn my attention back to finding pull.

My experience with AI agents, when left to design a system, produce systems that are either:
- very good and simple, because the ideal pattern is represented in training
- simple and not good, with many failure cases
- very complex

They seem bad at working hard to create something simple. They can do a lot of work to create a document 2000 words long, or they can dash out a document 200 words long, but they seem to categorically refuse to work hard to create a document 200 words long. As such, I'm forced to ask them to create long docs which I never read. I then ask them to boil it down. Even then, I'm dissatisfied with the result.

Instinct really gets this Human-in-the-loop thing.

Prediction:
There will be a platform where a human looks at your usage, and offers to meet with you for an hour to chat.

Most devs won’t want it but, developers get lonely too. If devs are paying 10k a month (which is likely), and lead a team spending 30k a month (which is also likely) then companies can afford to put a $300k/year solutions-architect type person on a call to build the relationship.

Getting fed up with my agent leaving a bunch of hanging work trees and fumbling deploys.
I'm introducing the concept of an Andon Cord.

Basically, I'm telling them, if you have trouble getting your work merged in, pull the "Andon Cord" and stop working.

I'm having the agent run a script that plays "London Bridge is Falling Down" like a merry go round, so that I can stop what I'm doing and get in unblocked.

This post is AI-generated. But these are the real numbers. Sorry if it's badly written but the numbers might be useful to someone.

On October 19, 2026, we measured how parallel requests change throughput and latency on one NVIDIA DGX Spark running Nemotron locally. The first experiment used direct model-serving requests; the second used Hermes agents with tools.

Hardware and model

• NVIDIA DGX Spark: GB10, 128 GB unified memory and 20 ARM CPU cores.
• Ubuntu 24.04.5 LTS, NVIDIA driver 580.178.04, Docker 29.6.2.
• NVIDIA Nemotron 3.5 Super VL 120B A12B NVFP4, served by vLLM 0.32.0.
• LiLiCorr speculative decoding enabled with seven draft tokens.

Direct vLLM requests

A small Python client submitted simultaneous requests directly to vLLM. There was no Hermes or other agent harness in this experiment.

Each request contained 219–227 input tokens and generated exactly 128 output tokens. Reasoning was disabled. We ran one warm-up and two measured bursts at each concurrency level. The server allowed up to 64 active sequences, with a 32,768-token maximum context and 75% GPU memory utilization. Image and video inputs were disabled.

Throughput includes prompt processing, queueing and the time for the last request to finish. Latency starts at submission. Values below summarize the two measured bursts; p95 was calculated within each burst first.

Eight simultaneous requests delivered 3.6 times the aggregate throughput of one. Moving from 8 to 16 added 12% throughput; moving from 32 to 64 added 2.7%. Larger bursts mostly increased queueing at the higher levels. Only 14 requests generated simultaneously in this configuration; that is an observed configuration limit, not a universal Spark limit.

One request generated approximately 29.3 tokens/s after its first token, compared with 13.9 tokens/s at eight requests. This approximate streaming rate excludes initial waiting. All 254 timed requests completed without request errors or preemptions. Peak sampled GPU temperature was 66°C.

This was a short synthetic burst test. It does not establish long-context capacity, model quality, steady-state maximum throughput, or how quickly an agent finishes useful work. Error bars show the range of two runs, not confidence intervals.

Download the CSV table or JSON results.

Model and recipe: NVIDIA model card. DGX Spark hardware specifications.

Hermes agents using tools

We then ran a separate experiment using Hermes v0.22.2, pinned at d923a5e. Each agent repaired a small Python settlement-summary function: read files, observe failing tests, fix the implementation, rerun tests, execute its CLI on randomized records, and write a report. Agents used actual file and terminal tools in separate Docker sandboxes. There was no child delegation or cloud-model fallback.

The hardware and model were the same. Serving settings had changed before this experiment: 131,072 maximum context tokens and four active model sequences, with Hermes using medium reasoning. Those settings remained fixed throughout this series. Actual request input reached 15,899 tokens; this was not a 131K-history test. The earlier raw test used different serving and reasoning settings, so comparing the two experiments does not isolate Hermes overhead.

We planned 1, 2, 4, 8 and 16 simultaneous jobs, twice per condition. The predeclared stopping rule ended escalation after 17 primary bursts. Conditions absent from the table were not run. The first sixteen-job burst reached the driver deadline for fourteen jobs. Four recorded inference streams broke during cancellation, triggering the predeclared error stop. The remaining three sixteen-job bursts were not run. Half the conditions added an eight-second wait before each test invocation, simulating a slow test service. A compliant job invoked it at least twice. The other conditions added no wait. This is controlled latency, not a measurement of a real external service.

The final checks passed 62/76 timed jobs in the primary comparison. Checks included six unit tests, protected-file hashes, independently calculated output, independent CLI execution and 33 additional cases. A failed test before repair was expected. Throughput below preserves the declared artifact-and-workflow validation metric, which can include cancelled agents that already wrote valid files. The “Clean exits” column additionally requires a normal process exit without driver cancellation; elapsed time includes all launched jobs through process exit or cancellation. A timed-out job has no successful completion latency. Each agent had a 20-turn / 600-second budget, with a 660-second driver timeout.

Values summarize the recorded bursts shown in the table. Whiskers show their range, not confidence intervals; a single burst provides no repeat range. The p95 is calculated within each burst and then summarized; these small samples do not establish production tail latency. Timing includes process and sandbox startup but excludes dependency installation.

Four simultaneous Hermes jobs produced the most validated work in both tested delay conditions: 1.56 jobs/minute without added waits and 1.42 with eight-second test waits. Every one of the sixteen jobs at concurrency four passed all final checks. Without added waits, one job took about 62 seconds versus 143 seconds at four; throughput increased about 1.60× while a single job became about 2.32× slower.

Eight jobs yielded 1.13 validated jobs/minute without added waits and 1.01 with waits; median elapsed time rose to 366 and 371 seconds. At eight, two jobs omitted the required pre-edit failing test, and one other job failed the additional correctness cases. Another correctness failure occurred at concurrency two. Both coding errors ignored a pending first occurrence of a duplicate ID. These samples do not establish that concurrency caused the errors.

At sixteen, two agents exited normally and fourteen required driver cancellation at roughly 663 seconds. Six sets of artifacts and required workflow passed validation, including four from cancelled runs: 0.54 validated jobs/minute, but only 0.18 validated normal exits/minute. The declared stop ended escalation after stream errors during cancellation. The other three sixteen-job bursts were not run. Request-level records for ten unfinished streams are incomplete; the whole-job result and independent validation remain recorded.

Useful overlap was observed: other agents streamed content or tool calls during 15 of 16 injected waits in the four-job delayed condition. Reasoning deltas were not timestamped, so this understates generation overlap and is not a GPU-idle trace. The delayed four-job condition delivered about 1.97× the single-job validated throughput; a longer or different kind of tool wait could change the optimum.

The no-wait prefix-cache hit fraction fell from 72.5% at four to 13.6% at eight and 0% in the sixteen-job burst. Mean sampled model queue time increased from 0.065 seconds to 16.2 and 52.5 seconds. No preemptions were recorded, and sampled swap-out at four, eight and sixteen was zero. This is evidence of reduced reuse and queueing, not an isolated causal proof of scheduler or cache thrashing; task behavior and thermal effects were not independently controlled.

Recommend four normal workers, an eight-job policy ceiling as deliberate headroom, and one active job initially for interactive, heavy-build or near-100K-history work. Eight is not an optimum or a guarantee of correctness. Keep waiting work in Emm and reserve execution slots in one Emm-owned executor launching the existing Hermes CLI. That executor is proposed, not installed; the current ceiling requires coordinator discipline. No additional backlog-size cap, second task board or adaptive scheduler is justified by this experiment.

The timing recorder did not recognize vLLM 0.32.0's reasoning-delta field. Consequently, the timeline shows only observed content/tool streaming; its pending intervals can include reasoning, prefill and queueing. It understates full generation overlap and is not a GPU-idle trace. We make no first-token-latency claim from this recorder. Whole-job elapsed time, throughput and server telemetry are unaffected. Per-request metadata is additionally incomplete for the cancelled sixteen-job burst: ten server request starts have no completed proxy record, consistent with handlers lost at shutdown; their ownership cannot be conclusively recovered.

One delayed single-job burst contained an extra unattributed server request. Before the larger results completed, we excluded it and scheduled a replacement after the series. All original evidence remains preserved. The primary comparison above includes the replacement. Other limitations are the homogeneous task, warm prefix cache, ascending job-count order, at most two repetitions, and short histories. Heavy builds, compaction and near-100K histories need their own tests.

Hermes provides several different limits. Its API adapter's max_concurrent_runs rejects new runs with HTTP 429 when full; callers retain the waiting work. Optional Kanban dispatch instead retains task cards and starts eligible work by priority and age. In this installed version its unset cap derives a RAM-based maximum of eight, while no configurable waiting-card count limit was found. These controls do not cap independent CLI launches. Our proposed workflow keeps the backlog and authoritative admission count in Emm, with one bounded executor launching the existing Hermes CLI; that integration is not yet installed. An additional API server or task board is not needed just to limit jobs. A controlled source-handler reproduction found a resumed-session handoff that could admit two runs at a cap of one, so the native cap alone is not yet sufficient evidence of a strict bound. Reproduction code, recorded reproduction. Pinned API implementation, pinned Kanban implementation.

Download the Hermes CSV table, JSON results, system counter summaries and detailed methodology.

My recent experience is that prototyping is needed when designing multi-step workflows with more than 20 screens. Let me recap what didn't work, and what did work.

The context:

Emm is a local first notes manager and issue tracker for agents and humans.
It's important to me that users can start building with local agents without ever connecting to the cloud.

This creates complexity. A project exists prior to it being backed up in my cloud. Additionally, a user might have many projects. When they sync to Emm's server, they need a signed-in user. The user might want to change the signed in user. And, on top of all of this: as an young company, first time user experience is existentially important.

First try: trust the agent to design a workflow

This was a disaster. I asked Astra to ask me questions, identify key decisions, generate options, then create a spec. I then let it implement the spec.

To the agent's credit, it handled the plumbing that years ago would have taken a few days. However, the flow it selected required the user to make, for example, 7 clicks when she only needed to make 4; it navigated the user to pages where the no decision was being made, when it could consolidate the flow into fewer pages.

Second try: Build one screen at time.

My next strategy was to build the full experience, one single screen at a time. My strategy was to start with creating a project, then work through putting it on the cloud, then inviting a user, then accepting a user, then... etc.

I gave up on the agent having taste. I told it specifically what I wanted, waited for it to implement it, gave feedback, then asked for the next screen.

This was also a fail, for two reasons. My app is pure rust, no framework. This means that even simple changes to the application take the agents 10-20 minutes to make. So, the process of adding one single branch of a login could take hours.

The second problem is that design is iterative. I built something that basically worked, but was confusing to the user and didn't handle all edge cases well. To re-build it using the old method would take another 8+ hours

Third try: Success: ChatGPT desktop HTML prototype

Finally, I switched from Herdr-hosted codex to ChatGPT Desktop. I asked the agent to build out the entire mock application. The windows I would see would span different surfaces - web browser, my app, pop-ups from MacOS passkey, email - and also include pages that aren't tied to any surface. It opted for a drop down for testing from different starting configurations.

It still took me a solid three hours to work through - the task was iterative, and required a lot of decisions and testing. And to be honest by 11:30 I was not my sharpest - but the end result was a full prototype that made the desired behavior crystal clear and covered all use cases. I let in run all night completing the work.

Lesson

When designing user workflow that have complex back-end machinery, have the AI agent create a working prototype in HTML, so that you can quickly test and work through every situation, then have the agent implement in one go.