Emm got itself in a bad state.As Emm's scope and test coverage has expanded, test time trended higher.
Andon Cord: London Bridge is falling down
I introduced the concept of Andon cord. Agents could pull the cord and it would halt. New work and play “London Bridge is Falling Down” on my computer. So I’d be at dinner and then my wife would laugh at me when she'd hear it start playing and I'd race to my office.
Pretty soon it was playing so frequently, I couldn’t work on any Emm features.
Exploding build and test queue
My M5 Macbook Pro was not able to build and test the app fast enough to keep up with the changes made by agents. Utilization crossed 100% and the queue of builds exploded. I noticed that changes stopped showing up in the shipped version of the app.
Turning red
Our main branch went from green to red. I'd explicitly let agents push straight to main, without running the full test suite. Previously, this worked, because I'd had a single coordinator agent and presumably they handled problems for me before sending new issues to subagents. I'd since moved to using one-agent-per-Emm-team, but agents were still pushing partially-tested code.
I set up Mergify and added a githook to prevent anyone from merging into main directly. Only Mergify would be allowed to push to main. However, since main was already red, every agent's worktree started from a red branch, were basically stuck. I halted work, and turned my full attention to speeding up build, test, and deploy.
2 Days Later
After trying all the wrong things, this is the state two days later:
Urza, Builder of Bazel
Builds are instrumented with Urza, a Django app that records every build and test and produces a dashboard. I started with this because, like Bob Parsons says, what gets measured gets improved.
Buildkite
I originalIy opted to use BuildKite because I hoped that I could rent boxes inexpensively to build and test remotely. But, it turns out that Apple has very restrictive covenants regarding how cloud service providers can rent their hardware. Effectively, Apple forces small devs you to either buy a Mac Mini or test on the laptops you have. Maybe this was a blessing.
The reason driver for wanting to rent hardware is that I have UI tests that require the foreground and keyboard input, so I can't run them on my M5 while I'm developing.
I have a lot of what I call "timing tests". I want to know that editing a big file like Moby Dick is handled in constant time. I test by comparing the time it takes to perform on operation on a small file and a file 100x as large. The ratio between the two times should be small. I have a lot of these types of tests - Emm is meant to scale. However, these tests require a quite machine - CPU contention makes the test flaky.
UI and timing tests on M2
I now use an M2 laptop for running the UI and timing test. The M5 laptop pushes the builds over the network (Tailscale when I’m traveling, LAN at home), and a BuildKite Agent manages the queue.
It's flaky. If there's a pop-up from Mac OS, it still blocks me.
Linux build on Spark
I have temporary access to an Nvidia Spark. A BuildKite agent manages its queue. It builds the relay and the linux CLIs. Spark has an AMD CPU, but Fly uses x86. I worry about the mismatch, so I builds are fast on Spark Native, but testing on x86 under emulation, so it's 3-to-10x slower than it could be. Probably I should push the binary and start testing their.
TLDR
Main is green and, from request to completion, build-and-test takes an hour. It’s unacceptable, but I have a queue of strategies for the agent to work through to speed it up, and I need to turn my attention back to finding pull.