unfiltered

Axle 0.30 is a big milestone in a few ways:

1. It marks the 30th release since I’ve started publishing the library consistently since June of last year. I typically release when I need to integrate changes into my core projects, and minor releases are when I make significant additions or breaking changes. That’s quite a pace, if I may say so myself.
2. More importantly, the compaction API has arrived at a good place. 0.30 solved a design conundrum I’ve been facing for a while, that is, what is the role of History — the dual objects of Messages and Turns — when it can be modified by an external actor. The answer is by removing Turn, the presentation layer, out of the Agent, and reduce the surface area of data that Agents work with. Now, the data model is coherent with a bit of conceptual tradeoff which we can bridge with built-in functions.

To celebrate, and to make sure I’m explaining the conceptual model well, the documentation site got a content revamp.

Sunnyday shipped skills. It’s something I’ve been wanting to do for a while, and having a concrete use case finally gave me a reason to built it.

The implementation went through a couple of cycles. The starting point was simple enough: it walks like a file and quacks like a file. But it soon became apparent that skills do indeed have a special status in an agent platform. I’ve settled on treating skills as files in the UI while handling them differently on the backend, which let’s me keep the UI and entry points simple. Though I doubt this is the last time I’ll have to think about it.

On the Sunnyday side:

• Agent sandboxes per session are fully resumable now, up to a maximum of 30 days. This sets up each agent up to take follow up commands, which will open up new avenues of interacting and correcting remote agents
• Assistant is now in beta. You can now prototype, create, and update Agents using the Assistant. This is something I’ve been mulling on for a while; as much as I think the UI is remarkably intuitive, the blank screen problem is still tricky to design around. This provides a way to turn sketches into infrastructure builds really quickly.

Because the work was based on Axle (we shipped an update for this around the internal ontology — 0.29.0), and most of the work took advantage of features in Axle recently, it was quick to come together. For me, this sets up some further thinking around conversations as the primary UI that I’ve been mulling about, and I believe will make some of the deeper cutting features, such as self-serve custom evals, possible. AI Agents are very good at pulling existing snippets together.

I’ve been thinking about the next iteration of axle-cli. I’ve been building axle-code for fun on the side, and it has been illuminating to see what features require more work (harness) and what works “out of the box”. Spoiler alert: frontier AI models are coding models by default. I am going to need a more sophisticated CLI soon: it’s going to look a bit like Cowork but with the conveniences of task runners. More to come!

AI generates code an order of magnitude faster than any human can. Hand-writing code is no longer a tenable proposition, and so the effort has shifted to doing whatever we can to hold AI code to our own quality bar. Some hand wave it away, comparing it to the evolution of C over Assembly, arguing that over time AI is going to be so good that we never have to think about the lower level code anymore. Others reach for the dark factory, where we validate the inputs and outputs and never care about what happens in between. The most developed version of this is a methodology now, promising to turn vibe coding into agentic engineering.

I’ve written about the conservation of complexity. The desired output is functionality, and the burden of knowing the details and verifying that it performs to the standard is something that still needs to be performed. In the agentic engineering case, the work shifts from software design and planning to a combination of bringing in the relevant contexts, using the appropriate set of prompt incantations, asking all the right questions, and rigorous testing.

I can see the allure. With the right process and guardrails, we can coerce AI to produce quality software. That said, the fragility of quality is often due to established processes being upended by unknowns, and it is impossible for us or anyone to know everything ahead of time. If the price of producing software approaches zero, we can take inspiration from practices from a recent past: prototyping.

In a formal Design process, we use prototypes extensively to validate desired behaviors and outcomes, which then gets handed over to the engineering process. Why not extend the idea: use AI to create prototypes, learn, and then generate specs, validation, and tests from them. More importantly, we discard the prototypes and rebuild the code from ground up using these produced artifacts. If anything changes with the artifacts, we discard the code and generate it from ground up again.

This separation between spec generation and code generation is important. A lot of bugs in software happen because of workarounds accumulated over time. Instructing AI to “make small changes” sounds good until it becomes a volley of patches layered on top of each other that becomes impossible to reason about.

This might sound a tad radical in a “we’ll rewrite it later” never world. However, consider the strengths of an AI agent: they are able to endlessly and tirelessly transform a set of words into another and conform them to an infinite suite of tests. Perhaps if production code is only produced with clean and locked down specs, we can maintain the rigor required of an industrial process. And notice the trick here, a fully deterministic set of requirements that produces a verifiable set of outputs sounds like a compiler. In the course of the rise of generative AI, we’ve been endlessly, tirelessly trying to coax probabilistic AI into deterministic outcomes, and maybe one day we will succeed.

The week was all about trimming and pruning. Sunnyday went through an AI code review and came out with roughly a thousand lines less. Most of the savings came from the deletion of duplicated logic, and many of that from AI code written across different sessions.

I prefer a well refactored codebase because it simplifies the mental model that I have to hold in my head. AI doesn’t have to do that, it is tireless in how it reads and spits out code. That said, none of us are immune from the effects of a sprawling codebase—losing track of what things are supposed to be and subtle differences across branches.

I think a lot about the speed-knowledge tradeoff these days. If I go fast, which I can go much faster than I am today, I truly lose sight of what goes into my codebase. I maintain some oversight on code input, at least in codebases that I care about, and even then I’m surprised by some of the decisions when I do code review. Do we end up in a world where the discipline reorganizes itself into an outside looking in motion? Attention is truly the scarce resource of our age.

I wrote this down a few months ago while working on Sunnyday. Recording here for posterity.

There are 4 levels to understand if an agentic trace was successful. It breaks down to the following:

• L0: did the run complete? These are binary outcomes at the infra level, basically if there are errors at the infrastructure level that prevented a run completion. Common examples are whether the LLM provider is at capacity, less obvious ones can be an example of a misconstructed tool call that loops forever.
• L1: did the agent do what it planned? At this level, we ask the AI agent to pre-register what it thinks it’s going to need to do based on the information that is available at the message. Then we grade the completed run against the plan to check for gaps that we can fill in earlier.
• L2: did it do the work well? Independent of the plan, did the agent encounter any obstacles when doing the work. Two separate tracks here. The first is if the run was optimal, where we detect for stumbling, error recovery, path efficiency, looping and flailing. Then there is the counterpart where we check if the tools and infrastructure is providing the right level of access and guidance to the agent.
• L3: did the customer get what they wanted? This is a check on the work quality. The user started the agent to achieve a certain outcome; did the agent produce it? We start at examining whether deliverables are completed e.g., prose or pdf files, and then we can inspect the content of those things, whether they reflect the request, and furthermore we can delve into whether the content is accurate or aesthetic. This can probably be further broken down into multiple layers.

Axle 0.28.0: a release long in the making focused on completing the compaction API and steering (changelog).

The hardest bit was finding the tradeoff sweet spot between complexity and API design. I first started with steer() – an API that allows the user to send a message that both cuts the queue and inserts in the first opportunity – which I backed out off because it introduced multiple queues in the scheduling internals. The compromise was to do less internally and expose a more granular API: stop() for a graceful stop, clear() to empty the queue, and chaining them together with a send() for steering semantics. I’m still finding the right line between opinionated and flexible; I’ll try the APIs out in a few Axle-Orbit experiments and see how they fare.

Other things shipped: make Axle baseline checks run in parallel; minor updates to Sunnyday fixing bugs and bumping up models and dependency.

Evals is on my mind. Even outside of the hype, having tools to slice and dice transcripts and corpus of transcripts feels useful. I’ve been thinking a lot about the how, and so far the most tenable way for me to make progress is to start bottoms up. I build a lot of fixtures to run Axle against LLM APIs as smoke tests. They are pretty rudimentary now, and perhaps iterating on those will lead me down more interesting conceptual paths.

One thing I’m trying to get myself to do now is to learn by doing. I have a tendency to want to think through problems thoroughly before attempting them. That’s one way of working through a problem. But sometimes, the smart and right thing to do is to stop thinking and start chipping at it. I need more of that in my practice.

I’ve been thinking about what kind of work I want to do next. It then occurred to me: instead of thinking about problems, I can just work on them. The cost of making is dramatically lower these days, why not spend the time trying things and getting hands-on. The trick is picking problems small enough that I can finish. I’ll write up and post my findings here, so that I actually do.

Everyone else is talking about evals these days. Unsurprisingly, since we are at the phase where AI adoption is hitting critical mass.

There are two ways to know a system. You can know it from the inside, by holding the rule that generates its behavior: I built this, so I know how it works and what it will do. Or you can know it from the outside: when it does this, it needs to do that; quacks like a duck, is a duck and all that. Roughly speaking, it can be mapped to implementation versus specification.

In traditional software teams, software engineering and product management are the two equivalents. With AI doing most of the software writing these days, the location of the inside knowledge gets displaced. Prior to AI, the software engineer in the code base absorbs the burden of a lot of the complexity for free. AI removes the hiding spot, the thing still gets built, but no accountable human acquires the inside knowledge along the way.

I want to recall Tesler’s law of conservation of complexity. In Tesler’s formulation, there is a minimum amount of complexity that is either handled by the programmer or the user. In this case however, the knowledge of the complexity that is inherent to functioning software needs to be captured by implementation (inside knowledge) or verification (outside knowledge).

All the chatter around evals and dark factory patterns is about this. At the end of the day, someone needs to know that the system is doing what it’s supposed to do, and while the work is easier, I don’t know if the knowing can ever be.

This morning, I was multitasking. I spent a few minutes making Claude Code go, and while it’s revving I’m writing in my paper notebook. Something catches the corner of my eye, I looked up and went to tend to the AI’s request. It whirls again, so it’s back to the notebook. Back and forth. As you can imagine, neither turned out particularly well, although if I were to be fair, the robot did a much better job than my writing. After all, I was only the one multitasking.

We have long known that multitasking reduces our cognitive abilities. There are numerous credible studies on this topic. Yet everyone is dual or triple wielding coding agents these days. Everyone feels more productive, and they are. They also say the work is not better ¹.

My observation is that the current crop of AI coding agents and models sit in the anti-flow zone of productivity: there is a little too much down time in between prompts, yet not enough to be able to effectively do something else. Once you hit enter on a prompt, a typical coding agent can take anywhere from minutes to tens of minutes, and while it’s doing its thing there is nothing much the human observer can do. We can try to follow along, but the terminal outputs are not really comprehension friendly. Of course we get bored and we multitask.

As a thought experiment, I wondered what would happen if coding agents are a magnitude faster². If we can get instant gratification it might solve the desire to let the attention wander. Yet at the same time, it raises an interesting question: if the robots can generate thousands of lines of code in a matter of seconds, then how are we able to really understand what’s going on? The temptation will be to do more and understand less since it’s the path of least resistance. Consequently, we will drown in systems that we do not understand.

When I ask people about this, everyone says that taste is going to be the thing that differentiates them. Make sense, since our perspective and judgement is what we really bring to the table and affect the world around us. That said, to be able to render accurate judgement requires us to understand the thing that we are judging. If our comprehension is being overwhelmed, then our judgement is what’s being overwhelmed.

I don’t know if there’s a neat little solution to this puzzle. My sense is that the status quo is not sustainable and things will need to change: either we develop tools to help us hold and understand more complexity, or we will have to delegate. Either the craft matters, or it becomes utilitarian.

• I love designing and building UIs. Like love, love. It is work that combines product thinking, UX, systems, and visuals all at the same time and it tickles every part of my brain. There are easily five iterations to get to a satisfying outcome: original idea, wireframes, implementation, iterating through the trough of disillusionment, and then the transcendence.
• I’ve gotten to a good state with the Sunnyday Agent configuration page. Good for now at least; and I don’t use the word good lightly. There are still rough edges, and aspects of the page which are candidates for a teardown and redo in the near future as my understanding evolves. But at least everything feels and works well within the constraints it’s supposed to right now.
• That’s one of the great things about working with coding AI agents. They give me higher degrees of freedom in working and can express in higher fidelity. If I were to put it in an analogy, it’s akin to the transition from 2d to 3d. We gain a new dimension that brings it closer to the real thing.
• After working on agents for the last few months, I have a lot of thoughts about performance. All the benchmarks tend to flatten them down to a single dimension, and to be fair that’s what benchmarks are supposed to do. However, agent performance is a much more nuanced, especially because there is the what is not said gap between the lines of the instruction prompt. I will have more to say about that as I collect my thoughts.