unfiltered

There has been constant fighting with the latest crop of AI models.

A few things I’ve noticed: AI models are much more eager to jump in and do instead of understand what you want. When they write, the output is jargon-filled and convoluted. Worst of all, the recent Opus 5 just flat out make things up; over the last few weeks, I’ve had multiple incidents where after I push on a specific point in a plan, the model tells me that it made a mistake and acquiesces.

While none of this is indicative of intelligence per se, these behaviors make the model unpleasant to use. It feels as if they don’t understand my goals, try to paper over subpar decisions, and the most egregious of all, lie. I know I’m using people words here, ascribing intentionality to floating point numbers computed across arrays of special-purpose silicon. However, language is inherently anthropomorphizing; it is what we use to express and communicate with each other, and now with computers.

The thing is, none of this is inherent to the mathematical properties that create artificial intelligence. This is all trained behavior. My guess is that the pursuit of AI models that perform well on benchmarks and long-horizon tasks that leads to choosing verifiable rewards over human feedback creates behaviors that look like this. Inattentive and dismissive.

I’ve long thought that intelligence is not a linear measure. I had a conversation recently with a friend about this where I argued that preferences–for example in language and art–are not something that can be captured by scaling laws. He disagrees. I was arguing that there are different dimensions to intelligence and maxxximizing one will diminish others. A better way to frame it would be to separate what it can do versus how it behaves.

Perhaps soon, people will start talking about IQ vs EQ again. Perhaps one day AI model makers will find the right balance between the know-it-all and the we-get-you. But I’m not holding my breath.

I spent a good part of this week working on a color system to token tool called dtg. It’s a somewhat niche tool that solves a long standing problem that I have. I’ll have more to say about it shortly, but suffice to say, I’m real stoked about it.

Axle got a 0.30.2 bump. Bug fix and a refinement to the prompt compactor API. It feels pretty good now. I’ve been test driving axle-code on the new stealth model on open router, ox alpha, and I’m overall really happy with how rock solid Axle is. Occasionally, I wished that I can compile typescript down to binary (yes I know about scriptc) just to really get access to real multithreading, but as it stands everything is fast enough.

Speaking of ox alpha, rumor has that it’s a GLM or GLM-based fine tune. It definitely has some Claude-isms in the text it emits (and less so GPT-isms). After running it for a bit, I’ll pin it at around a below Sonnet, above Haiku level of intelligence. In my subjective measure, coding ability is be inversely proportional to the amount of frustration I feel while using it. This new model often runs itself into a quagmire and needs me or a better model to bail it out; this is the first time I’ve had to revert my git history while using an AI agent.

I’m making little progress on the eval work that I’ve been mulling on. The approach that I’m taking — a test harness vocabulary built on knobs on LLM as a judge – feels like a dead end. It all still feels like syntactic sugar and not a meaningful step forward. Part of it is that it’s really hard to know what correct means over a long horizon, the other is that even if I do, I don’t know if I could trust it without seeing it.

Axle 0.30 is a big milestone in a few ways:

1. It marks the 30th release since I’ve started publishing the library consistently since June of last year. I typically release when I need to integrate changes into my core projects, and minor releases are when I make significant additions or breaking changes. That’s quite a pace, if I may say so myself.
2. More importantly, the compaction API has arrived at a good place. 0.30 solved a design conundrum I’ve been facing for a while, that is, what is the role of History — the dual objects of Messages and Turns — when it can be modified by an external actor. The answer is by removing Turn, the presentation layer, out of the Agent, and reduce the surface area of data that Agents work with. Now, the data model is coherent with a bit of conceptual tradeoff which we can bridge with built-in functions.

To celebrate, and to make sure I’m explaining the conceptual model well, the documentation site got a content revamp.

Sunnyday shipped skills. It’s something I’ve been wanting to do for a while, and having a concrete use case finally gave me a reason to built it.

The implementation went through a couple of cycles. The starting point was simple enough: it walks like a file and quacks like a file. But it soon became apparent that skills do indeed have a special status in an agent platform. I’ve settled on treating skills as files in the UI while handling them differently on the backend, which let’s me keep the UI and entry points simple. Though I doubt this is the last time I’ll have to think about it.

On the Sunnyday side:

• Agent sandboxes per session are fully resumable now, up to a maximum of 30 days. This sets up each agent up to take follow up commands, which will open up new avenues of interacting and correcting remote agents
• Assistant is now in beta. You can now prototype, create, and update Agents using the Assistant. This is something I’ve been mulling on for a while; as much as I think the UI is remarkably intuitive, the blank screen problem is still tricky to design around. This provides a way to turn sketches into infrastructure builds really quickly.

Because the work was based on Axle (we shipped an update for this around the internal ontology — 0.29.0), and most of the work took advantage of features in Axle recently, it was quick to come together. For me, this sets up some further thinking around conversations as the primary UI that I’ve been mulling about, and I believe will make some of the deeper cutting features, such as self-serve custom evals, possible. AI Agents are very good at pulling existing snippets together.

I’ve been thinking about the next iteration of axle-cli. I’ve been building axle-code for fun on the side, and it has been illuminating to see what features require more work (harness) and what works “out of the box”. Spoiler alert: frontier AI models are coding models by default. I am going to need a more sophisticated CLI soon: it’s going to look a bit like Cowork but with the conveniences of task runners. More to come!

AI generates code an order of magnitude faster than any human can. Hand-writing code is no longer a tenable proposition, and so the effort has shifted to doing whatever we can to hold AI code to our own quality bar. Some hand wave it away, comparing it to the evolution of C over Assembly, arguing that over time AI is going to be so good that we never have to think about the lower level code anymore. Others reach for the dark factory, where we validate the inputs and outputs and never care about what happens in between. The most developed version of this is a methodology now, promising to turn vibe coding into agentic engineering.

I’ve written about the conservation of complexity. The desired output is functionality, and the burden of knowing the details and verifying that it performs to the standard is something that still needs to be performed. In the agentic engineering case, the work shifts from software design and planning to a combination of bringing in the relevant contexts, using the appropriate set of prompt incantations, asking all the right questions, and rigorous testing.

I can see the allure. With the right process and guardrails, we can coerce AI to produce quality software. That said, the fragility of quality is often due to established processes being upended by unknowns, and it is impossible for us or anyone to know everything ahead of time. If the price of producing software approaches zero, we can take inspiration from practices from a recent past: prototyping.

In a formal Design process, we use prototypes extensively to validate desired behaviors and outcomes, which then gets handed over to the engineering process. Why not extend the idea: use AI to create prototypes, learn, and then generate specs, validation, and tests from them. More importantly, we discard the prototypes and rebuild the code from ground up using these produced artifacts. If anything changes with the artifacts, we discard the code and generate it from ground up again.

This separation between spec generation and code generation is important. A lot of bugs in software happen because of workarounds accumulated over time. Instructing AI to “make small changes” sounds good until it becomes a volley of patches layered on top of each other that becomes impossible to reason about.

This might sound a tad radical in a “we’ll rewrite it later” never world. However, consider the strengths of an AI agent: they are able to endlessly and tirelessly transform a set of words into another and conform them to an infinite suite of tests. Perhaps if production code is only produced with clean and locked down specs, we can maintain the rigor required of an industrial process. And notice the trick here, a fully deterministic set of requirements that produces a verifiable set of outputs sounds like a compiler. In the course of the rise of generative AI, we’ve been endlessly, tirelessly trying to coax probabilistic AI into deterministic outcomes, and maybe one day we will succeed.

The week was all about trimming and pruning. Sunnyday went through an AI code review and came out with roughly a thousand lines less. Most of the savings came from the deletion of duplicated logic, and many of that from AI code written across different sessions.

I prefer a well refactored codebase because it simplifies the mental model that I have to hold in my head. AI doesn’t have to do that, it is tireless in how it reads and spits out code. That said, none of us are immune from the effects of a sprawling codebase—losing track of what things are supposed to be and subtle differences across branches.

I think a lot about the speed-knowledge tradeoff these days. If I go fast, which I can go much faster than I am today, I truly lose sight of what goes into my codebase. I maintain some oversight on code input, at least in codebases that I care about, and even then I’m surprised by some of the decisions when I do code review. Do we end up in a world where the discipline reorganizes itself into an outside looking in motion? Attention is truly the scarce resource of our age.