I spent a good part of this week working on a color system to token tool called dtg. It’s a somewhat niche tool that solves a long standing problem that I have. I’ll have more to say about it shortly, but suffice to say, I’m real stoked about it.
Axle got a 0.30.2 bump. Bug fix and a refinement to the prompt compactor API. It feels pretty good now. I’ve been test driving [axle-code])(https://github.com/johncch/axle-code) on the new stealth model on open router, ox alpha, and I’m overall really happy with how rock solid Axle is. Occasionally, I wished that I can compile typescript down to binary (yes I know about scriptc) just to really get access to real multithreading, but as it stands everything is fast enough.
Speaking of ox alpha, rumor has that it’s a GLM or GLM-based fine tune. It definitely has some Claude-isms in the text it emits (and less so GPT-isms). After running it for a bit, I’ll pin it at around a below Sonnet, above Haiku level of intelligence. In my subjective measure, coding ability is be inversely proportional to the amount of frustration I feel while using it. This new model often runs itself into a quagmire and needs me or a better model to bail it out; this is the first time I’ve had to revert my git history while using an AI agent.
I’m making little progress on the eval work that I’ve been mulling on. The approach that I’m taking — a test harness vocabulary built on knobs on LLM as a judge – feels like a dead end. It all still feels like syntactic sugar and not a meaningful step forward. Part of it is that it’s really hard to know what correct means over a long horizon, the other is that even if I do, I don’t know if I could trust it without seeing it.