Everyone else is talking about evals these days. Unsurprisingly, since we are at the phase where AI adoption is hitting critical mass.
There are two ways to know a system. You can know it from the inside, by holding the rule that generates its behavior: I built this, so I know how it works and what it will do. Or you can know it from the outside: when it does this, it needs to do that; quacks like a duck, is a duck and all that. Roughly speaking, it can be mapped to implementation versus specification.
In traditional software teams, software engineering and product management are the two equivalents. With AI doing most of the software writing these days, the location of the inside knowledge gets displaced. Prior to AI, the software engineer in the code base absorbs the burden of a lot of the complexity for free. AI removes the hiding spot, the thing still gets built, but no accountable human acquires the inside knowledge along the way.
I want to recall Tesler’s law of conservation of complexity. In Tesler’s formulation, there is a minimum amount of complexity that is either handled by the programmer or the user. In this case however, the knowledge of the complexity that is inherent to functioning software needs to be captured by implementation (inside knowledge) or verification (outside knowledge).
All the chatter around evals and dark factory patterns is about this. At the end of the day, someone needs to know that the system is doing what it’s supposed to do, and while the work is easier, I don’t know if the knowing can ever be.