I wrote this down a few months ago while working on Sunnyday. Recording here for posterity.
There are 4 levels to understand if an agentic trace was successful. It breaks down to the following:
- L0: did the run complete? These are binary outcomes at the infra level, basically if there are errors at the infrastructure level that prevented a run completion. Common examples are whether the LLM provider is at capacity, less obvious ones can be an example of a misconstructed tool call that loops forever.
- L1: did the agent do what it planned? At this level, we ask the AI agent to pre-register what it thinks it’s going to need to do based on the information that is available at the message. Then we grade the completed run against the plan to check for gaps that we can fill in earlier.
- L2: did it do the work well? Independent of the plan, did the agent encounter any obstacles when doing the work. Two separate tracks here. The first is if the run was optimal, where we detect for stumbling, error recovery, path efficiency, looping and flailing. Then there is the counterpart where we check if the tools and infrastructure is providing the right level of access and guidance to the agent.
- L3: did the customer get what they wanted? This is a check on the work quality. The user started the agent to achieve a certain outcome; did the agent produce it? We start at examining whether deliverables are completed e.g., prose or pdf files, and then we can inspect the content of those things, whether they reflect the request, and furthermore we can delve into whether the content is accurate or aesthetic. This can probably be further broken down into multiple layers.
Posted Jul 28, 2026