The procedural compliance point hits close to home. At https://getechostack.com, we evaluate business signals across calls, forms, tickets, and emails, and the lesson we keep relearning is that output correctness is the wrong success metric. A call can end with a booked meeting and still have skipped qualification steps, missed escalation signals, or fired the wrong downstream action. Nobody catches it without trace-level auditing.
Our answer was manifest-driven evaluation: explicit criteria defined once, applied deterministically across every channel. It's not just about getting a decision. It's about whether the agent reached that decision through the path you specified, with the constraints you set honored throughout.
Your framing of governance as org chart design is exactly right. What we've ended up building is effectively a compliance layer on top of evaluation: playbook enforcement, structured outcome verification, audit trails. Not because we planned for it, but because production deployments demanded it.
The agent recommendation loop is the part I hadn't fully articulated before. Worth thinking hard about for anyone building infrastructure in this space.
The procedural compliance point hits close to home. At https://getechostack.com, we evaluate business signals across calls, forms, tickets, and emails, and the lesson we keep relearning is that output correctness is the wrong success metric. A call can end with a booked meeting and still have skipped qualification steps, missed escalation signals, or fired the wrong downstream action. Nobody catches it without trace-level auditing.
Our answer was manifest-driven evaluation: explicit criteria defined once, applied deterministically across every channel. It's not just about getting a decision. It's about whether the agent reached that decision through the path you specified, with the constraints you set honored throughout.
Your framing of governance as org chart design is exactly right. What we've ended up building is effectively a compliance layer on top of evaluation: playbook enforcement, structured outcome verification, audit trails. Not because we planned for it, but because production deployments demanded it.
The agent recommendation loop is the part I hadn't fully articulated before. Worth thinking hard about for anyone building infrastructure in this space.
thanks for sharing
Great write up of the conference! Appreciate the shout-out to Hyperparam
Congrats on winning best paper! well deserved