Speaker Lineup with Talk Titles and Abstracts

Agents have goals; they plan and act to achieve their goals, often alternating between  planning and action steps to refine their plans after reflecting on the results of their actions. Observing that agent failures arise when their goals, plans, and actions are not aligned, we introduce a framework for evaluating and improving an agent’s GPA or Goal-Plan-Action alignment. We create a set of five evaluation metrics – Goal Fulfillment, Plan Quality, Plan Adherence. Goal Fulfillment measures how well an agent’s actions achieves its goals. Logical Consistency checks that an agent’s actions are consistent with prior actions that it has taken. Execution efficiency checks whether the agent executed in the most efficient way to achieve its goal. Plan Quality checks whether an agent’s plans are aligned with its goals. Finally, Plan Adherence checks if an agent’s actions are aligned with their plan. Our experimental results on two benchmark datasets – an internal dataset for the Snowflake Intelligence data agent and the public GAIA dataset – shows that this framework  (a) provides a systematic way to cover a broad range of agent failures; (b) exhibits strong agreement between human and LLM judges, ranging from 80% to over 95%; and (c) localizes errors with 85% agreement with human annotations, thus enabling targeted improvement of agent performance.