Package dev.dokimos.core.evaluators.agents

Class PlanAdherenceEvaluator

java.lang.Object

dev.dokimos.core.BaseEvaluator

dev.dokimos.core.evaluators.agents.PlanAdherenceEvaluator

All Implemented Interfaces:

Evaluator

public class PlanAdherenceEvaluator
extends BaseEvaluator

Evaluates whether an agent's executed tool calls followed its stated plan using a rule-based check and an optional LLM check.

The plan is read from actualOutputs[reasoningStepsKey] (default "reasoningSteps") as a List<String> and the tool calls from actualOutputs[toolCallsKey] (default "toolCalls"), matching AgentTrace.toOutputMap().

Checks performed:

• plan_present (rule): a plan is present
• plan_followed (LLM): the executed tool calls followed the stated plan

Without a judge LLM, only the rule-based check runs. The score is the fraction of checks that passed. No tool calls short-circuits to a score of 1.0, and a missing tool calls key throws.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	PlanAdherenceEvaluator.Builder	Builder for constructing the evaluator.

Method Summary

Modifier and Type	Method	Description
static PlanAdherenceEvaluator.Builder	builder()	Creates a new builder for constructing the evaluator.

Methods inherited from class dev.dokimos.core.BaseEvaluator

evaluate, evaluateAsync, evaluateAsync, name, threshold

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

builder

public static PlanAdherenceEvaluator.Builder builder()

Creates a new builder for constructing the evaluator.

Returns:

a new builder