Package dev.dokimos.core.evaluators.agents

Class ToolTrajectoryEvaluator

java.lang.Object

dev.dokimos.core.BaseEvaluator

dev.dokimos.core.evaluators.agents.ToolTrajectoryEvaluator

All Implemented Interfaces:

Evaluator

public class ToolTrajectoryEvaluator
extends BaseEvaluator

Evaluates an agent's tool-call trajectory against an expected trajectory using a selectable match mode.

This is a deterministic, glass-box evaluator. It compares the sequence of tool calls the agent made against an expected sequence, where two calls match when their names are equal and their arguments match under an ArgumentMatcher. Argument matching is configurable per tool via ToolTrajectoryEvaluator.Builder.argumentMatcher(String, ArgumentMatcher); the default matcher compares arguments tolerantly (ArgumentMatcher.tolerant()). Use ToolTrajectoryEvaluator.Builder.argumentMatcher(ArgumentMatcher) with ArgumentMatcher.of(ArgMatchMode.IGNORE) to compare names only.

Match modes:

• ToolTrajectoryEvaluator.MatchMode.STRICT — same length, same order, each position matches. Binary score.
• ToolTrajectoryEvaluator.MatchMode.IN_ORDER — expected appears as an ordered subsequence; graded by longest common subsequence.
• ToolTrajectoryEvaluator.MatchMode.ANY_ORDER — order-independent overlap; graded.
• ToolTrajectoryEvaluator.MatchMode.SUPERSET — every expected call has a distinct match in actual (extra actual calls allowed). Binary score.
• ToolTrajectoryEvaluator.MatchMode.SUBSET — every actual call has a distinct match in expected (missing expected calls allowed). Binary score.
• ToolTrajectoryEvaluator.MatchMode.PRECISION — matched / number of actual calls.
• ToolTrajectoryEvaluator.MatchMode.RECALL — matched / number of expected calls.

The unordered modes use a maximum bipartite matching, so repeated tool names are counted optimally even with a stricter per-tool argument matcher. No LLM is required.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ToolTrajectoryEvaluator.Builder	Builder for constructing the evaluator.
static enum	ToolTrajectoryEvaluator.MatchMode	The comparison mode for evaluating a trajectory.

Method Summary

Modifier and Type	Method	Description
static ToolTrajectoryEvaluator.Builder	builder()	Creates a new builder for constructing the evaluator.

Methods inherited from class dev.dokimos.core.BaseEvaluator

evaluate, evaluateAsync, evaluateAsync, name, threshold

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

builder

public static ToolTrajectoryEvaluator.Builder builder()

Creates a new builder for constructing the evaluator.

Returns:

a new builder