All Classes and Interfaces
Class
Description
Base implementation of
ScopedRepository.Typed entry point for building an
EvalTestCase for the agent evaluators.Builder for constructing an
EvalTestCase that targets the agent evaluators.Wraps a complete agent execution trace for evaluation.
Builder for constructing agent traces.
Strategies for aggregating scores from multiple evaluation criteria.
A configured HTTP endpoint that receives a JSON alert when a completed run of one of its project's
experiments shows a significant pass-rate regression against its baseline.
CRUD over a project's regression-alert webhooks.
Delivers regression alerts to a project's enabled webhooks.
Tenant-scoped repository for
AlertWebhook.Entity-specific scoped finders for
AlertWebhook.Tenant-scoped implementation of the
AlertWebhook finders.Manages a project's regression-alert webhooks.
Public view of an
AlertWebhook.Exposes the per-evaluator judge-human alignment for a run: the agreement rate between automated
evaluator verdicts and human annotations.
Computes per-run, per-evaluator agreement between automated evaluator verdicts and human
annotations.
Per-run, per-evaluator agreement between automated evaluator verdicts and human annotations.
Agreement breakdown for a single evaluator across a run's annotated items.
A human reviewer's verdict on a single
ItemResult.Endpoints for the single human review annotation on a run item result.
Payload for creating or updating the annotation on a run item result.
Service for human review annotations on run item results.
A human reviewer's verdict on a single run item result.
Read model for an annotation on a run item result.
A scoped API key used to authenticate write requests against
/api/v1/**.Authenticator backed by two credential sources that coexist for backward compatibility: the
single legacy key configured via DOKIMOS_API_KEY (which maps to Role.ADMIN), and the
scoped keys stored in api_keys (each hashed, each carrying its own role and tenant).Servlet filter that authenticates
/api/v1/** by delegating credential resolution to an
Authenticator, then enforces role-based authorization on the resolved Principal.Manages scoped API keys.
Hashes raw API keys to the SHA-256 hex form stored in the database.
Configuration properties for API key authentication.
Mints and manages scoped API keys.
Metadata view of an
ApiKey.Controls how a tool call's arguments are compared against the expected arguments.
Strategy for comparing a tool call's actual arguments against the expected arguments.
Assertion utilities for evaluation-based testing.
A non-blocking task that produces a
TaskResult asynchronously.Strategy that resolves the
Principal behind an incoming request from its credential.Base class for implementing concrete evaluators.
The committed regression-gate baseline (format v1).
One item's projection.
Dataset summary metadata (advisory; never used for pairing).
One evaluator's recorded outcome for an item.
Judge provenance (advisory).
Build/judge context, kept separate from the measured signal and excluded from the byte-stability
guarantee.
Reads and writes the regression-gate baseline file, and reconstructs comparison-ready
RunResults from it.Optional metrics describing the LLM call that produced an item result.
Classification of how a metric or item changed between a baseline and a candidate.
Shared entity-to-core conversion and run-comparison invocation for the regression gate and the
per-case diff view.
Result of comparing two runs: the engine output, the pairing strategy, and the runs' loaded
item entities so callers (the diff view) can derive per-case input text without re-querying.
Evaluator that measures how relevant retrieved context chunks are to a user's input query.
Builder for constructing ContextualRelevanceEvaluator instances.
Represents the relevance score for a single context chunk.
Functional interface representing an application that can engage in
multi-turn conversations.
Orchestrates multi-turn conversations between a simulated user and an
application.
Builder for constructing conversation simulators.
Represents a complete conversation trajectory between a simulated user and an
application.
Builder for constructing conversation trajectories.
Request to register an alert webhook for a project.
Request to mint a scoped API key.
Response returned exactly once when a key is minted.
Request to register an OpenAI-compatible LLM connection.
Request to create or replace a trace eval rule.
A collection of examples for evaluation.
A named container that owns one or more immutable
DatasetVersions.Builder for constructing datasets.
JUnit ArgumentsProvider that loads
Examples from a Dataset.A single example within a
DatasetVersion.Marks a static field that supplies the
Reporter for DatasetSource tests.Tenant-scoped repository for
Dataset.Entity-specific scoped finders for
Dataset.Tenant-scoped implementation of the
Dataset finders.Thrown when a dataset cannot be correctly resolved or loading fails.
Resolves a dataset URI to a
Dataset.Singleton registry for dataset resolvers.
Opens and reports a
Reporter run for each DatasetSource test method.Collects the actual outputs and evaluation results a
DatasetSource test body produced
for one parameterized invocation.Service for server-owned versioned datasets.
Provides
Examples from a Dataset as arguments to a
parameterized test.An immutable snapshot of a
Dataset.One row of the per-case run-diff table: a single item compared across baseline and candidate,
with its per-evaluator old-to-new deltas.
A single evaluator's change on one item between baseline and candidate.
Exposes the per-case run-diff view: the same comparison the CI gate runs, presented as a full,
paginated table of every case with per-evaluator deltas.
Builds the per-case run-diff view for the UI.
Whole-run summary of a per-case run diff.
Combined per-case run-diff payload: the whole-run summary plus the first (or requested) page of
cases.
MCP server that exposes dokimos evaluation tools over stdio transport.
Async HTTP
Reporter that sends experiment results to a Dokimos server.Builder for
DokimosServerReporter.Describes a batch of items permanently dropped after all delivery retries were exhausted.
Thrown when a stored output value cannot be converted to the requested type by one of the typed
read accessors (for example
EvalTestCase.actualOutputAs(Class) or Example.expectedOutputAs(dev.dokimos.core.OutputType)).Entry point for evaluating Embabel agent runs with Dokimos.
Accumulates an Embabel agent's tool calls into a Dokimos
AgentTrace.Request to enqueue a server-side judge job for a run.
A unit of server-side scoring work: score every not-yet-evaluated item of a run with a single
evaluator configuration, using a registered
LlmConnection.Enqueues server-side judge jobs and reads the jobs registered for a run.
Lifecycle state of an
EvalJob.Public view of an
EvalJob, returned by the enqueue endpoint and the per-run job listing.The result of an evaluation.
Builder for constructing evaluation results.
A test case for evaluation.
Builder for constructing test cases with multiple inputs and outputs.
Defines a single evaluation dimension for trajectory evaluation.
Thrown when an evaluation cannot be executed successfully.
Evaluates test cases and produces scored results.
Change in a single evaluator's mean score between baseline and candidate.
Evaluator that checks for exact string match between actual and expected outputs.
A dataset example with inputs, expected outputs, and metadata.
Builder for constructing examples with multiple inputs and outputs.
An evaluation experiment that runs a task against a dataset and evaluates the
results.
Tenant-scoped repository for
Experiment.Entity-specific scoped finders for
Experiment.Tenant-scoped implementation of the
Experiment finders.Aggregated results from an experiment.
Utility class for exporting
ExperimentResult to various formats.Tenant-scoped repository for
ExperimentRun.Entity-specific scoped finders for
ExperimentRun.Tenant-scoped implementation of the
ExperimentRun finders.Evaluator that uses an LLM to check how much of the actual output is backed by the given context.
Resolves datasets from the filesystem.
Configuration for the server-free regression gate.
Builder for
GateConfig.How baseline and candidate items are paired.
What to do when an evaluator present in the baseline is missing from the candidate.
Exposes the CI regression gate.
Request to evaluate a CI regression gate for an already-ingested candidate run.
Verdict of a CI regression gate.
A per-item evaluator score drop.
A single regressed item, identified by its dataset item id when paired by id or by its
positional index otherwise.
A single evaluator's regression between baseline and candidate.
Evaluates a CI regression gate by comparing an already-ingested candidate run against a resolved
baseline run with the core
RunComparison engine and returning a pass/fail verdict.The result of a regression-gate comparison: the overall
status, the pass-rate move, the
regressed evaluators and cases, and any coverage-loss or threshold-drift warnings.A per-item evaluator score drop.
A single regressed item, identified by its dataset item id when paired by id or by its
positional index otherwise.
A single evaluator's significant regression between baseline and candidate.
Evaluator that uses an LLM to detect hallucinations in the actual output.
Marks an item batch (identified by idempotency key) as committed for a run.
Composite primary key.
Comparison of a single paired item across baseline and candidate.
The outcome of executing a single example: the example itself, the actual outputs the task
produced, the evaluator results, and optional metrics describing the underlying LLM call.
Shared, framework-internal JSON utilities backed by a single, immutable Jackson
ObjectMapper.Stores run records as a JSON array in a local file.
Unchecked failure raised when a judge HTTP call cannot complete.
The transactional steps of the judge worker, each in its own committed transaction so the worker can
make the LLM HTTP call between them without holding a database transaction open.
A detached view of an item result carrying only the fields the judge prompt needs.
Pairs an eval result with the id of the item result it belongs to, for batch persistence.
A language model used for evaluation.
Tuning knobs for the background judge worker, bound from the
dokimos.judge prefix.Drives a single judge scoring: builds the prompt from a criteria and the selected parameters, calls
the underlying
JudgeLM, and parses the response.The result of scoring one item: a numeric score, the judge's reasoning, and the pass decision.
Background worker that drains the judge job queue.
Utilities for integrating with LangChain4j.
A named, reusable pointer to an OpenAI-compatible endpoint used by the server-side judge.
The API surface an
LlmConnection's endpoint speaks.Tenant-scoped repository for
LlmConnection.Entity-specific scoped finders for
LlmConnection.Tenant-scoped implementation of the
LlmConnection finders.Registers and reads LLM connections.
Public view of an
LlmConnection.Resolves and protects the API key for an
LlmConnection.Evaluator that uses an LLM to evaluate outputs based on the specified
criteria.
Utility methods for processing LLM responses.
An LLM-based simulated user for multi-turn conversation testing.
Builder for constructing LLM simulated users.
Strategy for determining if a retrieved item matches an expected item.
A task that produces outputs together with optional
CallMetrics describing the
underlying LLM call.Represents a single message in a conversation.
The role of a message sender in a conversation.
A single typed run-metadata key-value pair for
DatasetSource.entries().A no-op implementation of
Reporter that does nothing.A
JudgeLM that calls an OpenAI-compatible chat completions endpoint over the JDK HTTP
client, with no vendor SDK.Utilities for integrating with the OpenAI Java SDK.
A
JudgeLM that calls an Open Responses endpoint (POST {baseUrl}/responses) over the
JDK HTTP client, with no vendor SDK.The OTLP
AnyValue union in its JSON encoding.The top level of the OTLP/HTTP JSON encoding of
ExportTraceServiceRequest.A single OTLP attribute: a key and an
OtlpAnyValue holding the typed value.Translates the protobuf
ExportTraceServiceRequest into the same DTO tree Jackson produces for
the OTLP/HTTP JSON encoding, so both encodings share one parser and persistence path.A resource and its attributes.
The spans emitted by one resource, with that resource's attributes (for example
service.name).The spans emitted by one instrumentation scope.
One OTLP span in the JSON encoding.
A span status.
Pure translation from the OTLP/HTTP JSON request DTOs into the server's own span model, with no
persistence.
A span translated from OTLP, carrying the merged attributes and derived input/output text.
The outcome of parsing: the valid spans and the count of malformed spans that were skipped.
A super-type token that captures a full generic type
T (including its type arguments) at
compile time so it survives erasure at runtime.Stable page envelope used by paginated endpoints.
Mirrors Spring's
PageableObject JSON shape.Mirrors Spring's
SortObject JSON shape.Evaluator that measures retrieval precision.
Builder for constructing PrecisionEvaluator instances.
Turns a model id and token counts into a USD cost, returning null when the model is unknown or a
count is missing (never throws, never fabricates a number).
Authenticated caller.
A project groups experiments.
Tenant-scoped repository for
Project.Entity-specific scoped finders for
Project, implemented over the tenant-scoped query helper.Tenant-scoped implementation of the
Project finders.Payload for promoting run item results into a new version of an existing dataset.
A single run item result to promote.
Evaluator that measures retrieval recall.
Builder for constructing RecallEvaluator instances.
Evaluator that checks if the actual output matches a regular expression pattern.
Published inside the run-completion transaction when a run regresses, and consumed only after that
transaction commits.
JSON body POSTed to a project's alert webhooks when a completed run regresses against its baseline.
Computes, inside the run-completion transaction, whether a just-completed run regressed against its
baseline, and publishes a
RegressionAlertEvent when it did.Compares a candidate experiment result against a committed baseline and produces a
GateVerdict.The run lifecycle around
RegressionGate: resolves whether to update, bootstrap, or compare
a baseline, performs the file I/O, and turns a FAIL verdict into an AssertionError.Reads the outside world the lifecycle depends on.
Interface for reporting experiment results to an external system.
Persistence layer for evaluation run records.
A run item surfaced in the review queue because it still needs a human verdict, carrying enough run,
experiment, and project context for a reviewer to act on it without opening the run first.
Surfaces run items that still need a human verdict so a reviewer can work through a single queue
instead of opening runs one at a time.
Role granted to an authenticated
Principal.Regression-comparison engine that compares a baseline set of runs against a candidate set.
Builder for
RunComparison.Result of comparing a baseline set of runs against a candidate set.
Detail view of a single run.
A handle representing an active experiment run.
Persistent record of a single evaluation run.
Single evaluator result for one example.
Detail for a single evaluated example.
Results from a single run of an experiment.
Status of an experiment run.
Summary view of a single run in a list.
Common tenant-scoped operations shared by every scoped repository.
Resolves dataset URIs of the form
dataset://<name>@<version> against a Dokimos server.Outcome of a statistical significance test on a paired comparison.
Functional interface for simulating user behavior in multi-turn
conversations.
Public view of a
TraceSpan, including its derived input/output text and flattened attributes.Serves the single-page React app and forwards client side routes to
index.html so a deep
link or a page refresh on a route such as /traces or /api-keys is handled by the SPA
router instead of returning 404.Utilities for evaluating Spring AI Alibaba (graph/agent) runs with the Dokimos
agent evaluators.
A Spring AI Alibaba agent run's output text paired with optional token usage, consumed by
SpringAiAlibabaSupport.measuredAsyncTask(Function, String, PriceTable).Utilities for integrating with Spring AI.
Compares the actual output against the expected output as JSON structures rather than as opaque
strings.
Builder for constructing structural match evaluators.
Controls how
StructuralMatchEvaluator compares an expected JSON structure against an actual
one.Evaluates whether an AI agent completed the user's requested tasks.
Builder for constructing the evaluator.
The outcome of executing a
MeasuredTask: the actual outputs and optional metrics
describing the underlying LLM call.Builds the SQL/JPQL tenant predicate for a
TenantScope against a tenant_id path.Immutable tenant visibility used by every scoped repository read and by service-side write stamping.
Reusable tenant-scoped query helper shared by every scoped repository implementation.
Derives the
TenantScope and principal id for the current request from the Principal
the auth filter placed on the request attribute.Default
ArgumentMatcher that compares arguments structurally with a few
deliberate tolerances.Builder for constructing the matcher.
Uses a judge LLM to assess whether tool call argument values are factually
grounded in the user's input and preceding tool call results.
Builder for constructing the evaluator.
Represents a single tool invocation made by an AI agent.
Builder for constructing tool calls.
Coercion helpers for turning a raw value into a typed
List<ToolCall>.Validates that tool calls are syntactically correct per their JSON schema definitions.
Builder for constructing the evaluator.
Checks whether the agent used the expected set of tools.
Builder for constructing the evaluator.
The comparison mode for evaluating tool correctness.
Describes an available tool's contract including its name, description, and JSON schema.
Builder for constructing tool definitions.
Evaluates tool description quality using a mix of rule-based checks and optional LLM checks.
Builder for constructing the evaluator.
Measures how efficiently an agent used its tools by detecting redundant calls.
Builder for constructing the evaluator.
Detects tool execution failures by inspecting
ToolCall.result().Builder for constructing the evaluator.
Implements the four MCP tool handlers for the dokimos evaluation framework.
Evaluates tool naming quality using a mix of rule-based checks and optional LLM checks.
Builder for constructing the evaluator.
Evaluates an agent's tool-call trajectory against an expected trajectory using a
selectable match mode.
Builder for constructing the evaluator.
The comparison mode for evaluating a trajectory.
A single distributed execution trace ingested over OTLP, holding one or more
TraceSpan rows.OTLP trace ingestion and read endpoints.
Full view of one trace: its metadata, all of its spans, and the online eval jobs scoped to those spans.
Decides which online eval jobs to create for a freshly ingested trace and persists them.
A unit of online scoring work: score one ingested
TraceSpan's derived output against one
TraceEvalRule's judge configuration.Lifecycle state of a
TraceEvalJob.The transactional steps of the trace eval worker, each in its own committed transaction so the worker
can make the LLM HTTP call between them without holding a database transaction open.
A claimed job paired with the span text snapshots the worker scores after the transaction closes.
Public view of a
TraceEvalJob, returned in the per-trace detail.A per-project rule that enqueues an online judge evaluation when an ingested span matches.
Manages per-project trace eval rules.
Tenant-scoped repository for
TraceEvalRule.Entity-specific scoped finders for
TraceEvalRule.Tenant-scoped implementation of the
TraceEvalRule finders.Manages per-project trace eval rules.
Public view of a
TraceEvalRule.Background worker that drains the online trace eval queue.
Result of an OTLP ingestion call.
Persists OTLP traces and enqueues online evaluations for matching spans.
How a
TraceEvalRule decides whether a span matches.Tuning knobs for trace ingestion and online evaluation, bound from the
dokimos.trace prefix.Online trace eval worker knobs, bound from
dokimos.trace.eval.Read side for ingested traces: a paginated list and a per-trace detail with spans and eval jobs.
Tenant-scoped repository for
Trace.Entity-specific scoped finders for
Trace plus the unrestricted ingestion and sweeper paths.Tenant-scoped implementation of the
Trace finders plus the unrestricted ingestion paths.Periodically deletes traces whose retention window has closed.
One span of an ingested
Trace.Compact view of a
Trace for the list endpoint, without its spans.Factory for creating pre-built evaluation criteria for trajectory evaluation.
Evaluates complete conversation trajectories using LLM-as-judge patterns.
Builder for constructing trajectory evaluators.
Payload for updating an alert webhook.
Payload for updating an LLM connection.
Factory for creating pre-built simulated user personas.