Skip to main content

Spring AI Integration

This page shows you how to evaluate a Spring AI application with Dokimos. You reuse your existing ChatClient and ChatModel, so you do not stand up a separate LLM client just to score answers.

What you get​

  • One-line judge: turn a Spring AI ChatClient or ChatModel into a Dokimos JudgeLM with SpringAiSupport.asJudge(...).
  • No extra setup: the judge runs on the same Spring AI infrastructure you already have.
  • Two-way conversion: move between Spring AI EvaluationRequest/EvaluationResponse and Dokimos EvalTestCase/EvalResult.

Step 1: Add the dependency​

Maven​

<dependency>
    <groupId>dev.dokimos</groupId>
    <artifactId>dokimos-spring-ai</artifactId>
    <version>${dokimos.version}</version>
</dependency>

Gradle (Groovy DSL)​

implementation 'dev.dokimos:dokimos-spring-ai:${dokimosVersion}'

Step 2: Make a judge​

A judge is the LLM that scores answers. You build one from a Spring AI component, then pass it to any LLM-based evaluator.

From a ChatClient​

Pass a ChatClient.Builder to SpringAiSupport.asJudge(...):

import dev.dokimos.core.*;
import dev.dokimos.core.evaluators.*;
import dev.dokimos.springai.SpringAiSupport;
import org.springframework.ai.chat.client.ChatClient;

ChatClient.Builder clientBuilder = ChatClient.builder(chatModel);

// Convert to JudgeLM
JudgeLM judge = SpringAiSupport.asJudge(clientBuilder);

// Use in evaluators
Evaluator correctness = LLMJudgeEvaluator.builder()
    .name("Answer Correctness")
    .criteria("Is the answer factually correct?")
    .evaluationParams(List.of(EvalTestCaseParam.INPUT, EvalTestCaseParam.ACTUAL_OUTPUT))
    .judge(judge)
    .threshold(0.8)
    .build();

From a ChatModel​

If you have a ChatModel, pass it directly. Dokimos wraps it in a ChatClient for you.

import dev.dokimos.core.*;
import dev.dokimos.core.evaluators.*;
import dev.dokimos.springai.SpringAiSupport;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.openai.OpenAiChatModel;
import org.springframework.ai.openai.OpenAiChatOptions;
import org.springframework.ai.openai.api.OpenAiApi;

OpenAiApi openAiApi = OpenAiApi.builder()
    .apiKey(System.getenv("OPENAI_API_KEY"))
    .build();

ChatModel chatModel = OpenAiChatModel.builder()
    .openAiApi(openAiApi)
    .defaultOptions(OpenAiChatOptions.builder().model("gpt-5.2").build())
    .build();

// Convert to JudgeLM
JudgeLM judge = SpringAiSupport.asJudge(chatModel);

// Use in evaluators
Evaluator faithfulness = FaithfulnessEvaluator.builder()
    .threshold(0.7)
    .judge(judge)
    .build();

Step 3: Convert test cases​

Dokimos evaluators read an EvalTestCase. Spring AI evaluators read an EvaluationRequest. These two helpers move data between them:

  • SpringAiSupport.toTestCase(request) builds an EvalTestCase from an EvaluationRequest.
  • SpringAiSupport.toEvaluationResponse(result) builds an EvaluationResponse from an EvalResult.
import dev.dokimos.core.*;
import dev.dokimos.springai.SpringAiSupport;
import org.springframework.ai.evaluation.EvaluationRequest;
import org.springframework.ai.evaluation.EvaluationResponse;
import org.springframework.ai.document.Document;

// Create Spring AI EvaluationRequest
List<Document> retrievedDocs = List.of(
    new Document("30-day money-back guarantee"),
    new Document("Contact support for refunds")
);

EvaluationRequest request = new EvaluationRequest(
    "What is the refund policy?",           // user text
    retrievedDocs,                           // retrieved documents
    "We offer a 30-day refund policy."      // response content
);

// Convert to Dokimos EvalTestCase
EvalTestCase testCase = SpringAiSupport.toTestCase(request);

// Run evaluation
EvalResult result = faithfulnessEvaluator.evaluate(testCase);

// Convert back to Spring AI EvaluationResponse
EvaluationResponse response = SpringAiSupport.toEvaluationResponse(result);

// Check results
System.out.println("Passed: " + response.isPass());
System.out.println("Score: " + response.getMetadata().get("score"));
System.out.println("Feedback: " + response.getFeedback());

Full example: run an experiment​

This puts the pieces together. It sets up a ChatModel, builds a dataset, runs the model as the task, scores answers with a Spring AI judge, and prints the pass rate.

import dev.dokimos.core.*;
import dev.dokimos.core.evaluators.*;
import dev.dokimos.springai.SpringAiSupport;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.openai.OpenAiChatModel;
import org.springframework.ai.openai.OpenAiChatOptions;
import org.springframework.ai.openai.api.OpenAiApi;

public class SpringAiEvaluation {

    public static void main(String[] args) {
        // 1. Set up ChatModel
        OpenAiApi openAiApi = OpenAiApi.builder()
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .build();

        ChatModel chatModel = OpenAiChatModel.builder()
            .openAiApi(openAiApi)
            .defaultOptions(OpenAiChatOptions.builder().model("gpt-5.2").build())
            .build();

        // 2. Create a dataset
        Dataset dataset = Dataset.builder()
            .name("customer-qa")
            .addExample(Example.of(
                "What is your return policy?",
                "30-day money-back guarantee"
            ))
            .addExample(Example.of(
                "How can I contact support?",
                "Email support@example.com"
            ))
            .build();

        // 3. Create Task
        Task task = example -> {
            String response = chatModel.call(example.input());
            return Map.of("output", response);
        };

        // 4. Set up evaluators with Spring AI judge
        ChatModel judgeModel = OpenAiChatModel.builder()
            .openAiApi(openAiApi)
            .defaultOptions(OpenAiChatOptions.builder().model("gpt-5.2").build())
            .build();

        JudgeLM judge = SpringAiSupport.asJudge(judgeModel);

        List<Evaluator> evaluators = List.of(
            LLMJudgeEvaluator.builder()
                .name("Answer Quality")
                .criteria("Is the answer helpful and accurate?")
                .evaluationParams(List.of(EvalTestCaseParam.INPUT, EvalTestCaseParam.ACTUAL_OUTPUT))
                .judge(judge)
                .threshold(0.8)
                .build(),
            ExactMatchEvaluator.builder().build()
        );

        // 5. Run experiment
        ExperimentResult result = Experiment.builder()
            .name("Spring AI Evaluation")
            .dataset(dataset)
            .task(task)
            .evaluators(evaluators)
            .build()
            .run();

        // 6. Display results
        System.out.println("Pass rate: " +
            String.format("%.0f%%", result.passRate() * 100));
        System.out.println("Answer Quality: " +
            String.format("%.2f", result.averageScore("Answer Quality")));
    }
}
tip

See Datasets for loading data from JSON or CSV, and Evaluators for the full list of evaluators.

Run many calls at once (async)​

A plain Task blocks one thread per example. When each example is an independent ChatClient call, asyncTask keeps many calls in flight instead. Wire it with Experiment.builder().asyncTask(...) and cap how many run at once with parallelism(...).

SpringAiSupport.asyncTask(client) reads the example input as the user message and writes the response under the default "output" key. It runs the blocking ChatClient call on the common ForkJoinPool through CompletableFuture.supplyAsync(...).

import dev.dokimos.core.*;
import dev.dokimos.springai.SpringAiSupport;
import org.springframework.ai.chat.client.ChatClient;

ChatClient client = ChatClient.builder(chatModel).build();
AsyncTask task = SpringAiSupport.asyncTask(client);

ExperimentResult result = Experiment.builder()
    .name("Spring AI Async")
    .dataset(dataset)
    .asyncTask(task)
    .parallelism(8)
    .evaluators(evaluators)
    .build()
    .run();

To read and write different keys, call asyncTask(client, inputKey, outputKey).

note

The common pool is shared across the whole process, and its effective parallelism is about one less than the CPU count. So it caps how many blocking calls actually run at once, even when parallelism is higher. For controlled, isolated concurrency, pass an Executor sized to your target throughput. Use asyncTask(client, executor) or the four-arg asyncTask(client, inputKey, outputKey, executor).

import java.util.concurrent.Executor;
import java.util.concurrent.Executors;

// A pool sized to match your desired concurrency
Executor executor = Executors.newFixedThreadPool(16);

AsyncTask task = SpringAiSupport.asyncTask(client, executor);

Experiment.builder()
    .dataset(dataset)
    .asyncTask(task)
    .parallelism(16)
    .evaluators(evaluators)
    .build()
    .run();

Reactive tasks​

If your pipeline already returns a Mono, bridge it directly instead of blocking on a pool. reactiveStringTask wraps a Mono<String> response under the default "output" key. reactiveTask adapts a Mono<TaskResult> when you want full control over the output map. Each Mono becomes a CompletableFuture through Mono.toFuture().

import dev.dokimos.core.*;
import dev.dokimos.springai.SpringAiSupport;

// Mono<String> -> output
AsyncTask stringTask = SpringAiSupport.reactiveStringTask(example ->
    reactiveChatClient.prompt()
        .user(example.input())
        .stream()
        .content()
        .collectList()
        .map(parts -> String.join("", parts)));

// Mono<TaskResult> -> full control over the output map
AsyncTask resultTask = SpringAiSupport.reactiveTask(example ->
    reactiveChatClient.prompt()
        .user(example.input())
        .stream()
        .content()
        .collectList()
        .map(parts -> TaskResult.of(Map.of("output", String.join("", parts)))));

Evaluate tool-calling agents​

When your Spring AI agent calls tools, toAgentTrace turns an AssistantMessage (and its ToolResponseMessages) into an AgentTrace. You feed that straight into the agent evaluators. Tool calls match their results by tool-call id. toToolDefinitions converts the Spring AI tool definitions the agent was given, so calls can be checked against them.

AgentTrace.toTestCase(userMessage, tools) builds the EvalTestCase the agent evaluators expect.

import dev.dokimos.core.*;
import dev.dokimos.core.agents.AgentTrace;
import dev.dokimos.core.agents.ToolDefinition;
import dev.dokimos.core.evaluators.agents.ToolCorrectnessEvaluator;
import dev.dokimos.springai.SpringAiSupport;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.messages.ToolResponseMessage;

// From your agent run: the assistant message and the tool responses produced for it
AssistantMessage assistantMessage = /* ... */;
List<ToolResponseMessage> toolResponses = /* ... */;

// Convert the tools the agent was given
List<ToolDefinition> tools = SpringAiSupport.toToolDefinitions(springAiToolDefinitions);

// Build a trace (tool calls matched to results by id) and a test case
AgentTrace trace = SpringAiSupport.toAgentTrace(assistantMessage, toolResponses);
EvalTestCase testCase = trace.toTestCase("What's the weather in Paris?", tools);

// Evaluate with an agent evaluator
EvalResult result = ToolCorrectnessEvaluator.builder().build().evaluate(testCase);
note

toAgentTrace(message) (without tool responses) builds a trace from the tool calls alone. Use it when you only need to check which tools the agent chose, not their results.

Bridge Spring AI evaluators​

If you already use Spring AI's built-in evaluators and want their scores tracked in Dokimos, convert the request and wrap the evaluator:

import dev.dokimos.core.*;
import dev.dokimos.core.evaluators.*;
import dev.dokimos.springai.SpringAiSupport;
import org.springframework.ai.evaluation.RelevancyEvaluator;

// Spring AI evaluator
RelevancyEvaluator springAiEvaluator = new RelevancyEvaluator(
    ChatClient.builder(chatModel)
);

// Create Spring AI EvaluationRequest
EvaluationRequest request = new EvaluationRequest(
    userQuestion,
    retrievedDocuments,
    generatedResponse
);

// Evaluate with Spring AI
EvaluationResponse springAiResponse = springAiEvaluator.evaluate(request);

// Convert to Dokimos for tracking in experiments
EvalTestCase testCase = SpringAiSupport.toTestCase(request);

// You can also create a custom Dokimos evaluator that wraps Spring AI evaluators
Evaluator dokimosEvaluator = new BaseEvaluator("relevancy", 1.0, List.of()) {
    @Override
    protected EvalResult runEvaluation(EvalTestCase testCase) {
        // Convert Dokimos -> Spring AI -> evaluate -> convert back
        EvaluationRequest req = /* build from testCase */;
        EvaluationResponse resp = springAiEvaluator.evaluate(req);

        return EvalResult.builder()
            .name(name())
            .score(resp.getMetadata().get("score"))
            .success(resp.isPass())
            .reason(resp.getFeedback())
            .build();
    }
};

Evaluate a RAG pipeline​

For a RAG system, your task retrieves documents and generates a response, then returns both under "output" and "context". FaithfulnessEvaluator reads the context to check the answer stays grounded.

import dev.dokimos.core.*;
import dev.dokimos.core.evaluators.FaithfulnessEvaluator;
import dev.dokimos.springai.SpringAiSupport;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.document.Document;
import org.springframework.ai.vectorstore.VectorStore;

// Your RAG setup
VectorStore vectorStore = /* your vector store */;
ChatClient chatClient = ChatClient.builder(chatModel)
    .defaultAdvisors(
        new QuestionAnswerAdvisor(vectorStore, SearchRequest.defaults())
    )
    .build();

// Create evaluation task
Task ragTask = example -> {
    String query = example.input();

    // Retrieve documents
    List<Document> retrieved = vectorStore.similaritySearch(
        SearchRequest.query(query).withTopK(3)
    );

    // Generate response
    String response = chatClient.prompt()
        .user(query)
        .call()
        .content();

    // Extract the context texts
    List<String> context = retrieved.stream()
        .map(Document::getText)
        .toList();

    return Map.of(
        "output", response,
        "context", context
    );
};

// Evaluate faithfulness
JudgeLM judge = SpringAiSupport.asJudge(chatModel);

Evaluator faithfulness = FaithfulnessEvaluator.builder()
    .threshold(0.8)
    .judge(judge)
    .build();

ExperimentResult result = Experiment.builder()
    .dataset(dataset)
    .task(ragTask)
    .evaluators(List.of(faithfulness))
    .build()
    .run();

Structured / typed output​

When your Spring AI call returns structured data (for example a record mapped from the model's JSON output), return that object under "output" instead of a string. Compare it with StructuralMatchEvaluator (numbers compare by value, formatting and key order do not count), and read it back type-safely with actualOutputAs(Record.class).

record Invoice(String id, double total, List<String> items) {}

Task task = Task.typed(example -> chatClient.prompt()
    .user(example.input())
    .call()
    .entity(Invoice.class));   // Spring AI maps the response to a record

Evaluator structural = StructuralMatchEvaluator.builder()
    .name("Invoice Match")
    .threshold(1.0)
    .build();

// In a custom evaluator, read the structured value back
Invoice actual = testCase.actualOutputAs(Invoice.class);

See the Structured & Typed Data hub for the full pipeline.

Field mappings​

EvaluationRequest -> EvalTestCase​

When converting from Spring AI to Dokimos:

Spring AIDokimos
getUserText()inputs["input"]
getResponseContent()actualOutputs["output"]
getDataList()actualOutputs["context"] (as List<String>)

EvalResult -> EvaluationResponse​

When converting from Dokimos back to Spring AI:

DokimosSpring AI
success()isPass()
score()metadata["score"]
reason()getFeedback()
metadata()getMetadata() (merged with score)

Best practices​

Combine with Spring Boot: in a Spring Boot application, inject your ChatModel beans and use them directly for evaluation:

@Component
public class AiEvaluationService {

    private final ChatModel chatModel;

    public AiEvaluationService(ChatModel chatModel) {
        this.chatModel = chatModel;
    }

    public ExperimentResult evaluate(Dataset dataset, Task task) {
        JudgeLM judge = SpringAiSupport.asJudge(chatModel);

        return Experiment.builder()
            .dataset(dataset)
            .task(task)
            .evaluators(List.of(
                FaithfulnessEvaluator.builder()
                    .judge(judge)
                    .build()
            ))
            .build()
            .run();
    }
}

JUnit integration​

Combine with JUnit to fail a build when an answer misses the mark. The @DatasetSource annotation feeds one Example per row into the test:

import dev.dokimos.junit.DatasetSource;
import org.junit.jupiter.params.ParameterizedTest;

@ParameterizedTest
@DatasetSource("classpath:datasets/qa-dataset-v1.json")
void chatResponseShouldBeAccurate(Example example) {
    // Generate response with Spring AI
    String response = chatClient.prompt()
        .user(example.input())
        .call()
        .content();

    // Create test case
    EvalTestCase testCase = EvalTestCase.of(
        example.input(),
        response,
        example.expectedOutput()
    );

    // Assert with evaluator
    Assertions.assertEval(testCase, exactMatchEvaluator);
}

Assert on the average score​

The parameterized test above fails if any single example fails. Often you want a different gate: assert that the average score across all examples clears a threshold. This fits when:

  • Individual examples may dip below the threshold, but overall quality should stay high.
  • You want different thresholds for different evaluators.
  • You run quality gates in CI/CD pipelines.
import dev.dokimos.core.*;
import dev.dokimos.core.evaluators.*;
import dev.dokimos.springai.SpringAiSupport;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

@Test
void experimentMeetsQualityThresholds() {
    Dataset dataset = DatasetResolverRegistry.getInstance()
        .resolve("classpath:datasets/qa-dataset.json");

    JudgeLM judge = SpringAiSupport.asJudge(chatModel);

    List<Evaluator> evaluators = List.of(
        FaithfulnessEvaluator.builder()
            .judge(judge)
            .contextKey("context")
            .build(),
        ContextualRelevanceEvaluator.builder()
            .judge(judge)
            .retrievalContextKey("context")
            .build(),
        LLMJudgeEvaluator.builder()
            .name("Answer Quality")
            .criteria("Is the answer helpful, clear, and accurate?")
            .evaluationParams(List.of(EvalTestCaseParam.INPUT, EvalTestCaseParam.ACTUAL_OUTPUT))
            .judge(judge)
            .build()
    );

    ExperimentResult result = Experiment.builder()
        .name("Agent Evaluation")
        .dataset(dataset)
        .task(task)
        .evaluators(evaluators)
        .build()
        .run();

    // Assert each evaluator's average meets 0.8
    assertAll(
        () -> assertTrue(result.averageScore("Faithfulness") >= 0.8,
            "Faithfulness: " + result.averageScore("Faithfulness")),
        () -> assertTrue(result.averageScore("ContextualRelevance") >= 0.8,
            "ContextualRelevance: " + result.averageScore("ContextualRelevance")),
        () -> assertTrue(result.averageScore("Answer Quality") >= 0.8,
            "Answer Quality: " + result.averageScore("Answer Quality"))
    );
}
tip

Use assertAll to run every assertion and report all failures at once, instead of stopping at the first. That way you see every threshold that missed in one run.

Use with Spring AI testing​

You can run Dokimos evaluators next to Spring AI's own testing utilities to build full test suites for your AI applications.

For AI agentsView as Markdown