Package dev.dokimos.springai

Class SpringAiSupport

java.lang.Object
dev.dokimos.springai.SpringAiSupport
public final class SpringAiSupport extends Object
Utilities for integrating with Spring AI.

This class provides bridge methods to use Spring AI components with the Dokimos evaluation framework.

Using Spring AI ChatClient as a Judge


 ChatClient.Builder clientBuilder = ChatClient.builder(chatModel);
 JudgeLM judge = SpringAiSupport.asJudge(clientBuilder);

 var evaluator = FaithfulnessEvaluator.builder()
         .judge(judge)
         .build();

Converting Spring AI Evaluation Objects


 // Convert Spring AI EvaluationRequest to Dokimos EvalTestCase
 EvaluationRequest request = ...;
 EvalTestCase testCase = SpringAiSupport.toTestCase(request);

 // Run Dokimos evaluation
 EvalResult result = evaluator.evaluate(testCase);

 // Convert back to Spring AI EvaluationResponse
 EvaluationResponse response = SpringAiSupport.toEvaluationResponse(result);

  • Field Details

    • OUTPUT_KEY

      public static final String OUTPUT_KEY
      Default key for the model output in evaluation results.
      See Also:

    • CONTEXT_KEY

      public static final String CONTEXT_KEY
      Default key for additional context in evaluation results.
      See Also:

    • INPUT_KEY

      public static final String INPUT_KEY
      Default key for reading input from dataset examples.
      See Also:

  • Method Details

    • asJudge

      public static JudgeLM asJudge(org.springframework.ai.chat.client.ChatClient.Builder builder)
      Creates a JudgeLM from a Spring AI ChatClient.Builder.

      Use this to create judges for LLM-based evaluators like LLMJudgeEvaluator, FaithfulnessEvaluator, etc.

      Example: 

      
 ChatClient.Builder clientBuilder = ChatClient.builder(chatModel);
 JudgeLM judge = SpringAiSupport.asJudge(clientBuilder);

 var evaluator = LLMJudgeEvaluator.builder()
         .judge(judge)
         .criteria("Is the response helpful?")
         .build();
      Parameters:
      builder - the ChatClient.Builder to use as judge
      Returns:
      a JudgeLM that delegates to the ChatClient

    • asJudge

      public static JudgeLM asJudge(org.springframework.ai.chat.model.ChatModel model)
      Creates a JudgeLM from a Spring AI ChatModel.

      This is a convenience overload that accepts a ChatModel directly instead of a ChatClient.Builder.

      Example: 

      
 ChatModel chatModel = OpenAiChatModel.builder()...build();
 JudgeLM judge = SpringAiSupport.asJudge(chatModel);

 var evaluator = FaithfulnessEvaluator.builder()
     .judge(judge)
     .build();
      Parameters:
      model - the ChatModel to use as judge
      Returns:
      a JudgeLM that delegates to the ChatModel

    • toTestCase

      public static EvalTestCase toTestCase(org.springframework.ai.evaluation.EvaluationRequest request)
      Converts a Spring AI EvaluationRequest to a Dokimos EvalTestCase.

      Maps the following fields:

      • getUserText() → input
      • getResponseContent() → actual output
      • getDataList() → context (list of document contents)

      Example: 

      
 EvaluationRequest request = new EvaluationRequest(
         userText,
         retrievedDocuments,
         responseContent);

 EvalTestCase testCase = SpringAiSupport.toTestCase(request);
 EvalResult result = faithfulnessEvaluator.evaluate(testCase);
      Parameters:
      request - the Spring AI evaluation request
      Returns:
      an EvalTestCase containing the request data

    • toEvaluationResponse

      public static org.springframework.ai.evaluation.EvaluationResponse toEvaluationResponse(EvalResult result)
      Converts a Dokimos EvalResult to a Spring AI EvaluationResponse.

      Maps the following fields:

      • score -> metadata["score"] (as float)
      • success -> pass/fail status
      • reason -> the reasoning text
      • metadata -> preserved in response metadata

      Example: 

      
 EvalResult result = evaluator.evaluate(testCase);
 EvaluationResponse response = SpringAiSupport.toEvaluationResponse(result);

 System.out.println("Score: " + response.getMetadata().get("score"));
 System.out.println("Passed: " + response.isPass());
 System.out.println("Feedback: " + response.getFeedback());
      Parameters:
      result - the Dokimos evaluation result
      Returns:
      an EvaluationResponse containing the result data

    • asyncTask

      public static AsyncTask asyncTask(org.springframework.ai.chat.client.ChatClient client)
      Creates an AsyncTask that calls a Spring AI ChatClient off the calling thread and writes the response under the default output key.

      The example's input is sent as the user message. The ChatClient.prompt() call is dispatched on the common ForkJoinPool via CompletableFuture.supplyAsync(java.util.function.Supplier) so the experiment's async execution path can keep many calls in flight without a blocked thread per example.

      Note: because the call blocks on the common pool, the experiment's parallelism bounds how many invocations are launched, but the effective concurrency of the blocking HTTP call is also limited by the common pool (~one less than the CPU count), which is shared process-wide. For higher, isolated concurrency use asyncTask(ChatClient, java.util.concurrent.Executor) (or the four-arg overload) to run calls on a pool you control.

      Example: 

      
 ChatClient client = ChatClient.builder(chatModel).build();
 AsyncTask task = SpringAiSupport.asyncTask(client);

 Experiment.builder()
     .asyncTask(task)
     .parallelism(8)
     .evaluators(List.of(evaluator))
     .build()
     .run();
      Parameters:
      client - the ChatClient to call, never null
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if client is null

    • asyncTask

      public static AsyncTask asyncTask(org.springframework.ai.chat.client.ChatClient client, String inputKey, String outputKey)
      Creates an AsyncTask that calls a Spring AI ChatClient off the calling thread using caller-chosen input and output keys.

      Behaves like asyncTask(ChatClient) but reads the user message from inputKey and writes the response under outputKey, for datasets or evaluators that use different key names.

      Parameters:
      client - the ChatClient to call, never null
      inputKey - the key to read the user message from the example inputs, never null
      outputKey - the key the response is written under in the result, never null
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if any argument is null

    • asyncTask

      public static AsyncTask asyncTask(org.springframework.ai.chat.client.ChatClient client, Executor executor)
      Creates an AsyncTask that calls a Spring AI ChatClient on the supplied Executor, with default input and output keys.

      Use this when you want the blocking call to run on a pool you control (sized to your desired concurrency) rather than the shared common ForkJoinPool. Pair the executor's size with the experiment's parallelism for predictable throughput.

      Parameters:
      client - the ChatClient to call, never null
      executor - the executor each blocking call runs on, never null
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if client or executor is null

    • asyncTask

      public static AsyncTask asyncTask(org.springframework.ai.chat.client.ChatClient client, String inputKey, String outputKey, Executor executor)
      Creates an AsyncTask that calls a Spring AI ChatClient using caller-chosen input and output keys, dispatching each blocking call on the supplied Executor (or the common ForkJoinPool when executor is null).

      Supplying an executor lets you control and isolate concurrency: the experiment's parallelism bounds in-flight invocations, and a pool sized to match gives true parallel blocking calls instead of the common pool's ~CPU-count, process-wide ceiling.

      Parameters:
      client - the ChatClient to call, never null
      inputKey - the key to read the user message from the example inputs, never null
      outputKey - the key the response is written under in the result, never null
      executor - the executor each blocking call runs on, or null for the common pool
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if client, inputKey, or outputKey is null

    • reactiveTask

      public static AsyncTask reactiveTask(Function<Example,reactor.core.publisher.Mono<TaskResult>> taskFunction)
      Adapts a Reactor Mono of TaskResult to an AsyncTask.

      For reactive Spring AI pipelines: supply a function that produces a Mono<TaskResult> for an example, and the resulting task converts each Mono to a CompletableFuture via Mono.toFuture().

      Example: 

      
 AsyncTask task = SpringAiSupport.reactiveTask(example ->
     reactiveChatClient.prompt()
         .user(example.input())
         .stream()
         .content()
         .collectList()
         .map(parts -> TaskResult.of(Map.of("output", String.join("", parts)))));
      Parameters:
      taskFunction - a function producing a Mono<TaskResult> for an example, never null
      Returns:
      an AsyncTask backed by the supplied Mono
      Throws:
      IllegalArgumentException - if taskFunction is null

    • reactiveStringTask

      public static AsyncTask reactiveStringTask(Function<Example,reactor.core.publisher.Mono<String>> taskFunction)
      Adapts a Reactor Mono of String output to an AsyncTask, wrapping the emitted string under the default output key.

      Convenience over reactiveTask(Function) for the common case where the reactive pipeline yields the model's textual response directly. A null emission is stored as an empty string.

      Example: 

      
 AsyncTask task = SpringAiSupport.reactiveStringTask(example ->
     reactiveChatClient.prompt().user(example.input()).stream().content().last());
      Parameters:
      taskFunction - a function producing a Mono<String> response for an example, never null
      Returns:
      an AsyncTask that writes the emitted string under the default output key
      Throws:
      IllegalArgumentException - if taskFunction is null

    • measuredAsyncTask

      public static AsyncTask measuredAsyncTask(org.springframework.ai.chat.client.ChatClient client, String modelId, PriceTable prices)
      Creates a measured AsyncTask that calls a Spring AI ChatClient and captures token usage, latency, and (when a PriceTable is supplied) cost, lighting up the run's metrics cards.

      Counterpart to asyncTask(ChatClient): instead of reading only .call().content(), it reads .call().chatResponse() so the Usage (prompt/completion tokens) is available, times the call, and composes cost via prices. The blocking call runs on the common ForkJoinPool; for isolated, true concurrency use measuredAsyncTask(ChatClient, String, PriceTable, Executor) with a pool you size to the experiment's parallelism (the common pool's ceiling is ~CPU-count, process-wide).

      Missing usage leaves the token fields null; a null prices (or a null lookup result) leaves cost null so only the Tokens and Latency cards light. Never throws on missing metrics.

      Parameters:
      client - the ChatClient to call, never null
      modelId - the model id used as the PriceTable lookup key, or null to skip pricing
      prices - the price lookup, or null to capture tokens and latency only
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if client is null

    • measuredAsyncTask

      public static AsyncTask measuredAsyncTask(org.springframework.ai.chat.client.ChatClient client, String modelId, PriceTable prices, Executor executor)
      Creates a measured AsyncTask that dispatches each blocking call on the supplied Executor so you control and isolate concurrency.
      Parameters:
      client - the ChatClient to call, never null
      modelId - the model id used as the PriceTable lookup key, or null to skip pricing
      prices - the price lookup, or null to capture tokens and latency only
      executor - the executor each blocking call runs on, never null
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if client or executor is null

    • measuredAsyncTask

      public static AsyncTask measuredAsyncTask(org.springframework.ai.chat.client.ChatClient client, String inputKey, String outputKey, String modelId, PriceTable prices, Executor executor)
      Creates a measured AsyncTask with caller-chosen input and output keys, dispatching each blocking call on the supplied Executor (or the common ForkJoinPool when executor is null).
      Parameters:
      client - the ChatClient to call, never null
      inputKey - the key to read the user message from the example inputs, never null
      outputKey - the key the response is written under in the result, never null
      modelId - the model id used as the PriceTable lookup key, or null to skip pricing
      prices - the price lookup, or null to capture tokens and latency only
      executor - the executor each blocking call runs on, or null for the common pool
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if client, inputKey, or outputKey is null

    • toAgentTrace

      public static AgentTrace toAgentTrace(org.springframework.ai.chat.messages.AssistantMessage message)
      Builds an AgentTrace from a Spring AI AssistantMessage.

      The message's text becomes the final response and its getToolCalls() become ToolCalls with parsed arguments. Tool results are not part of an AssistantMessage; use toAgentTrace(AssistantMessage, List) to attach them.

      Parameters:
      message - the assistant message (may be null)
      Returns:
      an agent trace, never null

    • toAgentTrace

      public static AgentTrace toAgentTrace(org.springframework.ai.chat.messages.AssistantMessage message, List<org.springframework.ai.chat.messages.ToolResponseMessage> toolResponses)
      Builds an AgentTrace from a Spring AI AssistantMessage and the tool responses produced for it.

      Each tool call is matched to its result by tool-call id from the supplied ToolResponseMessages, so the resulting trace carries both the agent's tool calls and what those tools returned. 

      
 AgentTrace trace = SpringAiSupport.toAgentTrace(assistantMessage, toolResponseMessages);
 EvalTestCase testCase = trace.toTestCase(userMessage, tools);
      Parameters:
      message - the assistant message (may be null)
      toolResponses - the tool response messages whose responses carry results (may be null)
      Returns:
      an agent trace, never null

    • toToolCalls

      public static List<ToolCall> toToolCalls(org.springframework.ai.chat.messages.AssistantMessage message)
      Extracts ToolCalls from a Spring AI AssistantMessage without results.
      Parameters:
      message - the assistant message (may be null)
      Returns:
      the tool calls in order, or an empty list

    • toToolCalls

      public static List<ToolCall> toToolCalls(org.springframework.ai.chat.messages.AssistantMessage message, List<org.springframework.ai.chat.messages.ToolResponseMessage> toolResponses)
      Extracts ToolCalls from a Spring AI AssistantMessage, attaching results from the supplied tool responses by tool-call id.
      Parameters:
      message - the assistant message (may be null)
      toolResponses - the tool response messages (may be null)
      Returns:
      the tool calls in order, or an empty list

    • toToolDefinitions

      public static List<ToolDefinition> toToolDefinitions(List<org.springframework.ai.tool.definition.ToolDefinition> toolDefinitions)
      Converts Spring AI ToolDefinitions to Dokimos ToolDefinitions so tool calls can be evaluated against the tools the agent was given.
      Parameters:
      toolDefinitions - the Spring AI tool definitions (may be null)
      Returns:
      the Dokimos tool definitions, or an empty list