Package dev.dokimos.openai

Class OpenAiSupport

java.lang.Object
dev.dokimos.openai.OpenAiSupport
public final class OpenAiSupport extends Object
Utilities for integrating with the OpenAI Java SDK.

This class provides factory methods to create JudgeLMs and to capture an OpenAI agent's tool calls as Dokimos AgentTrace, ToolCall, and ToolDefinition objects the agent evaluators understand.

Agent Evaluation


 ChatCompletion completion = client.chat().completions().create(params);
 ChatCompletionMessage message = completion.choices().get(0).message();

 AgentTrace trace = OpenAiSupport.toAgentTrace(message, id -> myApp.resultFor(id));
 EvalTestCase testCase = trace.toTestCase(userMessage, OpenAiSupport.toToolDefinitions(tools));

  • Method Details

    • asJudge

      public static JudgeLM asJudge(com.openai.client.OpenAIClient client, com.openai.models.ChatModel model)
      Creates a JudgeLM from an OpenAI OpenAIClient and a specific ChatModel.

      Use this to create judges for LLM-based evaluators like LLMJudgeEvaluator, TaskCompletionEvaluator, etc.

      Parameters:
      client - the OpenAI client to use as judge
      model - the chat model the judge uses
      Returns:
      a JudgeLM that delegates to the client

    • toAgentTrace

      public static AgentTrace toAgentTrace(com.openai.models.chat.completions.ChatCompletionMessage message)
      Builds an AgentTrace from an OpenAI ChatCompletionMessage.

      The message's content() becomes the final response and its function toolCalls() become ToolCalls carrying the tool name and parsed arguments.

      Parameters:
      message - the assistant message (may be null)
      Returns:
      an agent trace, never null

    • toAgentTrace

      public static AgentTrace toAgentTrace(com.openai.models.chat.completions.ChatCompletionMessage message, Function<String,String> resultLookup)
      Builds an AgentTrace from an OpenAI ChatCompletionMessage, looking up each tool call's result by its id.
      Parameters:
      message - the assistant message (may be null)
      resultLookup - maps a tool call id to its execution result (may return null)
      Returns:
      an agent trace, never null

    • toToolCalls

      public static List<ToolCall> toToolCalls(com.openai.models.chat.completions.ChatCompletionMessage message)
      Extracts function ToolCalls from an OpenAI ChatCompletionMessage in order.
      Parameters:
      message - the assistant message (may be null)
      Returns:
      the tool calls, or an empty list when there are none

    • toToolCalls

      public static List<ToolCall> toToolCalls(com.openai.models.chat.completions.ChatCompletionMessage message, Function<String,String> resultLookup)
      Extracts function ToolCalls from an OpenAI ChatCompletionMessage, looking up each tool call's result by its id.
      Parameters:
      message - the assistant message (may be null)
      resultLookup - maps a tool call id to its execution result (may return null)
      Returns:
      the tool calls, or an empty list when there are none

    • toToolCall

      public static ToolCall toToolCall(com.openai.models.chat.completions.ChatCompletionMessageToolCall toolCall, String result)
      Converts a single OpenAI function ChatCompletionMessageToolCall to a ToolCall.
      Parameters:
      toolCall - the OpenAI tool call (must be a function tool call)
      result - the result of executing the tool, or null if not executed
      Returns:
      the tool call

    • toToolDefinitions

      public static List<ToolDefinition> toToolDefinitions(List<com.openai.models.chat.completions.ChatCompletionTool> tools)
      Converts OpenAI ChatCompletionTools to ToolDefinitions so tool calls can be evaluated against the tools the agent was given.
      Parameters:
      tools - the OpenAI tools (may be null)
      Returns:
      the tool definitions, or an empty list

    • toToolDefinition

      public static ToolDefinition toToolDefinition(com.openai.models.chat.completions.ChatCompletionTool tool)
      Converts a single OpenAI function ChatCompletionTool to a ToolDefinition.
      Parameters:
      tool - the OpenAI tool (must be a function tool)
      Returns:
      the tool definition