Package dev.dokimos.langchain4j

Class LangChain4jSupport

java.lang.Object
dev.dokimos.langchain4j.LangChain4jSupport
public final class LangChain4jSupport extends Object
Utilities for integrating with LangChain4j.

This class provides factory methods to create Tasks and JudgeLMs from LangChain4j components.

RAG Evaluation


 // 1. Define your AiService to return Result<String>
 interface Assistant {
     Result<String> chat(String userMessage);
 }

 // 2. Build your assistant
 Assistant assistant = AiServices.builder(Assistant.class)
     .chatModel(chatModel)
     .retrievalAugmentor(DefaultRetrievalAugmentor.builder()
         .queryTransformer(compressingQueryTransformer)
         .contentRetriever(retriever)
         .contentAggregator(reRankingAggregator)
         .build())
     .build();

 // 3. Create a Task for evaluation
 Task task = LangChain4jSupport.ragTask(assistant::chat);

 // 4. Run evaluation with some metrics
 Experiment.builder()
     .task(task)
     .evaluators(List.of(faithfulness, contextRelevancy))
     .build()
     .run();

  • Field Details

    • OUTPUT_KEY

      public static final String OUTPUT_KEY
      Default key for the model output in evaluation results.
      See Also:

    • CONTEXT_KEY

      public static final String CONTEXT_KEY
      Default key for additional context in evaluation results.
      See Also:

    • INPUT_KEY

      public static final String INPUT_KEY
      Default key for reading input from dataset examples.
      See Also:

  • Method Details

    • asJudge

      public static JudgeLM asJudge(dev.langchain4j.model.chat.ChatModel model)
      Creates a JudgeLM from a LangChain4j ChatModel.

      Use this to create judges for LLM-based evaluators like LLMJudgeEvaluator, FaithfulnessEvaluator, etc.

      Example: 

      
 ChatModel gemini = VertexAiGeminiChatModel.builder()...build();
 JudgeLM judge = LangChain4jSupport.asJudge(gemini);

 var evaluator = LLMJudgeEvaluator.builder()
     .judge(judge)
     .criteria("Is the response helpful?")
     .build();
      Parameters:
      model - the ChatModel to use as judge
      Returns:
      a JudgeLM that delegates to the ChatModel

    • simpleTask

      public static Task simpleTask(dev.langchain4j.model.chat.ChatModel model)
      Creates a simple Task for Q&A evaluation.

      The task reads "input" from the example and returns a Map with "output".

      Example: 

      
 ChatModel model = OpenAiChatModel.builder()...build();
 Task task = LangChain4jSupport.simpleTask(model);

 // Dataset examples just need "input"
 Example example = Example.of("What is 2+2?", "4");
      Parameters:
      model - the ChatModel to evaluate
      Returns:
      a Task suitable for the Experiment

    • simpleTask

      public static Task simpleTask(dev.langchain4j.model.chat.ChatModel model, String outputKey)
      Creates a simple Task for Q&A evaluation that writes the response under a caller-chosen key.

      Behaves like simpleTask(ChatModel) but lets you override the default output key when your evaluators or dataset expect a different name.

      Example: 

      
 ChatModel model = OpenAiChatModel.builder()...build();
 Task task = LangChain4jSupport.simpleTask(model, "answer");
      Parameters:
      model - the ChatModel to evaluate
      outputKey - the key for the output in the result map
      Returns:
      a Task suitable for the Experiment

    • ragTask

      public static Task ragTask(Function<String,dev.langchain4j.service.Result<String>> assistantCall)
      Creates a RAG evaluation Task from a function that returns Result.

      This is the primary integration point for RAG evaluation. LangChain4j's Result class already contains the retrieved sources via result.sources().

      Example: 

      
 interface Assistant {
     Result<String> chat(String userMessage);
 }

 Assistant assistant = AiServices.builder(Assistant.class)
     .chatModel(chatModel)
     .retrievalAugmentor(retrievalAugmentor)
     .build();

 Task task = LangChain4jSupport.ragTask(assistant::chat);
      Parameters:
      assistantCall - a function that takes the input string and returns a Result
      Returns:
      a Task suitable for evaluation

    • ragTask

      public static Task ragTask(Function<String,dev.langchain4j.service.Result<String>> assistantCall, String inputKey, String outputKey, String contextKey)
      Creates a RAG evaluation Task with custom key names.

      Use this when your dataset or evaluators expect different keys.

      Example: 

      
 // Dataset uses "question" instead of "input"
 Task task = LangChain4jSupport.ragTask(
     assistant::chat,
     "question",        // input key
     "answer",          // output key
     "retrievalContext" // context key
 );
      Parameters:
      assistantCall - a function that takes the input string and returns a Result
      inputKey - the key to read from example inputs
      outputKey - the key for the output in the result map
      contextKey - the key for the retrieval context in the result map
      Returns:
      a Task suitable for RAG evaluation

    • customTask

      public static Task customTask(Task taskFunction)
      Creates a flexible Task that allows full control over output mapping.

      Use this for complex scenarios where you want to capture additional data beyond what the standard RAG task implementation provides.

      Example: 

      
 Task task = LangChain4jSupport.customTask(example -> {
     String query = example.input();

     // Track the latency
     long start = System.currentTimeMillis();
     Result<String> result = assistant.chat(query);
     long duration = System.currentTimeMillis() - start;

     return Map.of(
         "output", result.content(),
         "context", LangChain4jSupport.extractTexts(result.sources()),
         "latencyMs", duration,
         "sourceCount", result.sources().size()
     );
 });
      Parameters:
      taskFunction - a function that takes an Example and returns outputs
      Returns:
      a Task suitable for Experiment

    • measuredTask

      public static MeasuredTask measuredTask(dev.langchain4j.model.chat.ChatModel model, String modelId, PriceTable prices)
      Creates a measured Q&A MeasuredTask that captures token usage, latency, and (when a PriceTable is supplied) cost, lighting up the run's metrics cards.

      This is the metrics-bearing counterpart to simpleTask(ChatModel). Where the plain simpleTask returns a Task whose result structurally cannot carry CallMetrics, this returns a MeasuredTask, so switch the builder call from .task(...) to .measuredTask(...): 

      
 PriceTable prices = (model, in, out) -> ...;  // your price map, or null for tokens+latency only
 Experiment.builder()
     .measuredTask(LangChain4jSupport.measuredTask(model, "<your-model>", prices))
     .evaluators(...)
     .build()
     .run();

      The call uses the ChatRequest-based overload of ChatModel so the ChatResponse's TokenUsage is available; the String-returning chat(String) used by simpleTask does not expose usage. When usage is absent the token fields are null; when prices is null (or returns null) the cost stays null and only the Tokens and Latency cards light up. Never throws on missing metrics.

      Parameters:
      model - the ChatModel to evaluate, never null
      modelId - the model id used as the PriceTable lookup key, or null to skip pricing
      prices - the price lookup, or null to capture tokens and latency only
      Returns:
      a MeasuredTask suitable for Experiment.builder().measuredTask(...)
      Throws:
      IllegalArgumentException - if model is null

    • measuredTask

      public static MeasuredTask measuredTask(dev.langchain4j.model.chat.ChatModel model, String modelId, PriceTable prices, String outputKey)
      Creates a measured Q&A MeasuredTask that writes the response under a caller-chosen key.

      Behaves like measuredTask(ChatModel, String, PriceTable) but lets you override the default output key.

      Parameters:
      model - the ChatModel to evaluate, never null
      modelId - the model id used as the PriceTable lookup key, or null to skip pricing
      prices - the price lookup, or null to capture tokens and latency only
      outputKey - the key for the output in the result map, never null
      Returns:
      a MeasuredTask suitable for Experiment.builder().measuredTask(...)
      Throws:
      IllegalArgumentException - if model or outputKey is null

    • measuredRagTask

      public static MeasuredTask measuredRagTask(Function<String,dev.langchain4j.service.Result<String>> assistantCall, String modelId, PriceTable prices)
      Creates a measured RAG MeasuredTask from a function returning Result, capturing the token usage, latency, and (when a PriceTable is supplied) cost alongside the output and retrieved context.

      Metrics-bearing counterpart to ragTask(Function); use .measuredTask(...) on the builder. When the Result carries no usage the token fields are null; a null prices (or a null lookup result) leaves cost null and lights only the Tokens and Latency cards.

      Parameters:
      assistantCall - a function that takes the input string and returns a Result, never null
      modelId - the model id used as the PriceTable lookup key, or null to skip pricing
      prices - the price lookup, or null to capture tokens and latency only
      Returns:
      a MeasuredTask suitable for RAG evaluation
      Throws:
      IllegalArgumentException - if assistantCall is null

    • measuredRagTask

      public static MeasuredTask measuredRagTask(Function<String,dev.langchain4j.service.Result<String>> assistantCall, String inputKey, String outputKey, String contextKey, String modelId, PriceTable prices)
      Creates a measured RAG MeasuredTask with custom key names.
      Parameters:
      assistantCall - a function that takes the input string and returns a Result, never null
      inputKey - the key to read from example inputs, never null
      outputKey - the key for the output in the result map, never null
      contextKey - the key for the retrieval context in the result map, never null
      modelId - the model id used as the PriceTable lookup key, or null to skip pricing
      prices - the price lookup, or null to capture tokens and latency only
      Returns:
      a MeasuredTask suitable for RAG evaluation
      Throws:
      IllegalArgumentException - if assistantCall, inputKey, outputKey, or contextKey is null

    • asyncRagTask

      public static AsyncTask asyncRagTask(Function<String,dev.langchain4j.service.Result<String>> assistantCall)
      Creates an AsyncTask for RAG evaluation from a function that returns Result.

      Async version of ragTask(Function): the blocking assistant call is dispatched on the common ForkJoinPool via CompletableFuture.supplyAsync(java.util.function.Supplier), so the experiment's async execution path can keep many calls in flight without a thread blocked per example. The output and retrieved context are written under the default and context keys.

      Note: because the call blocks on the common pool, the experiment's parallelism bounds how many invocations are launched, but the effective concurrency of the blocking call is also limited by the common pool (~one less than the CPU count), which is shared process-wide. For higher, isolated concurrency use the Executor-accepting overload (asyncRagTask(Function, java.util.concurrent.Executor)) to run calls on a pool you control.

      Example: 

      
 interface Assistant {
     Result<String> chat(String userMessage);
 }

 Assistant assistant = AiServices.builder(Assistant.class)
     .chatModel(chatModel)
     .retrievalAugmentor(retrievalAugmentor)
     .build();

 AsyncTask task = LangChain4jSupport.asyncRagTask(assistant::chat);

 Experiment.builder()
     .asyncTask(task)
     .parallelism(8)
     .evaluators(List.of(faithfulness, contextRelevancy))
     .build()
     .run();
      Parameters:
      assistantCall - a function that takes the input string and returns a Result, never null
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if assistantCall is null

    • asyncRagTask

      public static AsyncTask asyncRagTask(Function<String,dev.langchain4j.service.Result<String>> assistantCall, Executor executor)
      Creates an AsyncTask for RAG evaluation with default key names, dispatching each blocking assistant call on the supplied Executor so you control and isolate concurrency.
      Parameters:
      assistantCall - a function that takes the input string and returns a Result, never null
      executor - the executor each blocking call runs on, never null
      Returns:
      an AsyncTask suitable for RAG evaluation
      Throws:
      IllegalArgumentException - if assistantCall or executor is null

    • asyncRagTask

      public static AsyncTask asyncRagTask(Function<String,dev.langchain4j.service.Result<String>> assistantCall, String inputKey, String outputKey, String contextKey)
      Creates an AsyncTask for RAG evaluation with custom key names.

      Behaves like asyncRagTask(Function) but reads the input from inputKey and writes the output and context under outputKey and contextKey, for datasets or evaluators that use different key names.

      Parameters:
      assistantCall - a function that takes the input string and returns a Result, never null
      inputKey - the key to read from example inputs, never null
      outputKey - the key for the output in the result map, never null
      contextKey - the key for the retrieval context in the result map, never null
      Returns:
      an AsyncTask suitable for RAG evaluation
      Throws:
      IllegalArgumentException - if any argument is null

    • asyncRagTask

      public static AsyncTask asyncRagTask(Function<String,dev.langchain4j.service.Result<String>> assistantCall, String inputKey, String outputKey, String contextKey, Executor executor)
      Creates an AsyncTask for RAG evaluation that dispatches each blocking assistant call on the supplied Executor (or the common ForkJoinPool when executor is null).

      Supplying an executor lets you control and isolate concurrency: the experiment's parallelism bounds in-flight invocations, and a pool sized to match gives true parallel blocking calls instead of the common pool's ~CPU-count, process-wide ceiling.

      Parameters:
      assistantCall - a function that takes the input string and returns a Result, never null
      inputKey - the key to read from example inputs, never null
      outputKey - the key for the output in the result map, never null
      contextKey - the key for the retrieval context in the result map, never null
      executor - the executor each blocking call runs on, or null for the common pool
      Returns:
      an AsyncTask suitable for RAG evaluation
      Throws:
      IllegalArgumentException - if assistantCall, inputKey, outputKey, or contextKey is null

    • asyncTask

      public static AsyncTask asyncTask(dev.langchain4j.model.chat.ChatModel model)
      Creates a simple AsyncTask for Q&A evaluation from a LangChain4j ChatModel.

      Async version of simpleTask(ChatModel): the blocking model.chat(...) call is dispatched on the common ForkJoinPool via CompletableFuture.supplyAsync(java.util.function.Supplier). The response is written under the default output key.

      Example: 

      
 ChatModel model = OpenAiChatModel.builder()...build();
 AsyncTask task = LangChain4jSupport.asyncTask(model);
      Parameters:
      model - the ChatModel to evaluate, never null
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if model is null

    • asyncTask

      public static AsyncTask asyncTask(dev.langchain4j.model.chat.ChatModel model, Executor executor)
      Creates a simple AsyncTask for Q&A evaluation with the default output key, dispatching each blocking model.chat(...) call on the supplied Executor so you control and isolate concurrency.
      Parameters:
      model - the ChatModel to evaluate, never null
      executor - the executor each blocking call runs on, never null
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if model or executor is null

    • asyncTask

      public static AsyncTask asyncTask(dev.langchain4j.model.chat.ChatModel model, String outputKey)
      Creates a simple AsyncTask for Q&A evaluation that writes the response under a caller-chosen key.

      Behaves like asyncTask(ChatModel) but lets you override the default output key.

      Parameters:
      model - the ChatModel to evaluate, never null
      outputKey - the key for the output in the result map, never null
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if any argument is null

    • asyncTask

      public static AsyncTask asyncTask(dev.langchain4j.model.chat.ChatModel model, String outputKey, Executor executor)
      Creates a simple AsyncTask for Q&A evaluation that dispatches each blocking model.chat(...) call on the supplied Executor (or the common ForkJoinPool when executor is null).

      Supplying an executor lets you control and isolate concurrency: the experiment's parallelism bounds in-flight invocations, and a pool sized to match gives true parallel blocking calls instead of the common pool's ~CPU-count, process-wide ceiling.

      Parameters:
      model - the ChatModel to evaluate, never null
      outputKey - the key for the output in the result map, never null
      executor - the executor each blocking call runs on, or null for the common pool
      Returns:
      an AsyncTask suitable for Experiment.builder().asyncTask(...)
      Throws:
      IllegalArgumentException - if model or outputKey is null

    • extractTexts

      public static List<String> extractTexts(List<dev.langchain4j.rag.content.Content> contents)
      Extracts text content from a list of LangChain4j Content objects.

      This is useful when building custom Tasks.

      Parameters:
      contents - the list of Content from result.sources()
      Returns:
      list of text strings, empty list if contents is null

    • extractTextsWithMetadata

      public static List<Map<String,Object>> extractTextsWithMetadata(List<dev.langchain4j.rag.content.Content> contents)
      Extracts text content with metadata from a list of LangChain4j Content objects.

      Returns a list of maps, where each map contains:

      • text - the segment text
      • metadata - the segment metadata as a map

      This is useful when you need source attribution in evaluations.

      Parameters:
      contents - the list of Content from result.sources()
      Returns:
      list of maps containing text and metadata

    • toAgentTrace

      public static AgentTrace toAgentTrace(dev.langchain4j.service.Result<?> result)
      Builds an AgentTrace from a LangChain4j Result.

      The result's content() becomes the final response and its toolExecutions() become ToolCalls carrying the tool name, parsed arguments, and the tool result string. Use it to evaluate tool-calling agents built with AiServices that return Result<T>. 

      
 Result<String> result = assistant.chat(userMessage);
 AgentTrace trace = LangChain4jSupport.toAgentTrace(result);
 EvalTestCase testCase = trace.toTestCase(userMessage, tools);
      Parameters:
      result - the LangChain4j result (may be null)
      Returns:
      an agent trace, never null

    • toToolCalls

      public static List<ToolCall> toToolCalls(dev.langchain4j.service.Result<?> result)
      Extracts ToolCalls from a LangChain4j Result in execution order.
      Parameters:
      result - the result (may be null)
      Returns:
      the tool calls, or an empty list when there are none

    • toToolCall

      public static ToolCall toToolCall(dev.langchain4j.service.tool.ToolExecution execution)
      Converts a single LangChain4j ToolExecution to a ToolCall.
      Parameters:
      execution - the tool execution
      Returns:
      the tool call

    • toToolDefinitions

      public static List<ToolDefinition> toToolDefinitions(List<dev.langchain4j.agent.tool.ToolSpecification> specifications)
      Converts LangChain4j ToolSpecifications to ToolDefinitions so tool calls can be evaluated against the tools the agent was given.
      Parameters:
      specifications - the tool specifications (may be null)
      Returns:
      the tool definitions, or an empty list

    • toToolDefinition

      public static ToolDefinition toToolDefinition(dev.langchain4j.agent.tool.ToolSpecification specification)
      Converts a single ToolSpecification to a ToolDefinition.
      Parameters:
      specification - the tool specification
      Returns:
      the tool definition