Skip to main content

LangChain4j Integration

This page shows you how to evaluate your LangChain4j AI Services and RAG pipelines with Dokimos. You write less glue code because Dokimos reads the retrieved documents straight out of LangChain4j's results.

Why use this integration​

Automatic context extraction: A LangChain4j Result<T> already holds the retrieved documents. Dokimos pulls them out for you, so you never track context by hand.

One-line conversion: Turn a ChatModel or an AI Service into a Dokimos Task with a single call.

Ready for RAG: Use FaithfulnessEvaluator to check that answers stay grounded in the retrieved documents.

Setup​

Add the integration dependency to your pom.xml:

<dependency>
    <groupId>dev.dokimos</groupId>
    <artifactId>dokimos-langchain4j</artifactId>
    <version>${dokimos.version}</version>
</dependency>

Basic usage​

Evaluate a simple ChatModel​

Wrap a LangChain4j ChatModel in a Task and run an experiment:

import dev.dokimos.langchain4j.LangChain4jSupport;
import dev.langchain4j.model.openai.OpenAiChatModel;

ChatModel model = OpenAiChatModel.builder()
    .apiKey(System.getenv("OPENAI_API_KEY"))
    .modelName("gpt-5.2")
    .build();

// Convert to Task
Task task = LangChain4jSupport.simpleTask(model);

// Run experiment
ExperimentResult result = Experiment.builder()
    .name("ChatModel Evaluation")
    .dataset(dataset)
    .task(task)
    .evaluators(evaluators)
    .build()
    .run();

simpleTask(model) writes the response under the default "output" key. To use a different key, pass it as the second argument:

// Writes the response under "answer" instead of "output"
Task task = LangChain4jSupport.simpleTask(model, "answer");

Use a ChatModel as an LLM judge​

Turn a ChatModel into a JudgeLM so an evaluator can use it to score answers:

import dev.dokimos.langchain4j.LangChain4jSupport;

ChatModel judgeModel = OpenAiChatModel.builder()
    .apiKey(System.getenv("OPENAI_API_KEY"))
    .modelName("gpt-5.2")
    .build();

// Convert to JudgeLM
JudgeLM judge = LangChain4jSupport.asJudge(judgeModel);

// Use in evaluators
Evaluator correctness = LLMJudgeEvaluator.builder()
    .name("Answer Correctness")
    .criteria("Is the answer factually correct?")
    .judge(judge)
    .threshold(0.8)
    .build();

Evaluate RAG systems​

Evaluating RAG is the main reason to reach for this integration. Here is a full example you can copy and adapt:

import dev.dokimos.langchain4j.LangChain4jSupport;
import dev.langchain4j.service.AiServices;
import dev.langchain4j.service.Result;

// 1. Define your AI Service interface (must return Result<String>)
interface Assistant {
    Result<String> chat(String userMessage);
}

// 2. Build your RAG pipeline
Assistant assistant = AiServices.builder(Assistant.class)
    .chatLanguageModel(chatModel)
    .contentRetriever(EmbeddingStoreContentRetriever.builder()
        .embeddingStore(embeddingStore)
        .embeddingModel(embeddingModel)
        .maxResults(3)
        .build())
    .build();

// 3. Create dataset
Dataset dataset = Dataset.builder()
    .name("customer-qa")
    .addExample(Example.of("What is the refund policy?", "30-day money-back guarantee"))
    .addExample(Example.of("How long does shipping take?", "5-7 business days"))
    .build();

// 4. Create Task (automatically extracts context from Result)
Task task = LangChain4jSupport.ragTask(assistant::chat);

// 5. Set up evaluators
JudgeLM judge = LangChain4jSupport.asJudge(judgeModel);

List<Evaluator> evaluators = List.of(
    // Check answer correctness
    LLMJudgeEvaluator.builder()
        .name("Answer Correctness")
        .criteria("Is the answer accurate and complete?")
        .judge(judge)
        .threshold(0.8)
        .build(),
    
    // Check faithfulness to retrieved context
    FaithfulnessEvaluator.builder()
        .threshold(0.7)
        .judge(judge)
        .build()
);

// 6. Run experiment
ExperimentResult result = Experiment.builder()
    .name("RAG Evaluation")
    .dataset(dataset)
    .task(task)
    .evaluators(evaluators)
    .build()
    .run();

// 7. Analyze results
System.out.println("Pass rate: " + result.passRate() * 100 + "%");
System.out.println("Faithfulness: " + result.averageScore("Faithfulness"));

How it works​

ragTask() reads the input, calls your AI Service, and pulls the retrieved context from Result.sources(). The output map holds both the answer and the context:

{
    "output": "We offer a 30-day money-back guarantee",
    "context": [
        "Refund policy: 30-day guarantee...",
        "Contact support to process refunds..."
    ]
}

FaithfulnessEvaluator then checks the answer against what was actually retrieved.

Async tasks​

Each RAG example is an independent, blocking model or assistant call. Async tasks let the experiment keep many of those calls in flight at once instead of blocking one thread per example. Wire them with Experiment.builder().asyncTask(...) and cap how many run at once with parallelism(...).

asyncTask(model) is the async version of simpleTask(model). asyncRagTask(assistantCall) is the async version of ragTask(...), and it still extracts the retrieved context from Result.sources() into the "context" key. Both run the blocking call on the common ForkJoinPool via CompletableFuture.supplyAsync(...).

import dev.dokimos.core.*;
import dev.dokimos.langchain4j.LangChain4jSupport;

// Simple Q&A
AsyncTask task = LangChain4jSupport.asyncTask(model);

// RAG (extracts context from Result.sources())
AsyncTask ragTask = LangChain4jSupport.asyncRagTask(assistant::chat);

ExperimentResult result = Experiment.builder()
    .name("LangChain4j Async RAG")
    .dataset(dataset)
    .asyncTask(ragTask)
    .parallelism(8)
    .evaluators(List.of(faithfulness, contextRelevancy))
    .build()
    .run();

To write under a different output key, use asyncTask(model, outputKey). For custom dataset keys, asyncRagTask has a four-argument overload: asyncRagTask(assistantCall, inputKey, outputKey, contextKey).

note

The common pool is shared across the whole process, and its effective parallelism is about one less than your CPU count. So it caps how many blocking calls actually run at once, even when you set parallelism higher. For controlled, isolated concurrency, pass an Executor sized to the throughput you want: asyncTask(model, executor) or asyncRagTask(assistantCall, executor).

import java.util.concurrent.Executor;
import java.util.concurrent.Executors;

// A pool sized to match your desired concurrency
Executor executor = Executors.newFixedThreadPool(16);

AsyncTask ragTask = LangChain4jSupport.asyncRagTask(assistant::chat, executor);

Experiment.builder()
    .dataset(dataset)
    .asyncTask(ragTask)
    .parallelism(16)
    .evaluators(List.of(faithfulness))
    .build()
    .run();

Advanced usage​

Custom dataset keys​

When your dataset uses different key names, map them in the ragTask call:

// Dataset with custom keys
Dataset dataset = Dataset.builder()
    .addExample(Example.builder()
        .input("question", "What is the refund policy?")
        .expectedOutput("answer", "30-day money-back guarantee")
        .build())
    .build();

// Map keys accordingly
Task task = LangChain4jSupport.ragTask(
    assistant::chat,
    "question",         // input key
    "answer",           // output key
    "retrievedContext"  // context key
);

Track extra metrics​

Use customTask() when you want to record latency, source counts, or other metrics alongside the answer:

Task task = LangChain4jSupport.customTask(example -> {
    long start = System.currentTimeMillis();
    Result<String> result = assistant.chat(example.input());
    long latency = System.currentTimeMillis() - start;
    
    return Map.of(
        "output", result.content(),
        "context", LangChain4jSupport.extractTexts(result.sources()),
        "latencyMs", latency,
        "numSources", result.sources().size()
    );
});

Context extraction utilities​

Pull retrieved context out of a Result in two formats:

// Simple text extraction
List<String> contextTexts = LangChain4jSupport.extractTexts(result.sources());
// ["Text from doc 1", "Text from doc 2"]

// With metadata (for source attribution)
List<Map<String, Object>> contextsWithMeta = 
    LangChain4jSupport.extractTextsWithMetadata(result.sources());
// [
//   {"text": "...", "metadata": {"source": "doc1.pdf", "page": 5}},
//   {"text": "...", "metadata": {"source": "doc2.pdf", "page": 12}}
// ]

RAG-specific evaluators​

Faithfulness evaluation​

Check that the output stays grounded in the retrieved context:

Evaluator faithfulness = FaithfulnessEvaluator.builder()
    .threshold(0.8)
    .judge(judge)
    .contextKey("context")  // Must match Task's context key
    .includeReason(true)
    .build();

The evaluator runs three steps:

  1. Extracts claims from the actual output.
  2. Verifies each claim against the retrieved context.
  3. Computes score = (supported claims) / (total claims).

Multi-dimensional RAG evaluation​

Score several quality aspects in one experiment:

List<Evaluator> evaluators = List.of(
    // Answer quality
    LLMJudgeEvaluator.builder()
        .name("Answer Quality")
        .criteria("Is the answer helpful and accurate?")
        .evaluationParams(List.of(
            EvalTestCaseParam.INPUT,
            EvalTestCaseParam.ACTUAL_OUTPUT
        ))
        .judge(judge)
        .threshold(0.8)
        .build(),
    
    // Faithfulness to sources
    FaithfulnessEvaluator.builder()
        .name("Faithfulness")
        .threshold(0.85)
        .judge(judge)
        .build(),
    
    // Context relevance
    LLMJudgeEvaluator.builder()
        .name("Context Relevance")
        .criteria("Is the retrieved context relevant to answering the question?")
        .evaluationParams(List.of(
            EvalTestCaseParam.INPUT,
            EvalTestCaseParam.METADATA  // Contains context
        ))
        .judge(judge)
        .threshold(0.75)
        .build()
);

Complete working example​

This example sets up an in-memory RAG pipeline, builds a dataset, and runs two evaluators end to end:

import dev.dokimos.core.*;
import dev.dokimos.langchain4j.LangChain4jSupport;
import dev.langchain4j.data.document.Document;
import dev.langchain4j.model.embedding.onnx.bgesmallenv15q.BgeSmallEnV15QuantizedEmbeddingModel;
import dev.langchain4j.model.openai.OpenAiChatModel;
import dev.langchain4j.rag.content.retriever.EmbeddingStoreContentRetriever;
import dev.langchain4j.service.AiServices;
import dev.langchain4j.service.Result;
import dev.langchain4j.store.embedding.inmemory.InMemoryEmbeddingStore;

public class RAGEvaluation {
    
    public static void main(String[] args) {
        // 1. Set up RAG components
        var embeddingModel = new BgeSmallEnV15QuantizedEmbeddingModel();
        var embeddingStore = new InMemoryEmbeddingStore<TextSegment>();
        
        // Ingest documents
        var documents = List.of(
            Document.from("We offer a 30-day money-back guarantee."),
            Document.from("Standard shipping takes 5-7 business days.")
        );
        
        EmbeddingStoreIngestor.builder()
            .embeddingModel(embeddingModel)
            .embeddingStore(embeddingStore)
            .build()
            .ingest(documents);
        
        // 2. Build AI Service
        interface Assistant {
            Result<String> chat(String userMessage);
        }
        
        Assistant assistant = AiServices.builder(Assistant.class)
            .chatLanguageModel(OpenAiChatModel.builder()
                .apiKey(System.getenv("OPENAI_API_KEY"))
                .modelName("gpt-5.2")
                .build())
            .contentRetriever(EmbeddingStoreContentRetriever.builder()
                .embeddingStore(embeddingStore)
                .embeddingModel(embeddingModel)
                .maxResults(2)
                .build())
            .build();
        
        // 3. Create dataset
        Dataset dataset = Dataset.builder()
            .name("customer-qa")
            .addExample(Example.of(
                "What is the refund policy?", 
                "30-day money-back guarantee"
            ))
            .addExample(Example.of(
                "How long does shipping take?", 
                "5-7 business days"
            ))
            .build();
        
        // 4. Set up evaluation
        var judgeModel = OpenAiChatModel.builder()
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .modelName("gpt-5.2")
            .build();
        
        JudgeLM judge = LangChain4jSupport.asJudge(judgeModel);
        
        List<Evaluator> evaluators = List.of(
            LLMJudgeEvaluator.builder()
                .name("Answer Quality")
                .criteria("Is the answer accurate?")
                .judge(judge)
                .threshold(0.8)
                .build(),
            FaithfulnessEvaluator.builder()
                .threshold(0.7)
                .judge(judge)
                .build()
        );
        
        // 5. Run experiment
        ExperimentResult result = Experiment.builder()
            .name("RAG Evaluation")
            .dataset(dataset)
            .task(LangChain4jSupport.ragTask(assistant::chat))
            .evaluators(evaluators)
            .build()
            .run();
        
        // 6. Display results
        System.out.println("Pass rate: " + 
            String.format("%.0f%%", result.passRate() * 100));
        System.out.println("Answer Quality: " + 
            String.format("%.2f", result.averageScore("Answer Quality")));
        System.out.println("Faithfulness: " + 
            String.format("%.2f", result.averageScore("Faithfulness")));
    }
}

Structured / typed output​

When your AI Service returns structured data, such as a record from a typed AI Service method, return that object under "output" instead of a string. Compare it with StructuralMatchEvaluator (numbers compare by value, so formatting and key order do not count), and read it back type-safely with actualOutputAs(Record.class).

record Invoice(String id, double total, List<String> items) {}

// A LangChain4j AI Service can return a typed value directly
interface Extractor {
    Invoice extract(String text);
}

Task task = Task.typed(example -> extractor.extract(example.input()));

Evaluator structural = StructuralMatchEvaluator.builder()
    .name("Invoice Match")
    .threshold(1.0)
    .build();

// In a custom evaluator, read the structured value back
Invoice actual = testCase.actualOutputAs(Invoice.class);

See the Structured & Typed Data hub for the full pipeline.

JUnit integration​

Combine this with JUnit to run evaluations as tests:

import dev.dokimos.junit.DatasetSource;
import org.junit.jupiter.params.ParameterizedTest;

@ParameterizedTest
@DatasetSource("classpath:datasets/rag-qa.json")
void ragSystemShouldAnswerCorrectly(Example example) {
    // Call your RAG system
    Result<String> result = assistant.chat(example.input());
    
    // Create test case with context
    Map<String, Object> outputs = Map.of(
        "output", result.content(),
        "context", LangChain4jSupport.extractTexts(result.sources())
    );
    EvalTestCase testCase = example.toTestCase(outputs);
    
    // Assert faithfulness
    Assertions.assertEval(testCase, faithfulnessEvaluator);
}

Best practices​

Always return Result<String>: Your AI Service interface must return Result<String>, not just String. That return type is how LangChain4j hands back the retrieved context.

// Good
interface Assistant {
    Result<String> chat(String message);
}

// Will not work (cannot extract context)
interface Assistant {
    String chat(String message);
}

Use a stronger model for judging: Judge with GPT-5.2 or similar, even when your application generates answers with a smaller model.

Track retrieval quality: Watch how many documents you retrieve and whether they are relevant. Add those metrics with customTask().

Test different retrieval settings: Run experiments that compare different maxResults values, embedding models, or reranking strategies.

For AI agentsView as Markdown