Package dev.dokimos.core.evaluators

Class ContextualRelevanceEvaluator

java.lang.Object

dev.dokimos.core.BaseEvaluator

dev.dokimos.core.evaluators.ContextualRelevanceEvaluator

All Implemented Interfaces:

Evaluator

public class ContextualRelevanceEvaluator
extends BaseEvaluator

Evaluator that measures how relevant retrieved context chunks are to a user's input query.

This evaluator uses pointwise relevance scoring, where each context chunk is scored independently between 0.0 (completely irrelevant) and 1.0 (highly relevant). The final score is the mean average of all individual chunk scores.

Example usage:

 var evaluator = ContextualRelevanceEvaluator.builder()
     .name("contextual-relevance")
     .judge(llmClient)
     .threshold(0.5)
     .build();

 var testCase = EvalTestCase.builder()
     .input("What are symptoms of dehydration?")
     .actualOutput("retrievalContext", List.of(
         "Dehydration symptoms include thirst, dry mouth, and fatigue.",
         "The Pacific Ocean is the largest ocean on Earth.",
         "Severe dehydration can cause dizziness and confusion."
     ))
     .build();

 EvalResult result = evaluator.evaluate(testCase);
 

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static class	ContextualRelevanceEvaluator.Builder	Builder for constructing ContextualRelevanceEvaluator instances.
static final record	ContextualRelevanceEvaluator.ContextScore	Represents the relevance score for a single context chunk.

Method Summary

Modifier and Type	Method	Description
static ContextualRelevanceEvaluator.Builder	builder()	Creates a new builder for constructing the contextual relevance evaluator.

Methods inherited from class dev.dokimos.core.BaseEvaluator

evaluate, evaluateAsync, evaluateAsync, name, threshold

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

builder

public static ContextualRelevanceEvaluator.Builder builder()

Creates a new builder for constructing the contextual relevance evaluator.

Returns:

a new builder