Package dev.dokimos.core

Class BaseEvaluator

java.lang.Object
dev.dokimos.core.BaseEvaluator
All Implemented Interfaces:
Evaluator
Direct Known Subclasses:
ContextualRelevanceEvaluator, ExactMatchEvaluator, FaithfulnessEvaluator, HallucinationEvaluator, LLMJudgeEvaluator, PrecisionEvaluator, RecallEvaluator, RegexEvaluator, TaskCompletionEvaluator, ToolArgumentHallucinationEvaluator, ToolCallValidityEvaluator, ToolCorrectnessEvaluator, ToolDescriptionReliabilityEvaluator, ToolNameReliabilityEvaluator, TrajectoryEvaluator
public abstract class BaseEvaluator extends Object implements Evaluator
Base class for implementing concrete evaluators.

Provides parameter validation before evaluation and async evaluation support.

  • Method Details

    • evaluate

      public final EvalResult evaluate(EvalTestCase testCase)
      Description copied from interface: Evaluator
      Evaluates the test case and returns a scored result.
      Specified by:
      evaluate in interface Evaluator
      Parameters:
      testCase - the test case to evaluate
      Returns:
      the evaluation result

    • name

      public String name()
      Description copied from interface: Evaluator
      Returns the evaluator name.
      Specified by:
      name in interface Evaluator
      Returns:
      the evaluator name

    • threshold

      public double threshold()
      Description copied from interface: Evaluator
      Returns the minimum score threshold for success.
      Specified by:
      threshold in interface Evaluator
      Returns:
      the threshold value

    • evaluateAsync

      public CompletableFuture<EvalResult> evaluateAsync(EvalTestCase testCase)
      Evaluates the test case asynchronously using the common fork-join pool.

      This method allows for non-blocking evaluation, which is useful when processing multiple test cases concurrently.

      Parameters:
      testCase - the test case to evaluate
      Returns:
      a CompletableFuture that will complete with the evaluation result

    • evaluateAsync

      public CompletableFuture<EvalResult> evaluateAsync(EvalTestCase testCase, ExecutorService executor)
      Evaluates the test case asynchronously using the provided executor.
      Parameters:
      testCase - the test case to evaluate
      executor - the executor to use for async execution
      Returns:
      a CompletableFuture that will complete with the evaluation result