Package dev.dokimos.core

Class Experiment.Builder

java.lang.Object
dev.dokimos.core.Experiment.Builder
Enclosing class:
Experiment
public static class Experiment.Builder extends Object

  • Constructor Details

    • Builder

      public Builder()

  • Method Details

    • name

      public Experiment.Builder name(String name)
      Sets the experiment name.
      Parameters:
      name - the experiment name
      Returns:
      this builder

    • description

      public Experiment.Builder description(String description)
      Sets the description.
      Parameters:
      description - The experiment's description.
      Returns:
      builder

    • dataset

      public Experiment.Builder dataset(Dataset dataset)
      Sets the dataset.
      Parameters:
      dataset - The dataset to use for the experiment.
      Returns:
      builder

    • task

      public Experiment.Builder task(Task task)
      Sets the task.
      Parameters:
      task - The task to generate outputs from examples.
      Returns:
      builder

    • measuredTask

      public Experiment.Builder measuredTask(MeasuredTask measuredTask)
      Sets a measured task that carries CallMetrics through to each ItemResult.
      Parameters:
      measuredTask - The task to generate outputs and metrics from examples.
      Returns:
      builder

    • asyncTask

      public Experiment.Builder asyncTask(AsyncTask asyncTask)
      Sets an asynchronous task that produces a TaskResult as a CompletableFuture.

      When an async task is set, the experiment runs through a dedicated non-blocking execution path that bounds the number of in-flight invocations to parallelism(int) using a semaphore. This path takes precedence over parallelism(int)-based parallel and sequential execution. Per-item failures are isolated exactly as in the synchronous paths: a failed future becomes a failed ItemResult and the run continues.

      An async task is mutually exclusive with a synchronous task(Task) or measuredTask(MeasuredTask): configuring both fails fast in build().

      Parameters:
      asyncTask - the asynchronous task to generate outputs and metrics from examples
      Returns:
      builder

    • evaluator

      public Experiment.Builder evaluator(Evaluator evaluator)
      Adds a single evaluator to the experiment.
      Parameters:
      evaluator - The evaluator to add.
      Returns:
      builder

    • evaluators

      public Experiment.Builder evaluators(List<Evaluator> evaluators)
      Adds multiple evaluators to the experiment.
      Parameters:
      evaluators - The list of evaluators to add.
      Returns:
      builder

    • metadata

      public Experiment.Builder metadata(String key, Object value)
      Adds a metadata entry to the experiment.
      Parameters:
      key - The metadata key.
      value - The metadata value.
      Returns:
      builder

    • metadata

      public Experiment.Builder metadata(Map<String,Object> metadata)
      Adds multiple metadata entries to the experiment.
      Parameters:
      metadata - The map of metadata entries to add.
      Returns:
      builder

    • reporter

      public Experiment.Builder reporter(Reporter reporter)
      Sets the reporter for this experiment.

      The reporter is called during experiment execution to report results to an external system. If not set, a no-op reporter is used.

      Parameters:
      reporter - the reporter to use
      Returns:
      builder

    • parallelism

      public Experiment.Builder parallelism(int parallelism)
      Sets the level of parallelism for running the experiment.

      When parallelism is greater than 1, examples within each run are processed concurrently using a fixed thread pool. Default is 1 for sequential execution.

      Ensure your task implementation is thread-safe when using parallelism.

      Parameters:
      parallelism - the number of examples to process in parallel, must be at least 1
      Returns:
      builder
      Throws:
      IllegalArgumentException - if parallelism is less than 1

    • runs

      public Experiment.Builder runs(int runs)
      Sets the number of times to run the experiment.

      Running an experiment multiple times helps reduce variance from LLM non-determinism and provides statistical confidence in the results. Results are automatically aggregated across runs.

      Runs execute sequentially while parallelism applies within each run.

      Parameters:
      runs - the number of experiment runs, must be at least 1
      Returns:
      builder
      Throws:
      IllegalArgumentException - if runs is less than 1

    • autoCloseReporter

      public Experiment.Builder autoCloseReporter(boolean autoCloseReporter)
      Controls whether Experiment.run() closes the reporter after the run completes.

      When enabled, the reporter is closed (in addition to being flushed) once all runs finish. Defaults to false so the caller retains ownership of the reporter lifecycle.

      Parameters:
      autoCloseReporter - whether to close the reporter after the run completes
      Returns:
      builder

    • build

      public Experiment build()
      Builds the experiment.
      Returns:
      a new experiment
      Throws:
      IllegalStateException - if dataset or task is not set, the dataset has no examples, or no evaluators were added