Package dev.dokimos.core

Class Experiment.Builder

java.lang.Object
dev.dokimos.core.Experiment.Builder
Enclosing class:
Experiment
public static class Experiment.Builder extends Object

  • Constructor Details

    • Builder

      public Builder()

  • Method Details

    • name

      public Experiment.Builder name(String name)
      Sets the experiment name.
      Parameters:
      name - the experiment name
      Returns:
      this builder

    • description

      public Experiment.Builder description(String description)
      Sets the description.
      Parameters:
      description - The experiment's description.
      Returns:
      builder

    • dataset

      public Experiment.Builder dataset(Dataset dataset)
      Sets the dataset.
      Parameters:
      dataset - The dataset to use for the experiment.
      Returns:
      builder

    • task

      public Experiment.Builder task(Task task)
      Sets the task.
      Parameters:
      task - The task to generate outputs from examples.
      Returns:
      builder

    • evaluator

      public Experiment.Builder evaluator(Evaluator evaluator)
      Adds a single evaluator to the experiment.
      Parameters:
      evaluator - The evaluator to add.
      Returns:
      builder

    • evaluators

      public Experiment.Builder evaluators(List<Evaluator> evaluators)
      Adds multiple evaluators to the experiment.
      Parameters:
      evaluators - The list of evaluators to add.
      Returns:
      builder

    • metadata

      public Experiment.Builder metadata(String key, Object value)
      Adds a metadata entry to the experiment.
      Parameters:
      key - The metadata key.
      value - The metadata value.
      Returns:
      builder

    • metadata

      public Experiment.Builder metadata(Map<String,Object> metadata)
      Adds multiple metadata entries to the experiment.
      Parameters:
      metadata - The map of metadata entries to add.
      Returns:
      builder

    • reporter

      public Experiment.Builder reporter(Reporter reporter)
      Sets the reporter for this experiment.

      The reporter is called during experiment execution to report results to an external system. If not set, a no-op reporter is used.

      Parameters:
      reporter - the reporter to use
      Returns:
      builder

    • parallelism

      public Experiment.Builder parallelism(int parallelism)
      Sets the level of parallelism for running the experiment.

      When parallelism is greater than 1, examples within each run are processed concurrently using a fixed thread pool. Default is 1 for sequential execution.

      Ensure your task implementation is thread-safe when using parallelism.

      Parameters:
      parallelism - the number of examples to process in parallel, must be at least 1
      Returns:
      builder
      Throws:
      IllegalArgumentException - if parallelism is less than 1

    • runs

      public Experiment.Builder runs(int runs)
      Sets the number of times to run the experiment.

      Running an experiment multiple times helps reduce variance from LLM non-determinism and provides statistical confidence in the results. Results are automatically aggregated across runs.

      Runs execute sequentially while parallelism applies within each run.

      Parameters:
      runs - the number of experiment runs, must be at least 1
      Returns:
      builder
      Throws:
      IllegalArgumentException - if runs is less than 1

    • build

      public Experiment build()
      Builds the experiment.
      Returns:
      a new experiment
      Throws:
      IllegalStateException - if dataset or task is not set