Getting Started

Get the Dokimos server running in under a minute. No building, no cloning—just Docker.

Start the Server

Run these two commands:

# Download the compose file
curl -O https://raw.githubusercontent.com/dokimos-dev/dokimos/master/docker-compose.yml

# Start the server
docker compose up -d

The server will now be running at http://localhost:8080.

No Docker?

If you don't have Docker installed, get it from docker.com.

Send Your First Results

Add the client dependency to your project:

<dependency>
    <groupId>dev.dokimos</groupId>
    <artifactId>dokimos-server-client</artifactId>
    <version>${dokimos.version}</version>
</dependency>

Create an experiment that reports to the server:

Java

import dev.dokimos.core.*;
import dev.dokimos.server.client.DokimosServerReporter;

public class MyFirstServerExperiment {
    public static void main(String[] args) {
        // Create dataset
        Dataset dataset = Dataset.builder()
            .name("Capital Cities")
            .addExample(Example.of("What is the capital of France?", "Paris"))
            .addExample(Example.of("What is the capital of Japan?", "Tokyo"))
            .build();

        // Connect to the local server
        DokimosServerReporter reporter = DokimosServerReporter.builder()
            .serverUrl("http://localhost:8080")
            .projectName("my-first-project")
            .build();

        // Run experiment
        ExperimentResult result = Experiment.builder()
            .name("capitals-qa")
            .dataset(dataset)
            .task(example -> {
                String answer = callYourLLM(example.input());
                return Map.of("output", answer);
            })
            .evaluators(List.of(
                ExactMatchEvaluator.builder()
                    .name("exact-match")
                    .threshold(1.0)
                    .build()
            ))
            .reporter(reporter)
            .build()
            .run();

        System.out.println("Pass rate: " + result.passRate());
    }
}

Kotlin

import dev.dokimos.kotlin.dsl.dataset
import dev.dokimos.kotlin.dsl.exactMatch
import dev.dokimos.kotlin.dsl.experiment
import dev.dokimos.kotlin.dsl.task
import dev.dokimos.server.client.DokimosServerReporter

fun main() {
    // Create dataset
    val dataset = dataset {
        name = "Capital Cities"
        example {
            input = "What is the capital of France?"
            expected = "Paris"
        }
        example {
            input = "What is the capital of Japan?"
            expected = "Tokyo"
        }
    }

    // Connect to the local server
    val reporter = DokimosServerReporter.builder()
        .serverUrl("http://localhost:8080")
        .projectName("my-first-project")
        .build()

    // Run experiment
    val result = experiment {
        name = "capitals-qa"
        dataset(dataset)
        task {
            val answer = callYourLLM(input())
            mapOf("output" to answer)
        }
        evaluators {
            exactMatch {
                name = "exact-match"
                threshold = 1.0
            }
        }
        reporter(reporter)
    }.run()

    println("Pass rate: ${result.passRate()}")
}

View Results in the UI

After running your experiment:

1. Open http://localhost:8080
2. Click on your project "my-first-project"
3. Click on the experiment to see pass rates
4. Click on a run to see individual test cases and evaluation details

Managing the Server

# View logs
docker compose logs -f server

# Stop the server 
docker compose down

# Stop and remove all data
docker compose down -v

Next Steps

• Configuration: Customize settings and environment variables
• Deployment: Share with your team or run in production
• Authentication: Secure write operations with an API key
• Client: Advanced reporter configuration

---

Building from Source (Development)

If you're contributing to Dokimos and need to build the server locally:

# Clone the repository
git clone https://github.com/dokimos-dev/dokimos.git
cd dokimos

# Use the development compose file
cd dokimos-server
docker compose -f docker-compose.dev.yml up --build

See the Server README for more details.