Package dev.dokimos.server.service

Class AlignmentService

java.lang.Object

dev.dokimos.server.service.AlignmentService

@Service
public class AlignmentService
extends Object

Computes per-run, per-evaluator agreement between automated evaluator verdicts and human annotations. A CORRECT verdict is treated as the item passing and INCORRECT as failing; an evaluator result agrees when its success flag matches that expectation. UNSURE verdicts and items with no annotation are excluded from the rate, and UNSURE counts are reported separately so the caller can show how much of the run is still ambiguous.

Constructor Summary

Constructors

Constructor	Description
AlignmentService(ExperimentRunRepository runRepository, ItemResultRepository itemResultRepository, AnnotationRepository annotationRepository)

Method Summary

Modifier and Type	Method	Description
AlignmentView	getAlignment(UUID runId)	Computes the judge-human alignment breakdown for a run.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

AlignmentService

public AlignmentService(ExperimentRunRepository runRepository,
 ItemResultRepository itemResultRepository,
 AnnotationRepository annotationRepository)

Method Details

getAlignment

@Transactional(readOnly=true)
public AlignmentView getAlignment(UUID runId)

Computes the judge-human alignment breakdown for a run. Items are loaded with their eval results in one query and their annotations batch-loaded by item id to avoid an N+1 fan-out. Each evaluator's rate uses only items it ran on whose human verdict was CORRECT or INCORRECT.

Parameters:

runId - the run to analyze

Returns:

the per-evaluator agreement breakdown, ordered by first appearance of each evaluator

Throws:

IllegalArgumentException - if runId is null or the run does not exist