eval_toolkit.claims#

ClaimReport

Machine-readable result of evaluating claim specs.

ClaimSpec

A claim plus the gates required before it can be treated as supported.

EvidenceGate

Named callable gate used inside a ClaimSpec.

GateResult

Result of one evidence gate.

evaluate_claims

Evaluate claim specs against a result payload and optional manifest.

external_diagnostic_gate

Require an external diagnostic payload, optionally thresholded.

headline_present_gate

Require a non-null headline/comparison block at an arbitrary path.

low_fpr_feasibility_gate

Require enough negatives for a low-FPR claim to be statistically feasible.

metric_threshold_gate

Require a numeric metric to satisfy a threshold comparison.

minimum_slice_size_gate

Require minimum total/positive/negative counts for a slice.

no_leakage_errors_gate

Fail if result config or manifest leakage report has error-severity findings.

no_scorer_errors_gate

Fail if any scorer block contains an error field.

paired_diff_present_gate

Require a paired-difference comparison under a slice.

required_metric_gate

Require a metric path under one scorer result.

required_scorer_gate

Require a scorer result under a slice.

required_slice_gate

Require by_slice.<slice_name> to exist.

source_role_gate

Require source-role metadata with the requested roles in the manifest.

strict_artifact_gate

Fail if result or manifest contains non-finite numeric values.