Skip to main content

Ctrl+K

eval-toolkit

Getting Started
What’s new
Examples
Worked example: metrics + bootstrap CIs
Worked example: slice-aware evaluate harness

GitHub
PyPI

Getting Started
What’s new
Examples
Worked example: metrics + bootstrap CIs
Worked example: slice-aware evaluate harness
Worked example: calibration with Platt + isotonic
Worked example: leakage detection
Worked example: claims + evidence gates
Worked example: paired bootstrap comparison
Worked example: prompt-injection classifier evaluation
Worked example: PyTorch + LoRA Scorer adapter
Worked example: declarative OOD slate loading
Worked example: character-injection adversarial sweep
Worked example: ActivationDeltaProbe (TaskTracker port)
Worked example: Spotlighting structural defenses
Worked example: RecallAtLowFPR loss training
Methodology
Splits
Comparison & confidence intervals
Reproducibility
Claims and Gates
Prediction Artifacts and Metric States
Evidence And Claims
Bootstrap
Calibration
Leakage
Text deduplication
Fairness & subgroup slicing
Stratified PR-AUC & the gap-flag report
Parallelism
Reading list
Testing your evaluation code
Threshold selection
Versioning Tier-2 implementations
API reference
eval_toolkit.analysis
eval_toolkit.artifacts
eval_toolkit.bootstrap
eval_toolkit.calibration
eval_toolkit.claims
eval_toolkit.config
eval_toolkit.docs
eval_toolkit.embeddings
eval_toolkit.evidence
eval_toolkit.harness
eval_toolkit.leakage
eval_toolkit.loaders
eval_toolkit.manifest
eval_toolkit.metrics
eval_toolkit.operating_points
eval_toolkit.paths
eval_toolkit.plotting
eval_toolkit.protocols
eval_toolkit.provenance
eval_toolkit.seeds
eval_toolkit.splits
eval_toolkit.text_dedup
eval_toolkit.thresholds
v0.6.x → v0.7.x migration
v0.7.x → v0.8.0 migration
v0.8.x → v0.9.0 migration
Extending eval-toolkit
Schema Reference
Roadmap
Repo Strategy
Deprecation policy
Migration guides
Releasing eval-toolkit

GitHub
PyPI

Section Navigation

eval_toolkit.metrics.DEFAULT_ASSUMED_PRIORS
eval_toolkit.metrics.ThresholdResult
eval_toolkit.metrics.brier_decomposition
eval_toolkit.metrics.brier_score
eval_toolkit.metrics.expected_calibration_error
eval_toolkit.metrics.expected_calibration_error_debiased
eval_toolkit.metrics.expected_calibration_error_equal_mass
eval_toolkit.metrics.expected_calibration_error_l2
eval_toolkit.metrics.expected_calibration_error_l2_debiased
eval_toolkit.metrics.headline_metrics
eval_toolkit.metrics.metrics_at_threshold
eval_toolkit.metrics.pr_auc
eval_toolkit.metrics.precision_at_prior
eval_toolkit.metrics.quantile_stratified_pr_auc
eval_toolkit.metrics.quantile_stratified_report
eval_toolkit.metrics.roc_auc
eval_toolkit.metrics.score_distribution_summary
eval_toolkit.metrics.single_class_threshold_metrics
eval_toolkit.metrics.stratified_recall

eval_toolkit.metrics
eval_toolkit.metrics.pr_auc

eval_toolkit.metrics.pr_auc#

.. currentmodule:: eval_toolkit.metrics

.. autofunction:: pr_auc

Edit on GitHub

Show Source

Created using Sphinx 9.1.0.

Built with the PyData Sphinx Theme 0.17.1.