Skip to main content

Ctrl+K

eval-toolkit

Getting Started
What’s new
Examples
Worked example: metrics + bootstrap CIs
Worked example: slice-aware evaluate harness

GitHub
PyPI

Getting Started
What’s new
Examples
Worked example: metrics + bootstrap CIs
Worked example: slice-aware evaluate harness
Worked example: calibration with Platt + isotonic
Worked example: leakage detection
Worked example: claims + evidence gates
Worked example: paired bootstrap comparison
Worked example: prompt-injection classifier evaluation
Worked example: PyTorch + LoRA Scorer adapter
Worked example: declarative OOD slate loading
Worked example: character-injection adversarial sweep
Worked example: ActivationDeltaProbe (TaskTracker port)
Worked example: Spotlighting structural defenses
Worked example: RecallAtLowFPR loss training
Methodology
Splits
Comparison & confidence intervals
Reproducibility
Claims and Gates
Prediction Artifacts and Metric States
Evidence And Claims
Bootstrap
Calibration
Leakage
Text deduplication
Fairness & subgroup slicing
Stratified PR-AUC & the gap-flag report
Parallelism
Reading list
Testing your evaluation code
Threshold selection
Versioning Tier-2 implementations
API reference
eval_toolkit.adversarial
eval_toolkit.analysis
eval_toolkit.artifacts
eval_toolkit.audit_citation_alignment
eval_toolkit.audit_sister_doc_concept_drift
eval_toolkit.audit_value_bindings
eval_toolkit.bootstrap
eval_toolkit.calibration
eval_toolkit.claims
eval_toolkit.config
eval_toolkit.docs
eval_toolkit.eda
eval_toolkit.embeddings
eval_toolkit.evidence
eval_toolkit.harness
eval_toolkit.leakage
eval_toolkit.loaders
eval_toolkit.losses
eval_toolkit.manifest
eval_toolkit.metric_specs
eval_toolkit.metrics
eval_toolkit.operating_points
eval_toolkit.paths
eval_toolkit.plotting
eval_toolkit.preprocessing
eval_toolkit.probes
eval_toolkit.protocols
Strict Tier-2 Protocols at v1.0
eval_toolkit.provenance
scorecard family — primary metric surface (v0.46+)
eval_toolkit.seeds
eval_toolkit.splits
eval_toolkit.stacking
sweep — unified text-transform enumeration (v0.47)
eval_toolkit.text_dedup
eval_toolkit.thresholds
v0.6.x → v0.7.x migration
v0.7.x → v0.8.0 migration
v0.8.x → v0.9.0 migration
Migrating to v0.46.0
Migrating to v0.47
Migrating to v0.48
Migrating to v0.49
Migrating to v0.50
Migrating to v0.51
Extending eval-toolkit
Schema Reference
Roadmap
Repo Strategy
Deprecation policy
Migration guides
Releasing eval-toolkit

GitHub
PyPI

Section Navigation

eval_toolkit.eda.audit_dataset
eval_toolkit.eda.class_balance
eval_toolkit.eda.length_quantiles
eval_toolkit.eda.summarize_split
eval_toolkit.eda.DataAudit
eval_toolkit.eda.SplitSummary
eval_toolkit.eda.Tokenizer

eval_toolkit.eda.analyze_obfuscation
eval_toolkit.eda.count_invisible_chars
eval_toolkit.eda.has_high_entropy_alnum_run
eval_toolkit.eda.has_rot13_marker
eval_toolkit.eda.is_leeted_token
eval_toolkit.eda.leetspeak_counts
eval_toolkit.eda.nfkc_changed
eval_toolkit.eda.nfkc_char_delta
eval_toolkit.eda.shannon_entropy
eval_toolkit.eda.ObfuscationProfile

eval_toolkit.eda.class_lexical_association
eval_toolkit.eda.competency_baselines
eval_toolkit.eda.default_tokenizer
eval_toolkit.eda.weighted_log_odds
eval_toolkit.eda.BaselineScore
eval_toolkit.eda.CompetencyResult
eval_toolkit.eda.LexicalAssociationResult
eval_toolkit.eda.StrTokenizer

eval_toolkit.eda.distribution_shift
eval_toolkit.eda.knn_purity
eval_toolkit.eda.maximum_mean_discrepancy
eval_toolkit.eda.median_bandwidth
eval_toolkit.eda.proxy_a_distance
eval_toolkit.eda.DistributionShiftResult
eval_toolkit.eda.KnnPurityResult
eval_toolkit.eda.MmdResult
eval_toolkit.eda.PadResult

eval_toolkit.eda.BASE64_ENTROPY_THRESHOLD
eval_toolkit.eda.DEFAULT_CHAR_NGRAM_RANGE
eval_toolkit.eda.DEFAULT_KNN_K
eval_toolkit.eda.DEFAULT_MAX_NEG_POS_RATIO
eval_toolkit.eda.DEFAULT_MIN_COUNT
eval_toolkit.eda.DEFAULT_MIN_NEG_POS_RATIO
eval_toolkit.eda.DEFAULT_MMD_PERMUTATIONS
eval_toolkit.eda.DEFAULT_PAD_C
eval_toolkit.eda.DEFAULT_PAD_FOLDS
eval_toolkit.eda.DEFAULT_PCT_OVER_CONTEXT_THRESHOLD
eval_toolkit.eda.DEFAULT_PRIOR_SCALE
eval_toolkit.eda.EDA_AUDIT_SCHEMA_VERSION
eval_toolkit.eda.HEX_ENTROPY_THRESHOLD

eval_toolkit.eda
eval_toolkit.eda.distribution_shift

eval_toolkit.eda.distribution_shift#

.. currentmodule:: eval_toolkit.eda

.. autofunction:: distribution_shift

previous

eval_toolkit.eda.StrTokenizer

next

eval_toolkit.eda.knn_purity

© Copyright 2026, Brandon Behring.

Created using Sphinx 9.1.0.

Built with the PyData Sphinx Theme 0.18.0.