Recipe: pod-Python reproducibility#

Pattern: pin the pod-side CPython interpreter via pod.python_version so successive pod runs use the exact same interpreter, closing the third leg of the reproducibility tripod (git SHA + lockfile hash + interpreter version).

Why this is a recipe, not a schema feature#

pod.python_version is a schema field — that part is upstream. What makes the surrounding workflow a recipe is the integration story: how you reconcile this YAML pin with requires-python in pyproject, which patch level vs minor pin you choose for which use case, and how you couple the pin with manifest provenance for paper-grade reproducibility claims. Those decisions are consumer-domain and project-specific.

A YAML that declares requires-python = ">=3.13" in pyproject and has no .python-version file can resolve to a different CPython minor version on successive pod runs (e.g. 3.13 today, 3.14 next month). Reproducibility claims that hinge on git SHA + HF revision + uv.lock get a third moving part nobody declared. The fix is below.

Pattern (YAML)#

pod:
  image: runpod/pytorch:2.4.0
  datacenters: [EU-RO-1]
  gpu_order: ["NVIDIA H100 80GB HBM3"]
  python_version: "3.13.5"   # exact pin (recommended for canonical sweeps)

The orchestrator auto-injects a preflight step:

uv python install 3.13.5 \
  && cd /workspace/repo \
  && uv python pin 3.13.5

uv python install is idempotent (uv-managed cache under ~/.local/share/uv/python/); subsequent runs of the same pod re-use the cached interpreter. uv python pin writes a .python-version file into the staged project dir so the user’s run-body uv sync honors the pin.

When to use minor vs patch pinning#

Pin	Use case	Trade-off
`"3.13.5"` (exact patch)	canonical sweeps, paper-publication runs	strongest reproducibility; install cost ~30s first time
`"3.13"` (minor only)	dev/smoke loops where patch-level drift is acceptable	uv resolves the latest available 3.13.x; slightly looser pin

Why not just rely on `requires-python` in pyproject?#

requires-python = ">=3.13" is a constraint, not a pin — uv picks the highest interpreter satisfying it. If you want canonical reproducibility, pair the pyproject constraint with pod.python_version in the runpod-deploy config. The YAML pin is the deploy-domain declaration of “this is the interpreter we are choosing”; the pyproject constraint is the package-domain declaration of “this is the floor.”

Coupling with manifest provenance#

The pulled run manifest (runpod_deploy_pull_manifest.json) records the local git SHA and the uv.lock hash. Adding pod.python_version to the YAML means the third leg of the reproducibility tripod is also explicit: declared in the config, pinned via the auto-injected preflight, surfaced in the manifest via the name / run_id echo of the YAML’s resolved values.

Failure mode#

If uv python install fails (network blip, no such version, uv missing from the base image), the run aborts at the auto-injected preflight step — before any user preflight commands or run-body execute. The operator pays ~30s of pod time to surface a fixable config issue, not a multi-minute mid-run failure with partial artifacts.

What lives where#

Concern	Owner
Honoring `pod.python_version` and injecting the install/pin preflight	`runpod-deploy run` (`orchestrator._build_python_pin_preflight`)
Choosing the exact patch version (`"3.13.5"`) vs minor (`"3.13"`)	Your project (canonical sweeps vs dev loops)
Setting `requires-python` floor in `pyproject.toml`	Your project (package-domain constraint)
Caching the pulled interpreter (`~/.local/share/uv/python/`)	`uv` (idempotent across pod re-runs)
Asserting all shards in a sweep used the same interpreter	Your post-run analysis (read `python_version` echo from each manifest)

Anti-pattern to avoid#

Do not rely on requires-python = ">=3.13" alone for canonical reproducibility. It’s a floor, not a pin — uv picks the highest available interpreter satisfying it, which can drift between pod provisioning runs as uv python install’s available-versions list updates.

Do not commit a .python-version file in the project root that disagrees with pod.python_version. The auto-injected preflight writes .python-version into the staged project dir using the YAML value, but a pre-existing committed file can confuse reviewers about which is authoritative. Either commit .python-version matching the YAML, or omit it and let the orchestrator manage it.