Skip to main content

CloneCratesio Overview

CloneCratesio is a Go command-line tool that mirrors crates.io — the Rust package registry — to local storage. It expands a local crates.io-index git checkout into downloaded .crate files (or rolling .tar.zst bundles), emits a JSONL audit manifest with per-file hashes, and exposes Prometheus/pprof endpoints while it runs.

It exists to answer one operator problem: "I need a complete, verifiable, offline copy of the Rust crate ecosystem, and I need to be able to prove it's correct later."

:::info Status CloneCratesio is paused at 85% complete. The manifest records stage = production and stability = stable, but a full build/test/release verification pass has not been executed since the last governance reconciliation (2026-07-08). The code and tests are believed to work — go test ./... is noted as passing — but this hasn't been re-verified end to end. :::

Who it's for​

A single operator who needs a durable, inspectable local mirror of crates.io — for offline Rust development, long-term archival, dataset generation, or moving a verified package set into an airgapped environment.

It is not a public mirror service. There's no multi-tenant access control or public-facing hosting story; it's built around one operator running it against their own storage.

The core concept​

CloneCratesio has three Go CLIs plus one Python wrapper, all operating over the same on-disk layout:

crates.io-index/     <-- git checkout of the crates.io index (source of truth for what exists)
crates/               <-- .crate files, mirrored 1:1 from the index (loose-file mode)
bundles/              <-- rolling .tar.zst archives (bundle mode, for airgap transport)
manifest.jsonl        <-- one JSON record per crate version: path, size, sha256, yanked, timestamp
  • download-crates walks the index and downloads crate files with high-concurrency HTTP/2, retries, and an incremental "verify existing" mode so re-runs don't redo work.
  • generate-sidecars produces per-crate .crate.json sidecar metadata (or an aggregated JSONL stream) for downstream tooling.
  • extract-bundles unpacks rolling .tar.zst bundles back into a shard layout.
  • Clone-Index.py is a thin wrapper that clones/updates the index and invokes the Go downloader with sensible defaults — the easiest entry point on Windows.

While a run is active, download-crates exposes /metrics (Prometheus), /api/status (JSON), and /debug/pprof/ — this is a long-running, observable process, not a fire-and-forget script.

Why Go (and why this shape)​

High-throughput concurrent downloading against a registry with hundreds of thousands of crate versions benefits from Go's concurrency primitives and compiled performance — this is explicitly a "high-performance" and "high-throughput" tool per its own tags, and the manifest/bundle design (JSONL audit log, rolling .tar.zst, sidecar metadata) is built around being restart-friendly and airgap-transportable rather than being a quick one-shot script.