Skip to main content

Usage

Quickstart (Windows / PowerShell)​

$root = "$env:USERPROFILE\Rust-Crates"
New-Item -Force -ItemType Directory $root, "$root\crates.io-index", "$root\mirror" | Out-Null

# First-time clone of the index
git clone https://github.com/rust-lang/crates.io-index "$root\crates.io-index"

# Build the CLIs
go build -o .\bin\download-crates.exe .\cmd\download-crates
go build -o .\bin\generate-sidecars.exe .\cmd\generate-sidecars

# Dry-run preflight — validates config, estimates work, downloads nothing
.\bin\download-crates.exe -index-dir "$root\crates.io-index" -out "$root\mirror" -concurrency 256 -dry-run -log-level info

# Real run, with metrics on :9090
.\bin\download-crates.exe -index-dir "$root\crates.io-index" -out "$root\mirror" -concurrency 256 -include-yanked -progress-interval 5s -listen :9090 -log-format json

Metrics: http://localhost:9090/metrics · Status: http://localhost:9090/api/status

The Python wrapper (Clone-Index.py)​

The wrapper is the simpler entry point — it clones/updates the index and invokes the downloader for you:

python Clone-Index.py --index-dir "A:\rust-lang\crates.io-index" --output-dir "A:\rust-lang\crates\" --include-yanked --logpath "A:\rust-lang"

Key flags: --threads (maps to -concurrency, default 128), --verify-existing (re-hash and verify existing files instead of trusting them), --bundle / --bundle-mode {only,add} / --bundle-size-gb (rolling archive output), --manifest (JSONL audit log path), --listen (metrics address, default :9090), --dry-run.

Several flags (--rate-limit, --resume, --verify) are kept only for backward compatibility and no longer do anything distinct — the Go downloader now owns retry/resume/verification behavior directly.

Airgap transport workflow​

CloneCratesio's manifest + bundle design exists specifically to support moving a verified crate set into an environment with no network access:

  1. Download with a manifest — every run emits manifest.jsonl with path, size, sha256, yanked, and timestamp for each crate.
  2. Bundle for transport (optional) — add -bundle -bundles-out /data/crates-bundles to roll files into .tar.zst archives, reducing inode count and copy time.
  3. Generate sidecars — generate-sidecars produces per-crate metadata for downstream tooling.
  4. Verify before transport: 
    jq -r 'select(.ok==true) | "\(.sha256)  \(.path)"' manifest.jsonl > checksums.txt
    sha256sum -c checksums.txt
  5. Move the mirror or bundle archives + manifest using robocopy (Windows) or rsync (Linux) to preserve timestamps and retry on transient errors.
  6. Re-verify offline — recompute hashes on arrival, spot-check a random sample, and confirm sidecar files match expected index metadata.

Performance notes​

  • Use NVMe storage; avoid mirroring onto network shares.
  • Consider disabling NTFS compression on the output directory for better throughput.
  • Resume is automatic — existing verified files are skipped via checksum/size checks, so re-running a mirror is cheap.