ytdlp-archiver

A resilient, cache-first YouTube channel and playlist archiver written in modern Rust.

ytdlp-archiver scans YouTube channels, generates deterministic local queue and plan files, maintains human- and machine-readable report/ directories, and coordinates robust downloads with automatic retries, profile rotations, live rate and ETA extrapolation, and duplicate file management.

---

Features

• Storage Pools (--output, repeatable): Span multiple directories as one contiguous archive. Existing channels are found wherever they live; new channels land on the output with the most free space; HOST:DIR specs fail fast unless mounted over sshfs from that host.
• Cache-First Architecture: Avoids redundant network queries by leveraging local snapshot, plan, and media scan caches.
• Resilient Downloads: Rotates extractor arguments and client impersonation profiles (chrome, safari, firefox) upon transient failures, and backs off on HTTP 403 rate limits.
• Proxy Rotation & Cookies: Supply authenticated proxies via YTDLP_ARCHIVER_PROXIES / YTDLP_ARCHIVER_PROXY_FILE; each worker holds a sticky proxy that rotates automatically on anti-bot challenges. --cookies, --cookies-from-browser, and --yt-dlp-arg cover the rest.
• Opt-In Parallel Channels: --jobs N downloads N channels concurrently with one sticky proxy per worker; rate limiting stays polite per IP. Off by default.
• Real-Time Progress & ETA: Live visual progress bars with elapsed time, processing rate (items/sec and sec/item), and extrapolated completion estimates.
• Orphan Video Policies: Flexible handling for uncurated channel uploads outside playlists via --orphans [skip|first|last].
• Zero-Loss Recovery: Automatically detects and recovers matching partial or misplaced media from the user's home directory.
• Recommendation Harvester: Grabs YouTube watch-next recommendations via InnerTube API (ytdlp-archiver recommend) and saves them alongside video archives.
• Exact Duplicate Deduplication: Groups identical media files across channels by size and SHA-256 hash, generating interactive deletion plans (ytdlp-archiver duplicates).
• Complete Inspection & Reports: Inspect completion matrices across channels with ytdlp-archiver report [status|incomplete|complete|open|print].
• Tiered URL Summaries: summarize turns any video, playlist, or channel URL into a succinct digest — metadata first, then opt in to top comments (--comments), subtitles (--transcript), and related videos (--recommendations). Piping switches to JSON automatically, and --download hands the target to the archiver.
• Actual + Projected Disk Usage: space renders a read-only ncdu-style tree of what every channel, playlist, and video occupies, and projects what the remaining plan would weigh from a locally calibrated rate. JSON when piped or with --json; --tui opens an interactive vim-keyed browser whose p/x/d marks feed prune for prioritized downloads, exclusions, and safe deletes.
• Efficient Media Recode: recode compares every media file against what YouTube still offers; re-acquires when a more efficient codec (AV1 > VP9 > H.264) or a smaller download is available, verifying each replacement before atomically swapping it in. Dry-run unless --apply; --encode adds local SVT-AV1 re-encoding for files with no better stream.
• Canonical Layout & Migration: migrate backfills channel identity, renames non-canonical directories, and merges duplicates with verified dedup.
• Shell Completions: Native completions generation for bash, elvish, fish, powershell, and zsh.

---

Prerequisites

• yt-dlp must be installed and available in your PATH.
• Rust (1.85+ recommended, 2024 edition supported).

---

Installation

Clone the repository and install the binary with Cargo:

git clone https://github.com/your-repo/ytdlp-archiver.git
cd ytdlp-archiver
cargo install --path .

The ytdlp-archiver executable will be placed in ~/.cargo/bin.

---

Quick Start

1. Storage Pools: --output

Every subcommand takes one or more --output directories that together form a storage pool. Plain directories are created on demand; HOST:DIR specs are verified against /proc/mounts and must be sshfs mounts from that host — the command exits 2 before touching anything otherwise.

# One output directory
ytdlp-archiver run --output ~/Videos/YouTube https://www.youtube.com/@example

# Two outputs: new channels land on whichever has more free space.
# 'media:' must be an sshfs mount from host 'media' or the run aborts.
ytdlp-archiver run \
    --output ~/Videos/YouTube \
    --output media:/mnt/scratch/videos \
    channels.txt --smallest-first

Pool rules:

• Channels are the allocation unit — a channel directory is never split across filesystems.
• Existing channels are discovered on any output; only new channels are allocated (most free bytes wins).
• report/ and duplicates.txt live on the first output that has a report/ directory (else the first output). Skip lists are unioned across outputs.

Proxies, Cookies, and Concurrency (optional)

Proxies come from the environment so credentials never appear in ps output:

export YTDLP_ARCHIVER_PROXIES='http://user:pass@10.0.0.1:8080
http://user:pass@10.0.0.2:8080'
# alternatively: export YTDLP_ARCHIVER_PROXY_FILE=~/.config/ytdlp-archiver/proxies.txt

• Each download worker keeps one sticky proxy. On a YouTube anti-bot or sign-in challenge the worker benches that proxy for 15 minutes, rotates to the next healthy one, and retries the same video. When none remain, the run aborts and suggests cookies or --yt-dlp-arg fixes.
• Proxy URLs are always redacted (***:***@) in logs and command displays.
• Cookies: --cookies cookies.txt or --cookies-from-browser firefox.
• Escape hatch: --yt-dlp-arg ARG (repeatable) appends raw flags after the baseline so it can override anything, e.g. --yt-dlp-arg '--extractor-args=youtube:player_client=web_safari'.

Run channels in parallel (opt-in; per-IP sleep intervals stay exactly the same, so this is most useful with several proxies):

YTDLP_ARCHIVER_PROXIES='...' ytdlp-archiver run \
    --output ~/Videos/YouTube channels.txt --jobs 2

• --jobs N (default 1 = fully sequential) processes N channels at a time; a channel is never split across workers. N is clamped to the number of proxies with a warning; running in parallel without proxies warns too.
• On the first channel error no new channels are started, in-flight channels finish, and the workflow exits 1 naming the failing URL.

2. Archive a Single Channel

Download all playlists and uncurated videos for a channel:

ytdlp-archiver run --output ~/Videos/YouTube https://www.youtube.com/@example

3. Archive Multiple Channels from a URL List

Create a text file containing channel URLs (one per line, # comments allowed):

# Educational channels
https://www.youtube.com/@3blue1brown
https://www.youtube.com/@mitocw

Run the archiver over the list, ordering batches from smallest to largest channel:

ytdlp-archiver run --output ~/Videos/YouTube channels.txt --smallest-first

4. Dry Run / Analysis Mode

Analyze what would be downloaded without invoking yt-dlp download transfers:

ytdlp-archiver run --output ~/Videos/YouTube channels.txt --analysis

5. Incremental / Since Layer Archiving

Only fetch items uploaded after the newest downloaded item in each playlist:

ytdlp-archiver run --output ~/Videos/YouTube channels.txt --since-layer

Or specify a strict cutoff date (YYYY-MM-DD or YYYYMMDD):

ytdlp-archiver run --output ~/Videos/YouTube channels.txt --since 2025-01-01

6. Find Anything: search

One read-only pass across channel directories, snapshots, report files, plan items, media filenames, recommendations, and every skip/cached/failure/archive list. URL queries flip direction and find every disk reference to that id.

# Find a channel by half-remembered name across a pool
ytdlp-archiver search 'two minute papers' --output ~/Videos/YouTube/watched --output ~/Videos/YouTube/unwatched

# Restrict hit kinds
ytdlp-archiver search 'transistor' --output ~/Videos/YouTube --video --metadata

# Deep-scan *.info.json descriptions and comment text (opt-in, expensive)
ytdlp-archiver search 'brachistochrone' --output ~/Videos/YouTube --comments

# Machine-readable output
ytdlp-archiver search dQw4w9WgXcQ --output ~/Videos/YouTube --json

7. Inspect One Entity: info

Give info a channel URL (or handle, or directory name), a playlist URL, or a video URL/id, and it aggregates the full local state: identity, disk usage, per-playlist breakdowns with plan/media counts, skip/cached/failure/archive tallies, media + sidecar files for videos, and — with --web — canonical identity fetched from YouTube (cached).

ytdlp-archiver info https://www.youtube.com/@example --output ~/Videos/YouTube
ytdlp-archiver info Two_Minute_Papers_-_Videos --output ~/Videos/YouTube --json
ytdlp-archiver info dQw4w9WgXcQ --output ~/Videos/YouTube --web

8. Quick URL Digest: summarize

Someone sends you a video, playlist, or channel link and you just want to know what it is. summarize prints a compact digest — title, channel, date, duration, views, likes, description, and (for collections) the item list:

# Tier 1: metadata only (works on videos, playlists, and channels)
ytdlp-archiver summarize https://www.youtube.com/watch?v=dQw4w9WgXcQ

# Tier 2: add top comments; tier 3: add the transcript; tier 5: related videos
ytdlp-archiver summarize https://www.youtube.com/watch?v=dQw4w9WgXcQ --comments
ytdlp-archiver summarize https://www.youtube.com/watch?v=dQw4w9WgXcQ --transcript --lang en
ytdlp-archiver summarize https://www.youtube.com/watch?v=dQw4w9WgXcQ --recommendations --max-recommendations 5

# Tiers apply per item on collections; --limit caps how many are detailed
ytdlp-archiver summarize https://www.youtube.com/@example --comments --limit 3

# Piped or --json emits structured JSON; --text forces human output
ytdlp-archiver summarize https://www.youtube.com/watch?v=dQw4w9WgXcQ --json | jq .title

Summary query results stay cached indefinitely to preserve archive data if YouTube access disappears. Pass --refresh to explicitly refetch and replace cached results.

Tier 4 adds --download, which hands the resolved target to the normal archiver workflow (and therefore requires the usual --output pool):

ytdlp-archiver summarize https://www.youtube.com/watch?v=dQw4w9WgXcQ --download --output ~/Videos/YouTube

9. See What the Archive Weighs: space

A read-only ncdu-style hierarchy of actual bytes per channel, playlist, and video, plus a projection of what the remaining plan would consume. The estimate is calibrated from media already downloaded; --mb-per-min overrides it.

# Actual + projected, two levels deep
ytdlp-archiver space --output ~/Videos/YouTube

# Drill into playlists and videos, ordered by what is left
ytdlp-archiver space --output ~/Videos/YouTube --depth 4 --sort remaining

# Project a channel list offline (add --refresh to fetch missing listings)
ytdlp-archiver space --output ~/Videos/YouTube --list channels.txt --json | jq .totals

# Interactive ncdu-style browser (vim keys; marks + prune)
ytdlp-archiver space --output ~/Videos/YouTube --tui

The TUI moves like ncdu: j/k (or arrows) to move, h/l to collapse or expand, e/c to expand or collapse everything, E/C to expand or collapse channels only (playlists stay as they are), gg/G for top/bottom, Ctrl-d/Ctrl-u to page, Enter to toggle, s to cycle child ordering, f to cycle a download-state filter (all → downloaded → not downloaded → partial → complete → incomplete), m to cycle a mark-state filter (all → not excluded → marked → unmarked → excluded), r to rescan the skip lists and the selection manifest after an external change, ? for help, q to quit. The bar and percentage show completion by bytes; each row reads on disk / projected total, with done · left item counts after the name.

While browsing you can also mark nodes: p prioritize (download first), x exclude (never download), d delete (drop the downloads on prune), with P/X/D applying the same mark to a whole subtree. The same key toggles off — including marks that came from existing skip files, which removes those entries. Quitting writes the marks to .space-selection.jsonl in the output root; the browser itself never touches media.

Curation workflow. space --tui is the picker: walk channels → playlists → videos and decide what belongs in the archive. Marks take effect on the estimates immediately — excluding a node shrinks its left/projected numbers on the spot (the manifest is overlaid on the scan, no disk writes) — so you can watch the projected total fall as you thin the plan. Use f and m to narrow the tree to just what's still unreviewed (m unmarked), p/x/d to decide, and P/X/D to apply the same call to a whole subtree at once. When the plan looks right, q writes the manifest and prune turns the marks into action. Nothing is deleted until you run prune --apply.

Apply the marks with prune (dry-run by default):

# What would happen?
ytdlp-archiver prune --output ~/Videos/YouTube

# Write the new input list (prioritized first, excluded omitted)
ytdlp-archiver prune --output ~/Videos/YouTube --emit-list next.txt

# Do it: deletes media + sidecars, updates skip/priority lists
ytdlp-archiver prune --output ~/Videos/YouTube --apply

Then reclaim space on existing downloads - recode compares each file against the formats YouTube still offers and plans replacements when a more efficient codec (AV1 > VP9 > H.264) or a smaller file is available. Every replacement is verified (codec, height, duration, size) before it atomically swaps in, keeping the original filename and extension:

# What would be replaced, and by what?
ytdlp-archiver recode --output ~/Videos/YouTube

# Machine-readable plan
ytdlp-archiver recode --output ~/Videos/YouTube --json | jq .plan.totals

# Execute the replacements
ytdlp-archiver recode --output ~/Videos/YouTube --apply

# Also allow local SVT-AV1 encoding when no better stream exists
ytdlp-archiver recode --output ~/Videos/YouTube --apply --encode

10. Resolve a Name to a URL (offline)

Fuzzy channel-name lookup over local identity sources — snapshots, report indexes, *.info.json sidecars, directory names. Offline, fast:

ytdlp-archiver resolve 'two minute papers' --output ~/Videos/YouTube --json

11. Migrate an Imported Library

For archives created by older tool versions, migrate backfills snapshots, renames non-canonical directories, and merges duplicate channel dirs (dry-run by default; --apply executes):

ytdlp-archiver migrate --output ~/Videos/YouTube
ytdlp-archiver migrate --output ~/Videos/YouTube --apply --adopt

---

Subcommands Overview

Subcommand	Description
run	Scan and download a channel URL or list of channel URLs into the --output pool.
playlist	Spot-download a single playlist resolving the parent channel from reports.
report	Inspect, view, or open local channel reports (status, incomplete, complete, open, print).
duplicates	Detect, inspect, and safely clean up identical media files (list, show, cleanup).
recommend	Fetch and query InnerTube recommendations (fetch, fetch-all, show, list, urls, clear).
search	Fuzzy-search all local state (dirs, reports, plans, media, lists) with scope flags (--channel, --playlist, --video, --media, --reference, --metadata, --comments, --all) and optional --web.
info	Inspect one entity in depth — channel URL/dirname, playlist URL, or video URL/id — with counts, per-playlist breakdowns, and optional --web identity enrichment.
summarize	Fetch a succinct digest of a video, playlist, or channel URL; tier up with --comments, --transcript, --recommendations, and --download. JSON when piped or with --json.
space	Read-only disk-usage hierarchy: actual bytes per channel/playlist/video plus projected size of the remaining plan. JSON when piped or with --json; --tui for an interactive browser.
prune	Apply space --tui selection marks: prioritize order, skip-file exclusions, and deletes (dry-run unless --apply); --emit-list writes the next input file.
recode	Re-acquire media when a more efficient codec or smaller download exists; dry-run unless --apply; --encode adds SVT-AV1 re-encoding.
resolve	Offline fuzzy channel-name -> name/URL/id lookup.
migrate	Canonicalize directory layout, backfill identity, merge duplicates (--apply, --adopt).
cache	Clear analysis, snapshot, or media scan caches (clear).
repair-home-downloads	Relocate misplaced media downloads from ~ into their target channel directories.
completions	Output shell completion scripts for bash, zsh, fish, etc.
docs	Built-in CLI reference and maintainer architectural documentation (--maintainer).

---

Shell Completions

Generate shell completions and install them for your active shell:

# Bash
ytdlp-archiver completions bash > ~/.local/share/bash-completion/completions/ytdlp-archiver

# Zsh
ytdlp-archiver completions zsh > ~/.zfunc/_ytdlp-archiver

# Fish
ytdlp-archiver completions fish > ~/.config/fish/completions/ytdlp-archiver.fish

---

Documentation & Architecture

ytdlp-archiver includes comprehensive in-tree documentation:

• Run ytdlp-archiver docs for the full command guide and storage-pool reference.
• Run ytdlp-archiver docs --maintainer for detailed architectural principles, caching methodology, and maintenance guidelines.
• Generate HTML API documentation with cargo doc --workspace --no-deps --open.

---

License

This project is licensed under the MIT License.