Matrix Surgeon

matrix-surgeon is a careful command-line toolbox for investigating and repairing a Synapse server.

It is meant for administrators who want to see the exact SQL, HTTP requests, and SSH commands before anything changes.

It is built around three safety defaults:

• dry-run by default
• always print the generated SQL or HTTP request first
• explain what an operation is trying to prove or change

The tool mixes three control surfaces:

• direct SQL probes and SQL mutations
• Synapse admin API requests
• Synapse client API requests
• composed safety-first workflows built from those primitives

It is written in Rust with a clean architecture and zero external system runtime dependencies (distributed as a standalone static binary).

In plain terms:

• use sql when you want the tool to write the query for you
• use admin and client when you want to see or send one concrete API request
• use workflow when you want the tool to walk you through a real maintenance task step by step

Documentation

• docs/cleanup-runbook.md — emergency and routine cleanup ladder (full disk → healthy)
• docs/emergency-disk-full-runbook.md — step-by-step recovery when the disk is at 100% and Postgres is struggling
• docs/postmortem-2026-10-personal-disk-full.md — post-mortem of the personal-server disk-full incident
• docs/automation.md — exit-code contract, canonical sample crontab, report → arm → purge rhythm
• docs/runbook.md — operational runbook: bootstrap, confidence passes, deep maintenance
• docs/maintenance-stories.md — worked maintenance scenarios
• backup-considerations-proposal.md — backup architecture options and constraints
• docs/auto-compressor-automation-sharing.md — running Synapse state compression
• TUTORIAL.md and CHEATSHEET.md — guided walkthrough and one-page reference

Configuration

Configuration is read only from:

~/.config/matrix-surgeon/config.toml

There is no JSON, legacy password-file, environment-variable, or alternate-path fallback. Create one profile table per homeserver:

[profiles.public]
homeserver_url = "https://matrix.example.net"
base_domain = "example.net"
user_id = "@admin:example.net"
access_token = "syt_..."
ssh_host = "db.example.net"          # required for SSH SQL transport + disk checks
# media_store_path = "/matrix/synapse/storage/media-store"
# Optional strictly protected accounts (never bulk-deactivated or destroyed):
# protected_user_ids = ["@admin:example.net", "@ops:example.net"]
# protected_user_prefixes = ["@jmpchat_", "@telegram_", "@whatsapp_"]
# Optional bot accounts for bridge intelligence:
# bot_user_ids = ["@helper:example.net"]
# bot_user_prefixes = ["@mybot_"]

[profiles.public.retention]
max_disk_usage_percentage = 85       # or max_disk_usage_gigabytes = 120.0
expected_keep_rooms = 10
media_retention = "180 days"
big_media_retention = "365 days"
big_media_size_bytes = 1000000
medium_media_retention = "270 days"
medium_media_size_bytes = 100000
remote_media_retention = "30 days"
big_room_message_retention = "365 days"
small_room_message_retention = "180 days"
big_room_threshold = 100

Protection floors (protect_newer_than_*), disk-pressure tiers ([profiles.<name>.retention.pressure]), protected accounts, and bot lists are all optional — described in their sections below. matrix-surgeon config example prints the complete annotated schema generated from the reader's own parser (so it cannot drift), and matrix-surgeon config onboard writes an interactive starter file.

Required profile fields are homeserver_url, base_domain, and access_token. user_id is required by profile token-refresh unless supplied with --user-id. ssh_host is required for SSH SQL transport and remote disk checks.

List profiles and inspect one:

matrix-surgeon profile list
matrix-surgeon profile show public

Check configured token against Synapse:

matrix-surgeon profile token-check public --execute

Request a new token through Matrix password login. Command stays dry-run unless --execute is supplied. Password is prompted without echo; use --password-stdin for automation:

matrix-surgeon profile token-refresh public --execute
matrix-surgeon profile token-refresh public --user-id @admin:example.net --execute

After successful login, the existing TOML file is backed up beside itself as config.toml.bak-<UTC timestamp> (only the 3 newest backups are kept; config, backups, and temp files are written 0600), then replaced atomically. Comments in the file are preserved — only the access_token line changes.

Bootstrap From The CLI

Generate a complete example using the same profile schema used by the reader:

matrix-surgeon config example
matrix-surgeon config example --output /tmp/matrix-surgeon-example.toml

Example output is printed by default. Writing an existing output requires --force; generated tokens and hosts are mock values.

Onboarding asks for missing homeserver, profile, and Matrix account values. It is dry-run by default, so it prints the config shape without contacting Synapse or writing credentials:

matrix-surgeon config onboard \
  --homeserver-url https://matrix.example.net \
  --base-domain example.net \
  --profile public \
  --username admin

Run with --execute to log in through Matrix password authentication and write ~/.config/matrix-surgeon/config.toml. Password is prompted without echo; use --password-stdin for automation.

matrix-surgeon --execute config onboard \
  --homeserver-url https://matrix.example.net \
  --base-domain example.net \
  --profile public \
  --username admin

Onboarding supports multiple profiles in one run. After each profile is logged in and written, it asks Add another profile? [y/N]; answering y prompts for the next homeserver, account, and password. Each new profile is merged into the existing config alongside whatever is already there, and every write creates a timestamped backup. --force is only needed to replace a profile name that already exists. Dry-run previews redact all access tokens.

Update only one account's access token without touching the rest of the config:

matrix-surgeon profile token-refresh public --execute
matrix-surgeon profile token-refresh private --execute

token-refresh re-authenticates that single profile and replaces only its access_token in the TOML, keeping other profiles and settings intact.

Current Assumptions

• The admin endpoint is {homeserver_url}/_synapse/admin.
• SQL can run either through the local runner transport or directly over SSH using the profile's ssh_host
• The default non-interactive SQL paths are:
  • local runner: run-remotely matrix /matrix/postgres/bin/cli-non-interactive
  • SSH runner: /matrix/postgres/bin/cli-non-interactive
  • SSH shell: /matrix/postgres/bin/cli

Quick Start

List available profiles:

matrix-surgeon profile list

Inspect a profile:

matrix-surgeon profile show public

List the SQL catalog:

matrix-surgeon sql catalog

Preview a probe:

matrix-surgeon sql show ghost-users --profile public

Find delinquent rooms with local members but no local admin-level user:

matrix-surgeon sql show delinquent-rooms --profile public

Inspect the effective power levels in one room:

matrix-surgeon sql show room-member-power-levels --profile public --param room_id='!roomid:example.net'

Print only the rendered SQL for side-car use:

matrix-surgeon sql print inactive-users --profile public --param months=9

Execute a probe for real:

matrix-surgeon --execute sql run ghost-users --profile public

Open the remote SQL shell over SSH:

matrix-surgeon sql shell public

Preview a room deletion request:

matrix-surgeon admin rooms delete public '!roomid:example.net'

Execute that room deletion request:

matrix-surgeon --execute admin rooms delete public '!roomid:example.net'

Send a generic admin API request:

matrix-surgeon admin request public GET '/v2/users?limit=10'

Preview the bridge-bot cleanup workflow:

matrix-surgeon workflow purge-bridge-bot public '@jmpchatbot:example.net'

Report delinquent rooms from live room inventory and state:

matrix-surgeon --execute workflow triage-delinquent-rooms public

Report stale bridge ghosts and their one-member rooms before mutating anything:

matrix-surgeon --execute workflow cleanup-bridge-ghosts public '@jmpchat_%'

Inspect one room and decide whether it should be recovered, shut down, or replaced:

matrix-surgeon --execute workflow resolve-delinquent-room public '!roomid:example.net'

Inspect a delinquent room and prepare a local admin recovery:

matrix-surgeon workflow recover-room-admin public '!roomid:example.net' '@ops:example.net'

Fetch live room state and membership before applying that repair:

matrix-surgeon --execute workflow recover-room-admin public '!roomid:example.net' '@ops:example.net'

Actually join and promote that local user:

matrix-surgeon --execute workflow recover-room-admin public '!roomid:example.net' '@ops:example.net' --join-first --apply

Inspect stale bot-signup candidates without deactivating them yet:

matrix-surgeon --execute workflow deactivate-inactive-users public --name-prefix '@bot_' --never-seen-only

Actually deactivate the filtered candidates:

matrix-surgeon --execute workflow deactivate-inactive-users public --name-prefix '@bot_' --never-seen-only --apply

Preview a room upgrade request:

matrix-surgeon client rooms upgrade public '!roomid:example.net' 11

Inspect a room first and get a concrete upgrade checklist before sending the upgrade:

matrix-surgeon --execute workflow guide-room-upgrade public '!roomid:example.net' 11

Run a broad read-only health sweep:

matrix-surgeon workflow integrity-sweep public

Preview the retention workflow from the profile's retention block:

matrix-surgeon workflow retention-purge public

Preview only room classification, with no mutations:

matrix-surgeon workflow retention-purge public --mode inventory-only

Explain the retention terms and modes from the CLI:

matrix-surgeon workflow retention-explain

Target only a known subset of rooms during a history pass:

matrix-surgeon workflow retention-purge public --mode history-only --include-room '!abc:example.net' --include-room '!def:example.net'

Print shell completion setup for bash or zsh:

matrix-surgeon completion bash
matrix-surgeon completion zsh

Run ad hoc SQL from a file:

matrix-surgeon sql raw --profile public --file scratch.sql

Command Families

• profile: inspect configured profiles
• sql: inspect built-in SQL probes, print SQL for manual use, run SQL, or open an SSH SQL shell
• admin: inspect or send one Synapse admin API request at a time
• client: inspect or send one Matrix client API request at a time
• workflow: guided multi-step maintenance tasks such as delinquent-room triage, bridge cleanup, retention cleanup, integrity sweeps, and sliding sync audits
• completion: emit bash and zsh completion scripts backed by the CLI's own completion engine

Help, Search, and Smart Suggestions

• matrix-surgeon help prints a rich command overview with descriptions for every family and subcommand.
• matrix-surgeon help search <query> (or shorthand matrix-surgeon search <query>) fuzzy-searches the entire CLI surface — command paths, aliases, flags, help text — plus the full SQL recipe catalog (titles, explanations, categories). Add --json for machine-readable output.
• Invalid commands and typos trigger deep-graph "Did you mean?" suggestions that reach across the whole command hierarchy and into SQL recipes (e.g. matrix-surgeon retenton-purge suggests workflow retention-purge). Typo'd flags are corrected similarly.

Safety Model

• Commands only execute when --execute is present.
• Two Dry-Run Modes:
  • --dry-run plan (default): 100% offline step description, zero remote network or SSH contact.
  • --dry-run live: Read-only live inspection contacting remote systems over SSH (disk checks via df -P /) and Synapse API (room inventory + SQL media estimates), computing per-stage gating verdicts without performing any mutations.
• Scan Intensities (--scan):
  • strict: fast & safe, queries disk and inventory, skips heavy media table scans.
  • sample (default): exact database-level volume estimates from media repositories.
  • full: exhaustive scans with unbounded detail listings.
• JSON Output (--json): Emits structured JSON (raw, no ANSI color) for automation and inspection.
• For reporting and inspection workflows, --execute means "apply live mutations now".
• When a workflow also has --apply, think of it like this:
  • no flags: show me the offline plan (--dry-run plan)
  • --dry-run live: query live systems, estimate disk/rooms/media impact, and show gating verdicts
  • --execute: perform live inspection/execution
  • --execute --apply: do the risky mutation I just reviewed
• Mutating SQL is labeled as dangerous in the catalog.
• HTTP commands print method, URL, and JSON body before execution.
• SQL commands print the SQL text and the shell command used to run it.
• Named SQL recipes can take repeated --param name=value arguments.
• Some high-risk workflows use a second safety gate: --execute for live inspection and --apply for the actual mutation step.

Gate Matrix

Surface	Mutates	Required flags	Cost / lock
sql show, probes, --diagnose	no	--execute to run	read-only
sql run recipe (safe write)	yes	--execute --apply	per recipe
sql run recipe (dangerous write)	yes	--execute --apply --dangerous	same gates as the guided workflow
sql raw --sql/--file --execute	yes	--execute + your own review	recorded in the ledger with --note
retention-purge --mode walk	yes	--execute --apply	chunked; disk re-checked after every slice
retention-purge --older-than ...	yes	--execute --apply (--confirm when under 1 year)	strict cutoff; floors still enforced
db-maintenance --vacuum	yes	--execute --apply	VACUUM (ANALYZE); no reader locks
db-maintenance --vacuum-full T	yes	--execute --apply --dangerous --confirm	ACCESS EXCLUSIVE; needs temp headroom
purge-stale-devices (API)	yes	--execute --apply --dangerous	logs those devices out
purge-stale-devices --queue-only	yes	--execute --apply --dangerous	prunes queued messages only
delete-user --level deep/nuclear	yes	--execute --apply --dangerous --confirm	irreversible; restore from backup
audit-sliding-sync --fix	yes	--fix --execute --apply	resets forgotten rooms + connections
admin media delete-local	yes	--execute (--confirm when under 1 year)	avatars protected unless --force

Irreversible rows (row deletes, --vacuum-full, delete-user) need a current pg_dump before you run them — see backup-considerations-proposal.md.

Legacy Coverage

The first pass directly covers machinery illustrated in legacy/:

• admin user listing
• user deactivate / update
• user detail and membership inspection
• room delete
• room delete status
• room replacement / broken-room migration inputs
• room membership and state inspection
• admin room join helper
• make room admin
• client room upgrade requests
• forward extremities inspection and deletion
• history purge
• media purge operations
• direct SQL probes and cleanup recipes from the notes
• a first composed bridge-bot cleanup workflow
• a first staged retention workflow driven by the profile's retention policy
• delinquent-room discovery and admin-recovery workflows
• delinquent-room triage reporting from live room state
• guided delinquent-room shutdown or replacement-room decision support
• stale account review and controlled deactivation workflows
• bridge-ghost reporting and optional room/user cleanup workflows
• guided room-upgrade inspection with power and version checks
• bundled read-only integrity sweep reporting
• sliding sync conflict auditing and remediation workflow
• shell completion generation for bash and zsh

The retention workflow now covers the main legacy stages:

• room inventory and empty-room classification
• SSH disk usage checks between stages
• remote media cache purge
• big / medium / general local media deletion
• small-room and big-room history purge passes

Each step is still printed before execution so the tool can be used as an operator side-car rather than a black box.

Integrity Guide

For the broader database and room-integrity jobs, these are the most concrete entry points:

• Find rooms that look abandoned: sql show delinquent-rooms --profile <profile> matrix-surgeon --execute workflow triage-delinquent-rooms <profile>
• Look closely at one room before taking action: sql show room-member-power-levels --profile <profile> --param room_id='!room:example.net' matrix-surgeon --execute admin rooms members <profile> '!room:example.net' matrix-surgeon --execute admin rooms state <profile> '!room:example.net'
• Try the safest repair first when local users are still there: matrix-surgeon --execute workflow recover-room-admin <profile> '!room:example.net' '@user:example.net'
• If recovery is not the right answer, inspect shutdown vs replacement-room behavior: matrix-surgeon --execute workflow resolve-delinquent-room <profile> '!room:example.net' then add --decision shutdown or --decision migrate and only add --apply when you are sure
• Find likely stale signups or bot accounts: sql show inactive-never-sent-local-users --profile <profile> matrix-surgeon --execute workflow deactivate-inactive-users <profile> --name-prefix '@bot_' --never-seen-only
• Find stale bridge ghosts and rooms that only contain one such user: sql show ghost-users --profile <profile> sql show single-ghost-member-rooms --profile <profile> sql show single-prefix-member-rooms --profile <profile> --param user_like='@jmpchat_%' matrix-surgeon --execute workflow cleanup-bridge-ghosts <profile> '@jmpchat_%'
• Find empty rooms: sql show empty-rooms --profile <profile> then use workflow retention-purge <profile> --mode rooms-only or a room delete request
• Find rooms that are still on an older room version: sql show rooms-not-version --profile <profile> --param target_version=11 matrix-surgeon --execute workflow guide-room-upgrade <profile> '!room:example.net' 11
• Run one broad read-only health pass: matrix-surgeon --execute workflow integrity-sweep <profile>
• Check specific database consistency problems one by one: sql show orphaned-events --profile <profile> sql show orphaned-memberships --profile <profile> sql show orphaned-state-events --profile <profile> sql show bad-user-directory --profile <profile> sql show missing-user-directory --profile <profile> sql show forward-extremities --profile <profile> sql show state-compressor-orphans --profile <profile>

Important room-upgrade caveat:

• Matrix room upgrades create a replacement room.
• They do not automatically transfer all membership events to the new room.
• Use upgrades when you explicitly want that replacement-room behavior, not as a casual fix for adminlessness.

Duration Formats, Safety Guards, and Policy Optimizer

Generalized Duration Units

matrix-surgeon accepts human-readable duration strings in CLI cutoffs and config.toml retention settings:

• Years: 5y, 3yr, 1year, 5 years (converted as 365 days/year)
• Months: 6mo, 18month, 24 months (converted as 30 days/month)
• Weeks: 2w, 12week, 52 weeks (converted as 7 days/week)
• Days: 90d, 180day, 365 days

Safety Note: Sub-day units (h, hr, min, m, s, ms) are intentionally rejected to prevent catastrophic data loss from typographical errors.

Safety Guards: Short Duration & Granular Protection Boundaries

1. Short Duration Guard (< 1 year): Purging local media or message history newer than 1 year (< 365 days) requires an explicit --confirm flag on the command line. Routine remote media cache purging does not require confirmation.
2. Hard Protection Boundaries: Set immutable safety ceilings in [profiles.<name>.retention] or override via --protect-newer-than <duration>:
   • protect_newer_than = "3 years": Global fallback safety boundary across all data types.
   • protect_newer_than_media (or protect_newer_than_local_media): Hard safety floor for local user media uploads (e.g. 3 years).
   • protect_newer_than_remote_media: Hard safety floor for disposable remote cache (e.g. 14 days or 180 days).
   • protect_newer_than_messages (or protect_newer_than_local_messages): Hard safety floor for local room message history (e.g. 5 years).
   • protect_newer_than_remote_messages: Hard safety floor for federated remote room message history (e.g. 180 days or 1 year). Any operation attempting to target data newer than the configured boundary aborts immediately with a clear error.

Algorithmic Retention Policy Optimizer

The policy recommendation engine analyzes live disk usage via SSH df -P /, compares against your target disk percentage (default: 80%, configurable via --target), queries remote media cache, and evaluates multi-tier local media storage distributions. It computes the minimal sequential eviction ladder required to bring disk usage safely below your target while respecting your --protect-newer-than boundary:

# Recommend optimal retention policy to reach 80% disk usage with a 3-year protection floor:
matrix-surgeon workflow retention-purge personal --recommend --protect-newer-than 3y

# Recommend retention policy targeting 70% disk usage in JSON format:
matrix-surgeon workflow retention-purge personal --recommend --target 70 --json

The optimizer produces an actionable step-by-step impact report and a copy-pasteable [profiles.<name>.retention] configuration block.

Media Retention and Timestamp Synchronization

Matrix Synapse evaluates media purge candidates using last_access_ts < cutoff (idle access time), not created_ts (upload date). Active clients re-requesting old rooms can update last_access_ts, preventing upload-age deletions. matrix-surgeon provides a safe timestamp synchronization and deletion pipeline that protects active user and room avatars (keep_profiles=true).

1. Inspect Media Storage and Age Distribution

Run live dry-run with histograms to see total uploads, idle storage, and API-deletable media:

matrix-surgeon workflow retention-purge personal --dry-run live

2. Synchronize Media Timestamps Safely

Align last_access_ts = created_ts for media older than a specific date cutoff:

# Preview timestamp sync and check avatar protection counts
matrix-surgeon admin media sync-access-times personal 1460d --size-gt 10000000

# Execute timestamp synchronization
matrix-surgeon --execute admin media sync-access-times personal 1460d --size-gt 10000000 --apply

3. Safely Delete Unlinked Media Files

After synchronization, use the official Synapse Admin API to delete unlinked files from disk while strictly preserving avatars:

# Preview remote media cache purge with live disk check and candidate queries
matrix-surgeon admin media purge-cache personal 180d --dry-run live

# Purge remote cached media originating from other federated servers
matrix-surgeon --execute admin media purge-cache personal 180d

# Preview local media deletion with live disk check, candidate counts, and avatar protection counts
matrix-surgeon admin media delete-local personal 1460d --size-gt 10000000 --keep-profiles --dry-run live

# Delete local media older than cutoff while protecting active user/room avatars
matrix-surgeon --execute admin media delete-local personal 1460d --size-gt 10000000 --keep-profiles

Targeted Large Media Inventory and Deletion

When disk space runs low, pinpointing and evicting specific large files gives the highest storage reclaim with the least user impact.

# List local files >= 5MB (default size floor 5MB, sorted by size descending):
matrix-surgeon admin media list personal

# Inspect files >= 20MB across both local repository and remote cache, sorted by upload date:
matrix-surgeon admin media list personal --min-size 20MB --source all --sort created --order desc

# Output report as JSON or CSV:
matrix-surgeon admin media list personal --min-size 10MB --json
matrix-surgeon admin media list personal --min-size 10MB --csv

# Generate copy-pasteable CLI deletion command for all listed non-avatar files:
matrix-surgeon admin media list personal --min-size 50MB --print-delete-command

# Dry-run preview deletion of specific media IDs:
matrix-surgeon admin media delete personal --media-id <id1> --media-id <id2>

# Execute deletion (unlinks file from disk and database; verified against active avatars):
matrix-surgeon admin media delete personal --media-id <id1> --execute

# Delete media IDs from a file list (lines starting with # or empty lines ignored):
matrix-surgeon admin media delete personal --from-file to_delete.txt --execute

Orphaned Media Store Files

Synapse stores the mapping from media IDs to files in PostgreSQL. A file on disk with no database row pointing at it is an orphan: Synapse can never serve it, no user can request it, and retention tooling cannot see it. Orphans accumulate when a row is deleted but the file removal fails, when a download is interrupted after the file is written but before the row commits, or after database restores. They consume real disk space while being invisible to every age- or size-based retention stage.

workflow media-orphans scans the four on-disk families (local_content, remote_content, local_thumbnails, remote_thumbnail), compares each file key against the database, and reports or deletes unreachable files:

# Read-only scan (all orphan ages are eligible by default):
matrix-surgeon workflow media-orphans public

# Scan remote families only, machine-readable:
matrix-surgeon workflow media-orphans public --source remote --json

# Narrow the purge to older orphans only (optional):
matrix-surgeon workflow media-orphans public --older-than 1d

# Actually delete matching orphans (requires --execute and --apply):
matrix-surgeon workflow media-orphans public --execute --apply

• --source local|remote|all (default all), --older-than CUTOFF (optional; by default every orphan age is eligible), and --min-size BYTES bound what can be removed.
• Files modified in the last hour are always excluded (in-flight upload grace), so a fresh orphan is never deleted while a request may still be completing. --older-than can only make the cutoff stricter.
• Without --apply the command runs the scan and prints per-family counts, bytes, zero-byte placeholders, oldest/newest orphan dates, and the 20 largest orphan files.
• Deletion only removes files that are absent from the database, then prunes directories left empty. Every purge is recorded in the introspective ledger with files and bytes removed.
• Exit code 3 means no orphans were found; 0 means the report/scan succeeded.

Media Quarantine and Protection

Quarantine problematic media by media ID, room ID, or user ID to immediately take it offline, or protect critical media from being quarantined:

# Quarantine specific media ID
matrix-surgeon --execute admin media quarantine public --media-id <media_id>

# Quarantine all media uploaded to a specific room
matrix-surgeon --execute admin media quarantine public --room-id '!roomid:example.net'

# Quarantine all media uploaded by a specific user
matrix-surgeon --execute admin media quarantine public --user-id '@spammer:example.net'

# Unquarantine media
matrix-surgeon --execute admin media quarantine public --media-id <media_id> --unquarantine

# Protect media from quarantine (or unprotect)
matrix-surgeon --execute admin media protect public <media_id>
matrix-surgeon --execute admin media protect public <media_id> --unprotect

User Moderation and Media Administration

Manage per-user media, shadow-banning, and user-scoped event redaction directly through the Synapse Admin API:

# List all media uploaded by a specific user
matrix-surgeon --execute admin users media public '@user:example.net' --limit 50

# Delete all media uploaded by a specific user (requires --confirm)
matrix-surgeon --execute admin users delete-media public '@user:example.net' --confirm

# Shadow-ban a user (silently discards messages to other users while keeping the sender unaware)
matrix-surgeon --execute admin users shadow-ban public '@spammer:example.net'

# Remove shadow-ban
matrix-surgeon --execute admin users shadow-ban public '@spammer:example.net' --delete

# Redact all events sent by a user across the server or in a specific room
matrix-surgeon --execute admin users redact public '@spammer:example.net' --confirm
matrix-surgeon --execute admin users redact public '@spammer:example.net' --room-id '!room:example.net' --limit 100 --confirm

Bridge & Protected User Account Safeguards

To prevent accidental deactivation or destruction of crucial administrative accounts and bridge bots (e.g. mautrix-jmpchat, mautrix-telegram, mautrix-whatsapp), matrix-surgeon enforces account protection across all bulk and single-user deactivation workflows:

1. Configurable Protections: Set protected_user_ids and protected_user_prefixes globally or under specific profile blocks in config.toml.
2. Automatic Exclusions: Workflows (deactivate-inactive-users, delete-user, kick-inactive-users, cleanup-bridge-ghosts, purge-bridge-bot) automatically filter out protected accounts and report explicit reasons.
3. Safety Override: For deliberate operational actions on protected accounts, pass the --force flag on admin users deactivate, admin users set, and workflow delete-user.

Separately, bot_user_ids / bot_user_prefixes (also global or per profile) mark non-appservice bots for the dead-room analysis: rooms with a bot creator, bot admin, or bot member are treated as bridged and score only on empty, ghost_only, and inactive. Appservice accounts are detected automatically from the database.

Room Kicking and Membership Management

matrix-surgeon supports three explicit membership removal (kick) transports with full explainability:

• cs-kick: standard Client-Server API POST /_matrix/client/v3/rooms/{room_id}/kick
• state-put: direct state event PUT /_matrix/client/v3/rooms/{room_id}/state/m.room.member/{user_id} (leave)
• sql: direct PostgreSQL update on room_memberships streamed safely via SSH stdin

Kick a single user from a room with explicit transport selection:

matrix-surgeon admin rooms kick public '!room:example.net' '@baduser:example.net' --transport cs-kick --reason 'Spamming'
matrix-surgeon --execute admin rooms kick public '!room:example.net' '@baduser:example.net' --transport auto --apply

User Deletion and Inactive User Kicks

User deletion is a ladder of increasing destruction. Pick the level explicitly with --level (the legacy --erase flag is kept as an alias for --level erase):

Level	What it does
deactivate	Kick from all joined rooms + admin-API deactivate (erase: false)
erase	Same, but deactivates with erase: true (GDPR message erasure)
deep	erase plus direct SQL purge of residual rows (directory, profile, activity stats, push rules/pushers, filters, account data, notifications, devices, tokens, IP history)
nuclear	deep plus purging every event the user ever sent, purging drained rooms, and hard-deleting the users row (requires --dangerous --confirm --apply --execute)

Delete a single account through the first two rungs:

matrix-surgeon workflow delete-user public '@former_user:example.net'
matrix-surgeon --execute workflow delete-user public '@former_user:example.net' --erase --delete-drained-rooms --apply

Batch deletion accepts repeatable --user flags and a --user-id-file (one Matrix ID per line):

matrix-surgeon workflow delete-user public --user '@a:example.net' --user '@b:example.net' --user-id-file leavers.txt --dry-run live
matrix-surgeon --execute workflow delete-user public --user-id-file leavers.txt --level deep --apply

Nuclear deletion additionally needs both acknowledgement flags and prints a federation caveat:

matrix-surgeon workflow delete-user public '@spammer:example.net' --level nuclear --dangerous --confirm --dry-run live
matrix-surgeon --execute workflow delete-user public '@spammer:example.net' --level nuclear --dangerous --confirm --apply

Every destructive preview shows exactly what will run before anything changes: kick/deactivate/room-purge stages render their actual admin-API calls (method, path, JSON payload), SQL stages print the full rendered DELETE statements plus the command they would be executed through, candidate rows include evidence (creation time, last seen, events sent / last activity per room) with human-readable UTC timestamps, and plan mode marks counts that require live probing as gated until you re-run with --dry-run live.

workflow deactivate-inactive-users now shares the same destruction engine and supports finer thresholds for "enough" interaction via an inactivity basis:

• --basis last-seen (default): server-side last-seen timestamp older than the cutoff.
• --basis sent-event: last event the user sent is older than the cutoff (SQL scan).
• --basis never-sent: the user has never sent any event (SQL scan).
• --since <cutoff> overrides the day-based window; accepts epoch ms, YYYY-MM-DD, RFC3339, or relative durations like 30d, 6mo, 2y.
• --min-events N caps total events a candidate may have sent.

matrix-surgeon workflow deactivate-inactive-users public --never-seen-only --since 2024-01-01
matrix-surgeon --execute workflow deactivate-inactive-users public --basis sent-event --min-events 5 --since 180d --level erase --apply

workflow kick-inactive-users kicks discovered users from the rooms they still occupy and accepts the same basis knobs:

matrix-surgeon workflow kick-inactive-users public --inactive-days 90
matrix-surgeon --execute workflow kick-inactive-users public --basis never-sent --inactive-days 180 --fail-fast --apply

Room-scoped eviction is handled by workflow kick-room-inactive, which finds joined room members whose last activity inside each room predates the cutoff (or who never sent an event there), optionally capped by per-room event count:

matrix-surgeon workflow kick-room-inactive public --inactive-days 90 --min-events 0 --room '!room:example.net'
matrix-surgeon --execute workflow kick-room-inactive public --since 2024-06-01 --fail-fast --apply

All destructive workflows default to plan-mode dry runs; pass --dry-run live for a live read-only preview, then add --apply --execute to mutate.

Room Health & Storage Overview Report

Gain immediate visibility into database room occupancy, DM vs group room taxonomy, space hierarchies, admin governance, and activity age distributions to accurately tune expected_keep_rooms retention safety tripwires:

# Print high-level room summary cards and activity distribution bar charts
matrix-surgeon workflow room-report public --summary-only

# Detailed room listing sorted by last activity timestamp
matrix-surgeon workflow room-report public --limit 50

# Filter direct messages or group rooms with search and pagination
matrix-surgeon workflow room-report public --category dm --limit 20
matrix-surgeon workflow room-report public --category group --min-members 10 --search 'matrix'

# Filter orphaned rooms with no local admins present
matrix-surgeon workflow room-report public --admin-status no_admin

# Machine-readable JSON and CSV outputs for scripting and audits
matrix-surgeon workflow room-report public --json
matrix-surgeon workflow room-report public --format csv > rooms.csv
matrix-surgeon admin rooms report public --summary-only

Dead Room Cleanup and Batch Migration

Analyze dead rooms read-only, then batch-clean a chosen selection:

# read-only: per-signal counts, combinations, top rooms, recipes
matrix-surgeon workflow dead-rooms public

# purge one explicit selection (--execute runs the probe, --apply deletes)
matrix-surgeon workflow clean-dead-rooms public --combo empty --purge --block --execute --apply

Batch auto-migrate rooms with no admin to a new admin user:

matrix-surgeon workflow auto-migrate-no-admin public --new-room-user-id '@admin:example.net'
matrix-surgeon --execute workflow auto-migrate-no-admin public --new-room-user-id '@admin:example.net' --apply

Per-Room Retention State and Health Audit

Set explicit m.room.retention state events on individual rooms:

matrix-surgeon client rooms set-retention-state public '!room:example.net' --max-lifetime-days 90
matrix-surgeon --execute client rooms set-retention-state public '!room:example.net' --max-lifetime-days 90

Audit per-room retention policy compliance:

• Check rooms missing m.room.retention state: matrix-surgeon sql show retention-state-missing --profile public
• Check rooms with events exceeding max_lifetime: matrix-surgeon sql show retention-state-overdue --profile public
• Run the dedicated retention state audit workflow: matrix-surgeon workflow retention-purge public --mode state-audit
• Include retention health in integrity sweeps: matrix-surgeon --execute workflow integrity-sweep public --section retention

Room History Purging

Purge historical events from a room up to an event ID or timestamp cutoff:

# Preview history purge up to 180 days ago:
matrix-surgeon admin rooms purge-history public '!room:example.net' --purge-up-to 180d

# Execute history purge with local event deletion (requires --confirm for cutoffs newer than 1 year):
matrix-surgeon --execute admin rooms purge-history public '!room:example.net' --purge-up-to 180d --delete-local-events --confirm

Sliding Sync Guide

Sliding sync keeps a per-user membership snapshot and applies room membership deltas as they arrive. If a room was marked forgotten while the delta stream still has pending membership changes, the sliding sync process can crash with a room-list assertion. Use these probes and the guided workflow when you need to find or repair that state.

Inspect the pieces first:

• background updates that are still pending: sql show sliding-sync-background-updates --profile <profile>
• currently active sliding sync client connections: sql show sliding-sync-connections --profile <profile> --param user_id='@user:example.net'
• conflicting forgotten rooms versus live delta streams (the actual crash candidates): sql show sliding-sync-conflicts --profile <profile>

Run the guided audit and generate a remediation plan:

• matrix-surgeon workflow audit-sliding-sync <profile>
• narrow to the users most at risk: --filter crashing or --filter high-risk
• widen back to every conflict with any room delta: --filter all-deltas
• inspect everything, including zero-delta forgotten rooms: --filter all
• render the full remediation SQL without applying it: --format sql
• apply the generated fix after reviewing it: matrix-surgeon --execute workflow audit-sliding-sync <profile> --filter all-deltas --fix --fix-mode both

The remediation SQL has two independent phases:

• unforget: reset forgotten=0 for the conflicted room memberships and their sliding sync snapshots
• reset-sessions: delete the stuck sliding sync connections so active clients reconnect cleanly

--fix-mode defaults to both, which runs the unforget phase and the session reset in one batch. The workflow stays dry-run until --execute is supplied. Sliding sync probes live under sql show ... like any other recipe; both mutation recipes are marked dangerous in the catalog:

• sql show reset-stuck-sliding-sync-connections --profile public
• sql show unforget-sliding-sync-rooms --profile public

Query And Apply Modes

The tool uses three operator modes repeatedly:

• dry-run: the default; prints the SQL, HTTP, and SSH actions without touching live systems
• query/report mode: usually --execute without --apply; fetches real data and prints reports, but stops before mutations in the higher-risk workflows
• apply mode: --execute plus --apply for the workflows that intentionally separate inspection from mutation

If you remember only one thing, remember this:

• --execute means "talk to the real server now"
• --apply means "make the risky change now"

Typical examples:

• print only the SQL for manual use: matrix-surgeon sql print delinquent-rooms --profile public
• run a live report with no mutations: matrix-surgeon --execute workflow cleanup-bridge-ghosts public '@jmpchat_%'
• apply the mutation phase after reviewing the report: matrix-surgeon --execute workflow cleanup-bridge-ghosts public '@jmpchat_%' --deactivate-users --delete-single-member-rooms --apply

When Not To Use A Workflow

These are the main places operators get themselves into trouble:

• workflow recover-room-admin: do not use it when you do not have a specific trusted local user to promote, and do not use it when you already know the room is headed for shutdown or replacement.
• workflow resolve-delinquent-room: do not use it as your first inspection step, and do not use it when the room still has a workable local admin recovery path.
• workflow guide-room-upgrade: do not use it as a casual cleanup command or as a vague fix for an adminless room. Use it when you intentionally want a replacement room.
• workflow cleanup-bridge-ghosts: do not use it with a wide prefix until you have reviewed the report, and do not use it for bridge users that are still actively serving traffic.
• workflow deactivate-inactive-users: do not use it as a blind age-based purge. Narrow the candidate set first and review the live report before applying anything.
• workflow purge-bridge-bot: do not use it while the bridge is still meant to be live or while the prefix still matches accounts you want to keep.
• workflow retention-purge: do not start with full mode unless you already understand the profile's retention settings and have reviewed inventory-only output first.

Shell Completion

matrix-surgeon can emit dynamic shell completion scripts. They stay aware of:

• configured profile names from ~/.config/matrix-surgeon/config.toml
• SQL recipe names
• SQL --param names
• workflow names and high-value option names
• retention mode names
• room-version candidates for upgrade commands
• the new guided workflow names and their main options

Print the scripts directly:

matrix-surgeon completion bash
matrix-surgeon completion zsh

Use the checked-in static files if you prefer:

• bash: completions/matrix-surgeon.bash
• zsh: completions/_matrix-surgeon

Example installation:

# bash
source completions/matrix-surgeon.bash

# zsh
fpath=("$PWD/completions" $fpath)
autoload -Uz compinit && compinit

Completion reads the fixed ~/.config/matrix-surgeon/config.toml path.

Tutorial

See TUTORIAL.md for end-to-end operator walkthroughs covering delinquent rooms, stale bridge ghosts, inactive users, retention cleanup, room upgrades, SQL side-car use, and shell completion setup.

See CHEATSHEET.md for a one-page operator reference with the most common commands and stop-sign reminders.

Retention Guide

The retention workflow uses the retention table in a profile from config.toml.

These settings mean:

• expected_keep_rooms: safety floor; if inventory finds fewer occupied rooms than this, the workflow aborts
• media_retention: age cutoff for the broad local-media cleanup pass
• big_media_retention: age cutoff for the large-media cleanup pass
• big_media_size_bytes: minimum size for the large-media cleanup pass
• medium_media_retention: age cutoff for the medium-media cleanup pass
• medium_media_size_bytes: minimum size for the medium-media cleanup pass
• remote_media_retention: age cutoff for cached remote media cleanup
• small_room_message_retention: how much history small rooms keep before purge
• big_room_message_retention: how much history big rooms keep before purge
• big_room_threshold: rooms at or above this joined_local_members count are treated as big rooms

The workflow stages mean:

• inventory: fetch rooms and classify them as empty, small, or big
• empty-room deletion: delete rooms with zero local joined members
• remote media purge: remove old cached remote media
• local media cleanup: remove large old media, then medium old media, then general old media
• history cleanup: purge old events from small rooms and big rooms separately
• disk checks: run df -P / over SSH before and between stages, stopping early once disk pressure is low enough

The retention modes are:

• all: run the full staged workflow
• inventory-only: fetch and classify rooms only; no mutations
• rooms-only: inventory plus empty-room deletion only
• media-only: media cleanup only; no room deletion or history purge
• media-by-created-age: direct SQL media deletion by upload/creation age (DANGEROUS: requires --dangerous, --apply, and --execute)
• history-only: inventory plus history purge only; no room deletion or media cleanup
• state-audit: audit per-room m.room.retention state event compliance and find overdue events
• walk: cron-friendly bucket-walk; runs the next fixed deletion bucket until disk is at/below the limit, one bucket per run (see "Retention Walk Mode" below)

The media age evaluation strategies (--media-age-by) are:

• last-access (default): safe deletion via official Synapse Admin API evaluating last_access_ts < cutoff
• created: direct SQL deletion evaluating upload created_ts < cutoff (DANGEROUS: requires --dangerous, --apply, and --execute)

Room targeting filters are:

• --include-room <room_id>: restrict room-based stages to one room ID; repeat as needed
• --exclude-room <room_id>: skip one room ID during room-based stages; repeat as needed
• --room-id-file <path>: read included room IDs from a file, one room ID per line, ignoring blank lines and # comments

Filter behavior:

• filters affect room previews, empty-room deletion, and history purge stages
• filters do not affect media-only cleanup, because the media admin endpoints in this workflow are not room-scoped
• the expected_keep_rooms safety check still uses the full occupied-room inventory, not just the filtered subset

Suggested operator sequence:

1. Start with matrix-surgeon workflow retention-explain.
2. Run matrix-surgeon workflow retention-purge <profile> --mode inventory-only.
3. Add room filters if you want a room deletion or history pass to target a known subset.
4. Run a narrower cleanup mode if you want to tackle one pressure source at a time.
5. Run the default all mode only after the printed plan matches your intent.

Retention Walk Mode (Cron)

--mode walk turns retention-purge into a cron-safe, target-driven evictor. It resolves a stop target, walks data oldest-first in small chunks, and halts the moment the target is met — deleting as little as possible per run:

# target: free at least 5 GiB of disk (also: percentage, used-GiB, reclaim-GiB)
matrix-surgeon workflow retention-purge public --mode walk --target-free-gigabytes 5

# dry-run shows the exact chunk plan and ladders, no SSH/SQL writes
matrix-surgeon workflow retention-purge public --mode walk --target-reclaim-gigabytes 3

# live: chunked execution with per-chunk disk re-checks
matrix-surgeon workflow retention-purge public --mode walk --execute --json

Targets (pick one; they are mutually exclusive):

• --max-disk-usage-percentage N / --max-disk-usage-gigabytes N — used-disk stop (falls back to the profile retention max_disk_usage_*, default 85%)
• --target-free-gigabytes N — stop once free space reaches N GiB
• --target-reclaim-gigabytes N — stop once this walk has freed N GiB

Emergency floor (emergency_min_free_gigabytes = 8 in the retention config): when free space is below the floor and no explicit target was given, the walk stops as soon as that headroom is restored, and the safe-clearing preflight runs first. Safe clearing is CHECKPOINT plus, with --os-clean (default on when the floor is active), docker system prune -f and journalctl --vacuum-size=200M over SSH. If that alone meets the target, the walk exits 0 without deleting any Matrix data.

Safety envelope for every destructive walk: the storage-composition probe refuses to proceed when non-Matrix data outweighs all Matrix storage (the disk pressure is not yours to fix), and the guard fails closed (exit 2) when the composition or reclaimable estimate cannot be measured — override with --no-storage-guard.

Media ordering (--walk-rank, walk mode only):

• bytes-days (default) — every bucket ladder is merged into one list ranked by size × age (bytes-days), largest regret first: a 1 GB one-year-old file is deleted before a 1 MB three-year-old file, and equal sizes prefer the older file.
• exact — per-file ranking from the database (media-ranked-candidates), top 5000 eligible files, deleted one by one via DELETE /media/{server}/{media_id}.
• buckets — legacy big → medium → all bucket order.

Simulated walk (read-only, --simulate / automatic in --dry-run live when a target or the emergency floor is active): prints every slice in walk order with its estimated eligible count and bytes, the running total, and a verdict (would run / TARGET MET / not needed / skipped / failed), plus the projected free space and an avatar-eligibility summary per bucket. Nothing is deleted; the simulation shares the exact plan builder with execution, so the two cannot disagree about ordering. Slow parts can be skipped with repeatable --simulate-skip orphans|media|history|eligibility (default: full depth). --max-delete-gigabytes N bounds the walk: the slices are simulated first and only those whose measurable payload fits under the cap are executed (slices with unmeasurable bytes, like room history, are not run under a cap).

# simulation of a bounded walk: show what 1 GiB would delete, then stop there
matrix-surgeon workflow retention-purge public --mode walk --target-reclaim-gigabytes 1 --dry-run live

# hard payload cap (simulated first, then applied) and a fast simulation
matrix-surgeon workflow retention-purge public --mode walk --max-delete-gigabytes 1 --execute
matrix-surgeon workflow retention-purge public --mode walk --simulate --simulate-skip orphans --dry-run plan

Pressure tiers ([profiles.<name>.retention.pressure], optional): once configured, a plain --mode walk auto-selects a tier from the live disk percentage at execute time (>= elevated_percentage / high_percentage / critical_percentage) and decays retention per that tier's overrides, stopping at reclaim_to_percentage unless an explicit target/--max-disk-usage-* was passed. Overrides are validated at config load: a value at or above the base retention, below the hard floors (7d remote media, 30d local media/history), below protect_newer_than_*, looser than a lower tier, or under an unknown key is rejected with an error naming the exact key. Override per run with --pressure-tier auto|none|elevated|high|critical (elevated/high/critical force a tier and require --mode walk). --pressure-tier none disables tiers for one run.

# see which tier would apply (read-only)
matrix-surgeon workflow retention-purge public --mode walk --dry-run live

# tiered cron walk (no target: reclaim_to_percentage from the pressure table)
matrix-surgeon workflow retention-purge public --mode walk --execute --json

# force the critical tier for one run
matrix-surgeon workflow retention-purge public --mode walk --pressure-tier critical --execute

# exact per-file ranking (top 5000 by size x age) instead of bucket cells
matrix-surgeon workflow retention-purge public --mode walk --walk-rank exact --execute

# legacy bucket order, or force the safe OS preflight explicitly
matrix-surgeon workflow retention-purge public --mode walk --walk-rank buckets --execute
matrix-surgeon workflow retention-purge public --mode walk --os-clean --execute

Strict one-time cutoff (--older-than DURATION, non-walk modes): one run deletes everything older than the horizon regardless of configured retention windows or pressure tiers (protection floors still apply, scoped to the stages the mode can touch: media-only checks media floors, history-only checks the history floor, all checks every floor). --mode defaults to all, so a single command covers empty rooms, remote cache, media (every size) and room history. Under this flag the workflow first aligns media last_access_ts to created_ts at the cutoff (exact SQL printed) so Synapse's /media/delete evaluates upload age, then deletes with keep_profiles=true. Rooms with no events older than the cutoff return HTTP 404 and are skipped without aborting the run; other per-room failures retry and are reported per stage.

Because keep_profiles=true protects avatar/profile media, a strict media sweep can report Eligibility: N matched, 0 deletable when every match is referenced as a user, member, directory, or room avatar. Inspect with sql run media-deletion-eligibility --profile <p> --param before_ts=90d --execute, or opt out explicitly with --no-keep-profiles (dangerous write; requires --dangerous --apply; rejected with --mode walk; old avatars are deleted and clients fall back to default images until re-uploaded).

# dry-run first: exact SQL and per-stage counts
matrix-surgeon workflow retention-purge public --older-than 2y --dry-run live

# one-shot strict cleanup: media + history + empty rooms older than 2 years
matrix-surgeon workflow retention-purge public --older-than 2y --execute --apply

# why did the sweep delete so few files? (read-only Synapse keep_profiles filter)
matrix-surgeon sql run media-deletion-eligibility --profile public --param before_ts=90d --execute

# include old avatars: keep_profiles=false (dangerous; preview first)
matrix-surgeon workflow retention-purge public --mode media-only --older-than 90d --no-keep-profiles --dry-run live
matrix-surgeon workflow retention-purge public --mode media-only --older-than 90d --no-keep-profiles --execute --apply --dangerous

Walk order (oldest first):

0. orphaned media-store files (no database row) — first line of defense; frees disk without touching any user-visible media, no age cutoff (1-hour in-flight grace)
1. empty rooms (0 joined members) — chunks of 25 rooms
2. remote media cache — age slices 10y -> 5y -> 3y -> 2y -> 1y -> 6mo -> 3mo -> 1mo
3. big local media — age slices, each also stepping size floors 50MB -> 10MB -> 2MB -> 512KB down to big_media_size_bytes
4. medium local media — same pattern down to medium_media_size_bytes
5. all old local media — same pattern down to media_retention
6. small-room history — chunks of 25 rooms (skipped when retention >= 5000 days)
7. big-room history — chunks of 25 rooms (skipped when retention >= 5000 days)

Each slice is one API call, followed by a disk re-check and a target check. Age slices start old and end exactly at the configured retention cutoff, so a walk never touches protected data — it only reaches younger files if older slices were not enough to hit the target.

Behavior and guards:

• SSH disk checks are mandatory: --mode walk refuses --no-disk-check
• bucket 0 sweeps orphaned media-store files (no database row) before touching any user-visible media; files modified in the last hour are skipped
• transient API failures are retried 3x with 5s/15s backoff; if a slice still fails it is recorded as failed and the walk continues to the next slice
• all retention safety guards apply per slice (--confirm for sub-1-year cutoffs, granular protect_newer_than_* floors, expected_keep_rooms)
• local media deletion uses Synapse's /media/delete, which falls back to upload creation age for never-accessed (NULL) files; fresh uploads are never over-deleted
• the final summary shows target, initial/final disk, total reclaimed, deleted files, failed slices, duration, and a per-slice table; --json emits RetentionWalkReport
• the summary is also written as ledger notes, so matrix-surgeon ledger explains what each cron run did
• exit codes are stable for cron: 0 progress, 1 every slice failed, 3 target already met / nothing to do

Storage-composition guard (anti-runaway layer):

Before deleting anything, the walk measures PostgreSQL size (SQL), the Synapse media store size (SSH du, bounded to 25s), and derives "everything else" = disk used - Matrix storage. If the media store scan is too slow for the VPS, the tool falls back to database-recorded media bytes and says so in a note, so a live probe never blocks for minutes. If the target would require freeing more than all Matrix storage, or more than the retention plan is allowed to reclaim, the walk refuses to run and exits 2 (safety gate). Composition, reclaimable ceiling, and "would abort" verdicts are shown in --dry-run live, included in --json (storage, reclaimable_bytes), and written to the ledger. Override with --no-storage-guard when you know the disk consumers better than the tool does.

# read-only: composition + whether the target is reachable
matrix-surgeon workflow retention-purge public --mode walk --target-reclaim-gigabytes 3 --dry-run live

# force the walk even if the guard would abort
matrix-surgeon workflow retention-purge public --mode walk --target-free-gigabytes 5 --no-storage-guard --execute

The profile max_disk_usage_* value is inert when --target-free-gigabytes or --target-reclaim-gigabytes is set: the header labels it (unused: a ... target is set). Use --target-free-gigabytes for a headroom goal or --target-reclaim-gigabytes for a bounded amount of deletion. Targets that imply deletion despite comfortable free space print a headroom note before any work starts.

Dead Rooms And Automated Cleanup

There are no profile policy blocks: dead-room targeting is explicit per invocation, so a cron line is self-documenting. The analysis and the selectors default to rooms hosted on this homeserver; federated rooms carry no actionable death signals (they can only be left), so they are hidden unless you pass --include-federated, which scores them on inactive alone. Spaces (m.space rooms) are excluded for now — a dedicated dead-space analysis is not implemented yet — and can be listed with --include-spaces. Start with the read-only analysis:

matrix-surgeon workflow dead-rooms public        # signals, combos, top rooms, purge recipes
matrix-surgeon workflow dead-rooms public --json # machine-readable analysis
matrix-surgeon workflow dead-rooms public --pick # pick combinations/rooms; deletes only with --apply + confirm

Room death signals are scored per room (weights in parentheses):

Signal	Weight	Meaning
empty	40	zero joined members
ghost_only	35	every joined member never sent an event
inactive	20	no activity for inactive_days or more
bridge_duplicate	25	non-canonical copy of a bridge family (same remote chat) that is empty, ghost-only, or inactive; the live copy is never flagged
abandoned	15	members present, no joined human admin (>= 50 power, local or remote), creator gone/deactivated and no bots involved
encryption_inert	15	m.room.encryption set, zero m.room.encrypted events ever

Scores are additive: a room that is empty, is abandoned, and has been inactive for a year scores 75 without any single signal dominating the decision. Only joined members count as admins: a departed admin's stale power-level entry no longer masks an abandoned room.

Bridged rooms (bot creator, bot admin, or bot member) are detected automatically from appservice accounts and from the bot_user_ids / bot_user_prefixes lists in the config (global or per profile, mirroring the protected-user lists). Bridge rooms keep only the empty, ghost_only, inactive, and bridge_duplicate signals; abandoned and encryption_inert are suppressed because a bridge bot is the expected administrator. They are marked [bridged] in reports.

Bridge ports are detected from their m.bridge / uk.half-shot.bridge state, so bot-run Signal/Telegram portal rooms are not treated as death candidates. The report groups rooms that mirror the same remote chat into bridge duplicate families and marks the live copy (keep: — most recent activity, newest creation as tie-break). Stale copies flagged in the family list surface as bridge_duplicate and can be purged with --require bridge_duplicate; a dormant family with no active sibling stays report-only for manual review.

Dead rooms

# proactive probes (also `sql show`-able standalone)
matrix-surgeon sql show ghost-only-rooms --profile public
matrix-surgeon sql show encryption-inert-rooms --profile public
matrix-surgeon sql show room-death-signals --profile public

# candidate selection: exactly one explicit selector is required
matrix-surgeon workflow clean-dead-rooms public --combo empty
matrix-surgeon workflow clean-dead-rooms public --require empty,ghost_only
matrix-surgeon workflow clean-dead-rooms public --require empty,ghost_only --any
matrix-surgeon workflow clean-dead-rooms public --min-score 75

# stale bridge copies left behind by bridge churn (see "Bridge duplicate families")
matrix-surgeon workflow clean-dead-rooms public --require bridge_duplicate --inactive-days 90

# delete the matched rooms after reviewing the report
matrix-surgeon workflow clean-dead-rooms public --combo empty --execute --apply

--combo matches an exact signal set, --require matches all listed signals (or at least one, with --any), and --min-score filters on the composite score. The header prints the selection and the effective --inactive-days (default 365), so the log records exactly what was targeted. Deletion still requires --execute --apply. --include-federated extends the selection to federated rooms (scored on inactive alone; leave candidates), --include-spaces extends it to spaces, and --exclude-bridged drops bridged rooms entirely (start with human rooms only).

Stale bridge copies are purgeable with --require bridge_duplicate (review the "Bridge duplicate families" section first; leave --exclude-bridged off so the portals are visible). The canonical live copy is never flagged.

--apply refuses unrecommended combinations unless --confirm is present: --inactive-days below 30 (too recent to judge), --include-federated (only local copies are removed), or --include-spaces (dead-space analysis is not implemented yet). A selection targeting bridge_duplicate combined with --exclude-bridged is rejected outright — the flag would hide exactly the rooms the selection targets.

Two safeguards refine any selection:

• --without-signal SIGNALS excludes rooms matching those signals (comma-separated). Use --without-signal inactive to bulk-clean rooms that are not merely quiet, so a room that pops back to life is never caught.
• --max-members N guards against surprise deletions: candidates above N joined members are listed for review but skipped by the deletion step.

A practical bulk-by-category sweep, reviewed before it mutates anything:

# dry run: lists what the selection would delete
matrix-surgeon workflow clean-dead-rooms public --combo encryption_inert --without-signal inactive --max-members 50 --execute

# same command with --apply once the list looks right
matrix-surgeon workflow clean-dead-rooms public --combo encryption_inert --without-signal inactive --max-members 50 --execute --apply

Dead users

# discovery only; defaults are last-seen basis, 180-day window
matrix-surgeon workflow deactivate-inactive-users public --execute

# after report-only runs look right: the flags carry level/basis/window
matrix-surgeon workflow deactivate-inactive-users public --inactive-days 365 --basis never-sent --level deactivate --execute --apply --confirm

deep/nuclear levels additionally require --dangerous --confirm. If the gate does not open, the run exits 2 and touches nothing.

Stable exit codes

Workflows safe to run unattended share one contract: 0 success, 1 operational error, 2 safety gate aborted, 3 nothing to do. See docs/automation.md for the full cron contract, a sample crontab, and the report → arm → purge rhythm.

Database Maintenance (PostgreSQL)

workflow db-maintenance handles the routine PostgreSQL housekeeping that Synapse accumulates after large purges: dead-tuple bloat, index churn, and stale user_ips rows.

# canonical read-only diagnostics: device_inbox size/orphans, state_groups_state
# bloat, compressor progress, top state rooms (prints exact SQL; no --execute = SQL only)
matrix-surgeon workflow db-maintenance public --diagnose --execute

# dry-run plan: shows which probes run and what each flag would do
matrix-surgeon workflow db-maintenance public --vacuum

# live probes (read-only): bloat, largest tables, pending background updates
matrix-surgeon workflow db-maintenance public --vacuum --execute

# apply: VACUUM (ANALYZE) the top bloat candidates
matrix-surgeon workflow db-maintenance public --vacuum --execute --apply

# apply + vacuum one exact table (bypasses the >=20% dead-tuple heuristic)
matrix-surgeon workflow db-maintenance public --vacuum-table device_inbox --execute --apply

# apply + CHECKPOINT: flush WAL after a large delete (I/O only, no data locks;
# prints WAL size before/after)
matrix-surgeon workflow db-maintenance public --checkpoint --execute --apply

# apply + reindex: REINDEX (CONCURRENTLY) bloat candidates (double-gated)
matrix-surgeon workflow db-maintenance public --reindex --execute --apply --dangerous --confirm

# apply + reap old user_ips rows (double-gated)
matrix-surgeon workflow db-maintenance public --purge-old-ips --purge-older-than 180d \
  --execute --apply --dangerous

# apply + purge orphaned device_inbox rows (to-device messages for deleted devices)
matrix-surgeon workflow db-maintenance public --purge-orphan-device-inbox \
  --execute --apply --dangerous

# shrink one table after a large purge (ACCESS EXCLUSIVE lock; temp headroom needed)
matrix-surgeon workflow db-maintenance public --vacuum-full device_inbox \
  --execute --apply --dangerous --confirm

# reclaim freed heap space automatically: VACUUM FULL every table with >= 250 MB
# reclaimable (pgstattuple_approx), largest-that-fits first, headroom-guarded
matrix-surgeon workflow db-maintenance public --reclaim-bloat --dry-run live
matrix-surgeon workflow db-maintenance public --reclaim-bloat \
  --execute --apply --dangerous --confirm

# trigger the installed rust-synapse-compress-state systemd unit over SSH
matrix-surgeon workflow db-maintenance public --compress-state --execute --apply

# systemd-independent compressor run: fixed number of chunks via the same container image
matrix-surgeon workflow db-maintenance public --compress-state --compress-chunks 1000 \
  --execute --apply

# device_inbox ownership report: who owns the queue and how stale (read-only)
matrix-surgeon workflow device-inbox public --execute

# exact per-device counts (one full scan; cached for 6h)
matrix-surgeon workflow device-inbox public --exact --execute

# delete devices idle >= 90d (their queued messages go too), then sweep + shrink
matrix-surgeon workflow purge-stale-devices public --older-than 90d \
  --execute --apply --dangerous

# or, zero logouts: prune only queued rows of stale devices, then shrink in one command
matrix-surgeon workflow purge-stale-devices public --queue-only --older-than 30d \
  --vacuum-full --execute --apply --dangerous

Rules the workflow enforces:

• VACUUM (ANALYZE) is the default; --vacuum-full <table> is opt-in per table and requires --dangerous --confirm --apply --execute together (ACCESS EXCLUSIVE lock, needs temp disk roughly the size of the rewritten table)
• --reclaim-bloat automates that: it re-probes exact heap bloat (pgstattuple_approx, installing the pgstattuple extension automatically if missing), selects tables with at least --reclaim-min-mb (default 250) reclaimable heap, and VACUUM FULLs them greedily — largest-reclaim-that-fits first, re-reading free disk after each — so it never starts a rewrite it cannot finish. Required headroom = projected new total (current total − reclaimable heap; indexes assumed not to shrink) + 1 GiB margin. Same gates as --vacuum-full; never deletes rows and never drops indexes (index bloat is fixed as a side effect of the rewrite). --dry-run live prints the selection, SQL and headroom before anything runs
• bloat candidates: pg_stat_user_tables with >= 1000 dead tuples and >= 20% dead ratio, capped at the 10 worst tables
• --vacuum-table <name> runs exactly VACUUM (ANALYZE) "<name>"; for one table, for the case where the 20% heuristic skips a table you know is churned
• --checkpoint runs CHECKPOINT; (I/O only, no data locks) and prints WAL status before and after — useful after a big delete to let WAL space return
• canonical probe recipes replace ad-hoc SQL: sql run table-size --param table=<name>, sql run media-remaining-at-cutoff --param before_ts=90d, sql run wal-status, sql run replication-slots, sql run device-inbox-orphan-count, plus the media access-time sentinel set: local-media-access-census (literal 0 vs NULL vs real access), local-media-never-accessed-months, local-media-recent-rows, remote-media-access-census (all read-only, exact SQL echoed, required params validated; every before_ts/*_ts param accepts a relative duration like 90d/2y, YYYY-MM-DD, RFC 3339, or epoch ms)
• REINDEX CONCURRENTLY never blocks writers, but still requires --dangerous --confirm --apply --execute together
• --purge-old-ips deletes user_ips rows last seen before the cutoff; privacy-friendly reaping of per-IP access history, gated with --dangerous --apply
• --purge-orphan-device-inbox deletes device_inbox rows whose (user_id, device_id) no longer exists in devices — the queued to-device messages of deleted devices — in bounded 50k-row chunks (gated with --dangerous --apply); follow with --vacuum-full device_inbox to return the space to the OS. When zero orphans are found, it prints the ownership analysis and the exact next commands instead of a bare "nothing to do"
• workflow device-inbox reports who owns the queue (also used by --diagnose and by zero-orphan purge runs): sampled by default (~1% x100, seconds), --exact does one full scan; exact results are cached under ~/.cache/matrix-surgeon/probes.json for 6h (--no-cache, --refresh, --cache-ttl 1h)
• workflow purge-stale-devices --older-than 90d deletes devices not seen since the cutoff via the admin API (--dangerous --apply only); protected_user_* and bot accounts are excluded unless --include-protected, and it prints the follow-up orphan sweep and --vacuum-full device_inbox commands
• workflow purge-stale-devices --queue-only prunes only queued device_inbox rows of devices idle past the cutoff, in 100k-row SQL chunks (up to 300 per run): no devices or sessions are deleted and nobody is logged out. --vacuum-full runs VACUUM (ANALYZE) (stats refresh), sizes the rewrite from live rows + a sampled row width, and runs VACUUM (FULL, ANALYZE) device_inbox only when free disk covers the estimate plus a 1 GiB margin; otherwise it prints the exact retry commands. --vacuum-full-force skips the estimate (check df yourself first; FULL still needs temp space for the rewritten table)
• --compress-state triggers matrix-synapse-auto-compressor.service over SSH and verifies Result=success; --compress-chunks N instead reproduces the unit's container run directly (same image/env-file/network, one-run-at-a-time container guard). Compression progress is durable; a killed run resumes next time
• probes are safe read-only recipes: table-bloat, largest-tables, sliding-sync-background-updates, device-inbox-stats, device-inbox-orphan-count, device-inbox-ownership-sample, device-inbox-ownership-exact, stale-devices, wal-status, replication-slots, table-size, table-indexes, table-bloat-exact (auto-installs pgstattuple if missing), server-settings, device-lists-changes-census, media-remaining-at-cutoff, local-media-access-census, local-media-never-accessed-months, local-media-recent-rows, remote-media-access-census, old-user-ips

Exit codes match the cron contract: 3 when there is no bloat and no repair action required.

Introspective Ledger

Whenever you run a command with --execute, matrix-surgeon automatically records an introspective journal entry to ~/.local/state/matrix-surgeon/ledger.jsonl (or the path set by $MATRIX_SURGEON_LEDGER_PATH).

Each journal line captures:

• exact invocation details: timestamp, profile, full command path, workflow, --apply flag
• exit code, duration in milliseconds, and outcome badge (ok, nothing, gated, error, interrupted)
• pre-execution and post-execution state snapshots (disk usage via SSH df -P, database entity counts via state-snapshot, and room taxonomy via room-overview-summary for room operations)
• detailed diffs: only keys that changed are highlighted

Use the ledger command to inspect past executions and verify exactly what each run altered:

# View the 20 most recent entries with formatted state diffs
matrix-surgeon ledger

# View recent runs of a specific workflow
matrix-surgeon ledger --workflow clean-dead-rooms

# View only runs that failed or were gated by safety checks
matrix-surgeon ledger --failed

# Export entries as raw JSONL (ideal for piping into jq or auditing tools)
matrix-surgeon ledger --json --last 100

Example ledger output:

2026-09-02 02:30:15 UTC [OK] workflow retention-purge public --mode walk --execute --json (public) - 2410ms
    disk.used_percent: 86.4% → 83.1%
    disk.used_bytes: 46385623040 → 44616253440
    state.local_media_count: 14205 → 13890
    state.local_media_bytes: 12450324480 → 10680954880

2026-09-02 03:00:02 UTC [NOTHING] workflow clean-dead-rooms public --combo empty --execute (public) - 180ms
    [note] No empty / dead rooms found matching filters

To bypass ledger recording for any run, pass the global --no-ledger flag.

Automation & Crontab

See docs/automation.md for the full cron contract (exit-code table, locking, sample crontab for a single-VPS Synapse) and the recommended report → arm → purge rhythm. For audited, copy-paste command sequences (bootstrap checks, confidence pass, exact crontab, 3–6 month maintenance session), see docs/runbook.md.