Python Media Indexer

Posted May 24, 2026 Updated Sep 7, 2026

By Brad Jones

9 min read

Self-hosted media library management and encoding queue. The service scans watch directories for video files (.mp4, .m4v, .mkv), records codec/resolution/track metadata to PostgreSQL via MediaInfo, surfaces an encoding queue for files that do not match the configured target codec/extension/subtitle-language rules, and moves files to resolution-bucketed HandBrake ingest directories. A background thread inside the backend container watches the HandBrake output directory and moves completed encodes back automatically.

An optional auto-encode feature lets the worker drain the encoding queue unattended, one file at a time. The UI is a dark-mode HTML/JS frontend served by nginx.

Architecture

Browser
  └── nginx  (port 80, frontend_container)
        ├── serves  compose/html/  (static UI pages)
        └── proxies /api/*  →  FastAPI backend  (port 8000)
                                  ├── main.py        API routes + lifespan startup/recovery
                                  ├── encoder.py     Move-queue thread + encoding helpers
                                  ├── worker.py      Output-watcher + reanalysis + auto-encode
                                  ├── analyzer.py    MediaInfo parse + PostgreSQL upsert
                                  ├── cache.py       Redis get/set/delete with rate-limited error logging
                                  ├── config.py      INI read + MEDIA_* env-var overlay
                                  ├── auth.py        Basic Auth → session cookie; admin/viewer roles
                                  └── db_context.py  ThreadedConnectionPool (max 20)
                                          │
                                          ├── PostgreSQL 18  (external, on proxy network)
                                          │     ├── media_files
                                          │     ├── media_file_tracks
                                          │     ├── encode_jobs
                                          │     └── users / sessions
                                          │
                                          └── Redis 7  (redis container)
                                                ├── media:watchdirs_content   TTL 60 s
                                                ├── media:encoding_queue      TTL 30 s
                                                ├── media:encode_jobs         TTL 25 s
                                                ├── media:output_files        TTL 10 s
                                                └── media:config_media        TTL 300 s

Container Summary

Container	Image	Role
`frontend_container`	`nginx:alpine-slim`	Static UI + `/api/*` reverse proxy
`backend`	Custom Python 3.13-slim image	FastAPI + uvicorn + output-watcher thread
`redis`	`redis:7-alpine`	Response cache; TTLs 10–300 s per endpoint

PostgreSQL 18 is external to the compose stack — it runs separately and is reached via the shared proxy Docker network. All three compose services join that same network. The backend waits for redis to pass its health check (redis-cli ping) before starting.

Database schema (tables, indexes, ENUM types, migrations) is initialized idempotently on every backend startup by db_init.py’s ensure_schema(). No manual SQL import is required.

Backend Module Map

File	Role
`main.py`	FastAPI app, all API route handlers, lifespan startup/recovery
`encoder.py`	Move-queue thread (`queue_encode_job`), `build_encoding_queue_query()`, `get_dest_path()`
`worker.py`	Output-watcher loop, flag-based re-analysis, auto-encode step, reanalysis scheduler
`analyzer.py`	MediaInfo parsing (`_parse_mediainfo`) and PostgreSQL upsert (`parse_and_upsert`)
`cache.py`	Redis get/set/delete; errors suppressed with rate-limited warnings (one per 60 s)
`config.py`	INI read + env-var overlay; exposes `get_cfg()` and `env_controlled_fields()`
`db_context.py`	`ThreadedConnectionPool` (min 1, max 20); `db_conn(cfg)` context manager
`auth.py`	HTTP Basic Auth → session cookie; two groups: `admin` (read/write) and `viewer` (GET only)
`utils.py`	`json_error`, `internal_error`, `split_tracks`, `read_file_tail`, `get_watch_dirs()`
`db_init.py`	Idempotent schema bootstrap; `ensure_schema()` called on every startup

Worker Thread

worker.py runs as a daemon thread inside the backend container — not a separate container. Each poll cycle has four phases, in order:

Reanalysis scheduler check — maybe_trigger_reanalyze() fires if the configured interval_hours has elapsed since the last bulk re-flag.
Stable-file mover — scans the output directory for files unchanged for stable_seconds (default 300 s), matches them to encode_jobs by basename/stem, commits the DB cleanup row before calling shutil.move, then calls parse_and_upsert() directly to re-analyze the moved file.
Flag-based re-analysis — fetches up to chunk_size (default 50) media_files rows where analyzed_flag = TRUE, calls parse_and_upsert() for each, and clears the flag on success. Files currently in encode_jobs are skipped.
Auto-encode step — if [auto_encode] enabled = true, submits one file from the encoding queue to the move-queue (pauses if any job is already in-flight).

A separate heartbeat sub-thread writes HEARTBEAT <unix_ts> to worker.log every 10 seconds. /api/worker-health reads that timestamp to report liveness.

Encoding Queue

There is no SQL VIEW. The queue is built at runtime from a parameterized query in encoder.build_encoding_queue_query() driven by three [media] config values:

final_codec — comma-separated codec substrings (case-insensitive). Files whose codec matches any fragment are excluded.
final_file_extension — files whose basename does not end with this are included.
keep_subtitle_languages — files with at least one subtitle track in a language not listed here are included.

Only files with a known resolution bucket (480 / 576 / 720 / 1080 / 2160) appear in the queue.

Auto-Encode

When [auto_encode] enabled = true, auto_encode_step() runs at the end of each worker poll cycle. It:

Checks encode_jobs for any row with status NOT IN ('error') — pending, copying, or complete. If one exists, it returns immediately. A complete row means the file is in the HandBrake ingest bucket and HandBrake has not yet finished; only once the worker moves the output file back (and deletes the encode_jobs row) is the next file submitted.
Queries the encoding queue (via build_encoding_queue_query) excluding files that already have an encode_jobs row. Ordering is controlled by sort_order (size_asc = smallest first, size_desc = largest first).
Picks the first result, inserts an encode_jobs row (ON CONFLICT … DO UPDATE to reset stale entries), and calls queue_encode_job().

Set MEDIA_AUTO_ENCODE_ENABLED=false (or toggle in the UI) to stop it at any time; any copy already started will still complete.

Quick Start

Prerequisites

Docker + Docker Compose
External Docker network named proxy

docker network create proxy

1. Configure the environment

cp compose/.env.example compose/.env

Edit compose/.env and fill in at minimum:

MEDIA_DB_USER=<your-db-user>
MEDIA_DB_PASSWORD=<your-db-password>

Also set MEDIA_REDIS_HOST=redis (and MEDIA_REDIS_PORT=6379) so the backend can reach the Redis container. All other variables have working defaults.

2. Deploy

./build_app.sh

Or manually:

  
docker build -t <registry>/media-indexer:latest app/
docker push <registry>/media-indexer:latest
cd compose && docker compose up -d --force-recreate

3. Access

Service	URL
Web UI	`http://<host>/`
API docs (FastAPI)	`http://<host>/docs`
Backend API direct	`http://<host>:8000/api/`

First run: A default admin account is seeded if the users table is empty. Change the default password immediately via the Users page (/users.html) or PATCH /api/users/{id}.

Configuration

Configuration is read from /config/media_indexer.ini inside the backend container. The file is optional — all settings can be supplied entirely through MEDIA_* environment variables, which take precedence over any INI value.

Key environment variables

Env var	Default	Description
`MEDIA_DB_DBNAME`	`media`	PostgreSQL database name
`MEDIA_DB_USER`	—	PostgreSQL user
`MEDIA_DB_PASSWORD`	—	PostgreSQL password
`MEDIA_DB_HOST`	`postgres`	PostgreSQL host
`MEDIA_WATCH_DIRS`	—	Comma-separated watch directories
`MEDIA_OUTPUT_DIR`	—	HandBrake output directory
`MEDIA_2160P_WATCH`	—	HandBrake ingest dir for 2160p
`MEDIA_1080P_WATCH`	—	HandBrake ingest dir for 1080p
`MEDIA_720P_WATCH`	—	HandBrake ingest dir for 720p
`MEDIA_FINAL_EXT`	`.mkv`	Target file extension
`MEDIA_FINAL_CODEC`	`hevc,hvc1,…`	Codec substrings indicating already-encoded files
`MEDIA_KEEP_LANGS`	`eng,en,en-us`	Subtitle languages to keep
`MEDIA_WORKER_POLL_INTERVAL`	`30`	Output-dir poll interval (seconds)
`MEDIA_WORKER_STABLE_SECS`	`300`	Min seconds unchanged before a file is “stable”
`MEDIA_REDIS_HOST`	—	Redis hostname; if unset, caching is disabled
`MEDIA_REANALYZE_INTERVAL_HOURS`	`0`	Hours between auto bulk re-analysis; `0` = off
`MEDIA_AUTO_ENCODE_ENABLED`	`false`	Set to `true` to enable automatic queue draining
`MEDIA_AUTO_ENCODE_SORT_ORDER`	`size_asc`	`size_asc` (smallest first) or `size_desc`
`MEDIA_AUTH_MODE`	`basic`	Auth backend: `basic` or `tinyauth`
`MEDIA_SESSION_TTL_HOURS`	`0`	Session lifetime; `0` = never expire

Authentication

All routes except GET /api/health, GET /api/auth/challenge, and POST /api/auth/logout require authentication.

Flow: The login page (/login.html) posts to POST /api/auth/login with form fields (username, password). The backend verifies the password against a bcrypt hash in the users table, creates a server-side session, and returns an HttpOnly SameSite=Strict cookie (session_id). Subsequent requests use the cookie (fast UUID lookup, no bcrypt round-trip). Direct HTTP Basic Auth to any protected endpoint is also supported.

Roles:

Role	Permissions
`admin`	Full read/write access to all endpoints, including user management
`viewer`	GET / HEAD / OPTIONS only; `405` on any mutation

Changing a user’s password or setting is_active = false immediately invalidates all existing sessions for that user. The last active admin account is protected — it cannot be deleted or demoted to viewer.

API Overview

Interactive docs (FastAPI auto-generated): http://<host>/docs

Health and diagnostics

Method	Path	Description
GET	`/api/health`	Liveness check; always returns `{"status": "ok"}`
GET	`/api/dbcheck`	Tests the PostgreSQL connection
GET	`/api/worker-health`	Reports output-watcher and move-worker liveness via heartbeat

Encoding

Method	Path	Description
GET	`/api/encoding-queue`	Files that need encoding per current `[media]` rules
POST	`/api/encode`	Queues files for encoding; max 500 paths per request
GET	`/api/encode-jobs`	All encode job records, newest first
DELETE	`/api/encode-jobs/completed`	Remove all `complete`-status job records (admin only)
GET	`/api/output-files`	Files in the configured output directory

Media

Method	Path	Description
GET	`/api/watchdirs-content`	Lists all video files under each watch directory
GET	`/api/file-info?path=…`	Runs MediaInfo, upserts to DB, returns codec/resolution/track detail
POST	`/api/reanalyze-trigger`	Immediately flags all watch-dir files for re-analysis (admin only)

Logs

GET /api/service-log?service=<name>&lines=<n>

Valid values for service: backend, sql, worker, nginx_access, nginx_error. Returns the last lines lines (default 300) as plain text.

Database Schema

Schema is initialized by db_init.py’s ensure_schema() on every backend startup. All DDL is idempotent (CREATE TABLE IF NOT EXISTS, ALTER TABLE … ADD COLUMN IF NOT EXISTS, etc.). No manual SQL import is required.

`media_files`

One row per unique file path.

Column	Type	Notes
`id`	`SERIAL PK`
`path`	`TEXT UNIQUE`	Absolute path
`basename`	`TEXT`	Filename only
`size_bytes`	`BIGINT`
`codec`	`TEXT`	Raw codec ID from MediaInfo
`media_type`	`TEXT`	`movie` or `show` (inferred from path)
`resolution`	`INTEGER`	Actual video height in pixels
`resolution_bucket`	`INTEGER`	Bucketed to 480 / 576 / 720 / 1080 / 2160
`last_modified`	`TIMESTAMP`	File mtime
`run_at`	`TIMESTAMP`	Last analysis timestamp
`analyzed_flag`	`BOOLEAN`	`TRUE` = queued for re-analysis by worker

`encode_jobs`

Lifecycle record for each file sent for encoding.

Column	Type	Notes
`id`	`SERIAL PK`
`media_id`	`INTEGER FK`	References `media_files.id`; nullable (`ON DELETE SET NULL`)
`source_path`	`TEXT UNIQUE`	Original file path
`dest_path`	`TEXT`	Path in the resolution ingest directory
`status`	`TEXT`	`pending` → `copying` → `complete` / `error`
`progress`	`INTEGER`	Copy progress 0–100
`queued_at`	`TIMESTAMP`

Logs

All services write to the shared compose/logs/ volume.

File	Source	Contents
`backend.log`	FastAPI	HTTP request log + application messages (RotatingFileHandler, 100 MB, 5 backups)
`sql.log`	FastAPI	SQL queries at DEBUG level
`worker.log`	worker thread	Poll activity, file moves, re-analysis events, `HEARTBEAT <ts>` lines
`nginx_access.log`	nginx	ISO 8601 access log
`nginx_error.log`	nginx	nginx errors

All Python log output is also mirrored to stdout for docker logs.

Commands

  
# Lint (Python)
./run_lint.sh

# Lint (HTML)
./run_html_lint.sh

# Tests (safe API suite — default)
./run_tests.sh

# Tests with Playwright browser tests
./run_tests.sh --browser

# Tests with real file moves (destructive)
./run_tests.sh --destructive

# Build + push + restart stack
./build_app.sh

Tests are integration-only and require a live backend at http://localhost:8000/api. They do not mock any dependencies and do not touch real media files — all encode tests use nonexistent paths or paths outside configured watch directories.

Known Issues / Roadmap

Issue	Title	Status
#1	Add OIDC authentication via TinyAuth (forward-auth)	Open
#15	Auto-encoder TOCTOU race with manual encode submissions	Open — fix planned: merge check+insert into a single transaction with `INSERT … SELECT … WHERE NOT EXISTS` guard

homelab

This post is licensed under CC BY 4.0 by the author.

Python Media Indexer

Architecture

Container Summary

Backend Module Map

Worker Thread

Encoding Queue

Auto-Encode

Quick Start

Prerequisites

1. Configure the environment

2. Deploy

3. Access

Configuration

Key environment variables

Authentication

API Overview

Health and diagnostics

Encoding

Media

Logs

Database Schema

media_files

encode_jobs

Logs

Commands

Known Issues / Roadmap

Trending Tags

`media_files`

`encode_jobs`