One Postgres, hybrid search: a knowledge base so the agents stop guessing about the lab

Purpose

The hand-written “map” of the lab drifted, and the chat UI and the Hermes agents kept getting facts wrong. The replacement is a small generated knowledge base: facts, Kubernetes state, plans, repos and the doc site’s own posts, searchable by keyword and by meaning, and served over MCP to both the chat UI and the agents.

It runs inside the Postgres the cluster already had. There is no separate vector database: at this size it would be one more thing to back up for no gain.

Deployment

The MCP server is one Deployment in the k3s cluster, image homelab-kb:0.1.31. The ArgoCD app is Synced and Healthy, and it builds from a dedicated k3s branch of the repo.

Ingest runs on the agent host as a systemd timer, not in the cluster. The daily full pass starts at 05:15. It takes four inputs:

• the daily Docker, ZFS and node facts JSON
• a read-only Kubernetes snapshot (environment variable names only, never Secrets)
• every Gitea repo the bot can read
• the doc site’s public posts

flowchart LR
    A["Facts JSON"]
    B["Kubernetes snapshot"]
    C["Gitea repos"]
    D["Public posts"]
    E["Scrubber (fails closed)"]
    F["Chunk + embed<br/>(Ollama)"]
    G["homelab_kb<br/>in cluster Postgres"]
    H["MCP server"]
    I["Open WebUI + Hermes agents"]
    W["Gitea push webhook"]
    A --> E
    B --> E
    C --> E
    D --> E
    W --> E
    E --> F
    F -->|"pipelines write"| G
    G -->|"agents only read"| H
    H --> I

Configuration

The Postgres image is now pgvector/pgvector:0.8.7-pg16-trixie, on the existing data directory. The knowledge base is its own database, homelab_kb, with three tables: sources, facts and chunks. Open WebUI’s own document uploads use the same pgvector setup in a separate openwebui database; that was verified end to end with a test file.

Two roles keep writers and readers apart. Ingest uses a write role and the server uses a select-only role. An insert as the read role was tested and fails with permission denied.

Search is hybrid. A keyword search (tsvector plus pg_trgm) and a pgvector cosine search are merged by reciprocal rank fusion in one SQL function, kb_search. Lab content is full of hostnames, IPs and tags that embeddings miss, so keyword search carries real weight. The indexes are GIN on the text, trigram, and HNSW cosine on the embedding:

create index chunks_vec on chunks using hnsw (embedding vector_cosine_ops);

Embeddings are nomic-embed-text:v1.5 (768 dimensions) from Ollama on the Intel GPU Docker host, used at ingest and at query time. Without Ollama the search degrades to keyword-only.

Chunking (chunk.py) splits markdown by heading and then by size, YAML per document or top-level key, and plain text by size. The limits are 3200 characters (about 800 tokens) with 400 characters (about 100 tokens) of overlap. Re-embedding is gated on a content hash, so an unchanged source costs nothing.

Everything passes through a scrubber before chunking. It fails closed: it drops secret-looking files and redacts token, password and URL-credential values and long blobs.

For freshness, a Gitea system webhook triggers a re-index of one repo@branch about 2 s after a push, with a 10 s debounce. The receiver checks an HMAC-SHA256 signature. A push to a branch that is not indexed gets a 200 and an INFO log line. Before that change the log filled with ERROR lines for feature and Renovate branches.

Storage

The data lives in the existing Postgres, on its existing data directory. The ingest writes a status file that the health board reads. It raises findings for stale runs, errors and index age per kind.

Size on 2026-11-03 (UTC), from kb_status: 678 repo files, 104 plan files, 71 posts, 19 Kubernetes sources, 16 fact files and 17 report sources. The last full pass finished with 0 errors. The agents’ daily report issues are indexed as kind report, with snapshots kept for 14 days.

Access

The server speaks streamable HTTP MCP over a read-only database session with a 15 s statement timeout. Each client has its own bearer token (Open WebUI and Hermes), so one can be revoked alone. The tools are kb_search, kb_get, kb_facts and kb_status, plus kb_reports, added later.

Open WebUI registers it as an MCP tool server. The Hermes agents (default and docs profiles) use it too. The doc and brief writer skills treat it as the first fact source, under one rule: live tool > knowledge base > docs, and the newer generated_at wins. See the Open WebUI, Postgres MCP, docs pipeline and agents overview posts for those pieces.

Notes

Built on 2026-10-07 and 10-08 in phases. Phase 0, the pgvector swap, took about 20 minutes. All phases were closed by 2026-10-19. The first ingest covered 103 sources, 489 chunks and 361 facts in 42 s. A 10-question eval scored 9/10 on 2026-10-08, and it has not been re-run since. A forced full re-embed on 2026-11-01 took 160 s for 887 sources, 3,362 chunks and 375 facts, with 78 redactions, 0 errors and no chunk without an embedding.

Gotchas:

• Match the Debian suffix, not just the major version. The previous plain Postgres 16 image was the Debian 13 build, and the un-suffixed pgvector tag is Debian 12. Every database logged “collation version mismatch: created with 2.41, OS provides 2.36”. The fix was the -trixie tag plus REINDEX DATABASE on each existing database. The knowledge base databases were created only after the image was right.
• Open WebUI PersistentConfig. RAG_EMBEDDING_ENGINE and RAG_EMBEDDING_MODEL are stored in Open WebUI’s own database. The first-boot values (an empty engine and all-MiniLM-L6-v2) silently overrode the environment until they were set to ollama and nomic-embed-text:v1.5 in the config store.
• Scrubber false positives. A report line token (k3s-cp-1) was redacted although no secret was involved. $(openssl rand …) placeholders are now kept as placeholders. A read-only run of the new scrubber over all live chunks found 0 real secrets.
• Tsquery errors. A query containing a service URL used to fail with a tsquery syntax error. It now returns hits.