CDS Data Pipeline
CDS Data Pipeline
Purpose
A scheduled ETL pipeline that moves data between an object store, a message queue, and downstream delivery endpoints — pulling from S3-compatible object storage and Kafka, validating and reformatting payloads against a schema, and delivering the results over FTPS/SFTP. This is a generalized description of a pipeline I’ve built and maintained professionally, not a homelab deployment.
Architecture
- Ingest — modules pull files from S3-compatible object storage and consume messages from Kafka topics, with support for compressing payloads before handoff downstream.
- Validate/transform — a validation stage checks incoming JSON against a schema before it’s allowed further into the pipeline, catching malformed or out-of-contract payloads before they propagate.
- Delivery — validated output is pushed out over FTPS (explicit and implicit) and SFTP to downstream consumers.
- State tracking — pipeline runs and file transfer status are tracked in a relational database table, so retries and reprocessing pick up cleanly instead of reprocessing everything from scratch.
- Housekeeping — scheduled jobs purge old logs and already-transferred files, and a separate watchdog “touches” files that appear stuck mid-pipeline so they get picked back up.
- Status dashboard — a lightweight HTML/JS dashboard renders pipeline health from a periodically-refreshed status JSON file, so failures are visible without digging through logs.
Scheduling & reliability
Each stage runs as its own systemd service + timer rather than one monolithic script, so ingest, validation, delivery, and cleanup fail and retry independently. The codebase carries a full pytest suite — unit tests per module plus integration tests against real S3, Kafka, SFTP, database, and email targets — run through CI on every push, with linting enforced via a pre-commit hook.
Why this shape
Splitting ingest, validation, delivery, and cleanup into independent, independently-scheduled stages means a slow or failing downstream delivery target doesn’t block ingest, and a bad batch of files can be quarantined at the validation stage without touching the transfer code at all. It’s a pattern that scales down to a single scheduled job and up to a pipeline with real throughput requirements.