Open-source library that integrates dbt and Apache Airflow to implement a functional Data Mesh architecture. Runs dbt models in parallel in Airflow, where each model becomes an independent task while preserving dependencies between domains.
About the Project
DMP.AF (Data Management Platform) is an open-source library that brings together two of the most powerful tools in the modern data stack — dbt and Apache Airflow — and turns the chaos of Data Mesh into a manageable, scalable data ecosystem.
The core idea is simple: dbt is the de facto standard for data transformations, and Airflow is a battle-tested orchestrator. DMP.AF bridges them so that each dbt model becomes an independent Airflow task, running in parallel while preserving all dependencies between models and domains. The result is a production-ready Data Mesh architecture that actually works.
The Problem We Solve
Organisations adopting Data Mesh often run into the same set of painful issues:
- Duplication of transformation and ETL logic across teams
- No transparency — it is unclear what was updated, when, and why
- Every department writes its own pipeline, making maintenance difficult and changes risky
- Manual work, errors, wasted time, and a steady erosion of trust in data
DMP.AF eliminates these problems by providing a unified framework where analytics teams stay in dbt (no Airflow knowledge required), while infrastructure is handled automatically.
Key Features
- Domain-Driven Architecture — separate models by domain into different DAGs, run in parallel — perfect for Data Mesh
- dbt-First Design — all configuration lives in dbt model configs; no need to write Airflow DAGs manually
- Auto-Generated DAGs — automatically creates Airflow DAGs from your dbt project, organised by domain and schedule, with seamless cross-domain dependency handling
- Flexible Scheduling — multiple schedules per model (@hourly, @daily, @weekly, @monthly, and more)
- Idempotent Runs — each model is a separate Airflow task with date intervals passed to every run, guaranteeing reliable backfills and reruns
- Enterprise Features — multiple dbt targets, configurable test strategies, built-in maintenance, and Kubernetes support
- Team-Friendly — analytics teams stay in dbt; infrastructure is handled automatically
How It Works
- Define configuration in your dbt models
- Install DMP.AF as a Python package in your Airflow environment
- DMP.AF reads your dbt project and automatically generates Airflow DAGs
- Each model becomes an independent task with dependencies preserved
- Domains run in parallel, schedules are respected, and everything just works
Pricing
DMP.AF is available in three tiers:
- Open-Source — completely free. Get started right now on GitHub
- Cloud-Based Solution — managed infrastructure starting from £3.14/month
- Bespoke Solution — on-premise deployment with custom modifications and team training
