About This Talk
At SmartData 2024, Nikita Yurasov and Leonid Kozhinov presented the open-source dbt-af library — a year after sharing the initial approach to integrating dbt and Airflow at Toloka.ai. The library enables seamless integration between dbt and Airflow, turning the data mesh concept from hype into a working production system.
Key Ideas
System Architecture — The talk opened with a refresher on the overall data platform setup and where data mesh, dbt, and Airflow fit together. Understanding the architecture is essential before diving into the tooling.
Competitive Landscape — A review of alternative solutions for dbt-Airflow integration. There aren’t many direct competitors to dbt-af, and the talk covered why existing options fell short of production requirements.
dbt-af in Practice — A live demonstration of how conveniently dbt-af handles its intended tasks: automatic DAG generation from dbt models, dependency resolution across domains, and seamless orchestration through Airflow.
Who Benefits — The library addresses a common organizational tension: DWH teams under pressure from analysts who want faster iteration, and analysts frustrated by slow-moving DWH teams. dbt-af gives both sides the tools to work independently while maintaining data quality.
Why It Matters
Data mesh promises domain-oriented data ownership, but practical implementations require solid tooling. dbt-af bridges the gap between dbt’s transformation power and Airflow’s orchestration capabilities, making data mesh achievable with widely available open-source tools.
