DBT : Organizing DBT Models in Subdirectories: A Guide to YAML Configuration


DBT (Data Build Tool) is an essential tool for data engineers and analysts to build, test, and document data pipelines using SQL. DBT projects rely on YAML files for configuration and organization. In this article, we will explore how to organize your DBT models into subdirectories and appropriately configure your YAML files.

1. Organizing Models in Subdirectories

To create a more structured and maintainable DBT project, you can organize your models into subdirectories within the models folder. Grouping related models into subdirectories allows for easier navigation and project management.

  1. Creating a More Structured Project: When you work on a data transformation project using DBT, you often have multiple SQL models that represent different aspects of your data analysis or reporting. To maintain order and clarity, it’s a good idea to structure your project in an organized way.
  2. Organizing Models into Subdirectories: The recommendation is to group related models into subdirectories within the “models” folder of your DBT project. This means that instead of having all your SQL models in a single flat directory, you create subfolders to categorize and group models based on their purpose, source, or any other relevant criteria.
  3. Benefits:
    • Easier Navigation: Organizing models into subdirectories makes it much easier to find specific models when you’re working on them or need to reference them.
    • Project Management: It simplifies project management because related models are grouped together, which can be especially helpful in larger projects.
    • Improved Clarity: A structured organization improves the clarity of your project’s layout, making it more understandable for you and your team members.

For example, you can create the following subdirectories for a project:

    ├── customers/
    │   ├── dim_customers.sql
    │   └── fct_customer_orders.sql
    ├── orders/
    │   ├── dim_orders.sql
    │   └── fct_order_items.sql
    └── products/
        ├── dim_products.sql
        └── fct_product_sales.sql

In this example, we have three subdirectories: customers, orders, and products. Each subdirectory contains related models.

2. Configuring YAML Files for SubdirectoriesTo apply configurations for models in specific subdirectories, create a YAML file within each subdirectory. This file will contain the configurations that apply to all models within that subdirectory.

For example, create a schema.yml file within the customers subdirectory:


version: 2

  - name: dim_customers
    description: Dimension table containing customer information
      - name: customer_id
        description: Unique identifier for customers
          - unique
          - not_null

  - name: fct_customer_orders
    description: Fact table containing customer order information
      - name: order_id
        description: Unique identifier for orders
          - unique
          - not_null
Repeat this process for each subdirectory, tailoring the YAML configuration to the models within that subdirectory.

3: Configuring the DBT Project File
In the main dbt_project.yml file, ensure that your source paths include the subdirectories. For example:


name: my_project
version: 1.0

profile: my_profile

  - models/customers
  - models/orders
  - models/products

  - tests

This configuration informs DBT to look for models within the specified subdirectories.

Get more useful articles on dbt

  1. ,
Author: user

Leave a Reply