Autoloading existing Dagster definitions
This guide covers using existing Dagster definitions with a dg-compatible project. To convert an existing project to use dg, see "Converting an existing project to use dg".
In projects created with create-dagster, all definitions are typically kept in the defs/ directory. However, if you've converted an existing project to use dg, you may have definitions located in various other modules. This guide will show you how to move these existing definitions into the defs directory in a way that will allow them to be automatically loaded.
Example project structure
Let's walk through an example of migrating your existing definitions, with a project that has the following structure:
tree
.
├── my_existing_project
│ ├── __init__.py
│ ├── analytics
│ │ ├── __init__.py
│ │ ├── assets.py
│ │ └── jobs.py
│ ├── definitions.py
│ ├── defs
│ │ └── __init__.py
│ └── elt
│ ├── __init__.py
│ ├── assets.py
│ └── jobs.py
├── pyproject.toml
└── README.md
5 directories, 11 files
At the top level, we load definitions from various modules:
from pathlib import Path
import dagster as dg
import my_existing_project.defs
from my_existing_project.analytics import assets as analytics_assets
from my_existing_project.analytics.jobs import (
regenerate_analytics_hourly_schedule,
regenerate_analytics_job,
)
from my_existing_project.elt import assets as elt_assets
from my_existing_project.elt.jobs import sync_tables_daily_schedule, sync_tables_job
defs = dg.Definitions.merge(
dg.Definitions(
assets=dg.load_assets_from_modules([elt_assets, analytics_assets]),
jobs=[sync_tables_job, regenerate_analytics_job],
schedules=[sync_tables_daily_schedule, regenerate_analytics_hourly_schedule],
),
dg.components.load_from_defs_folder(project_root=Path(__file__).parent.parent),
)
Each of these modules contains a variety of Dagster definitions, including assets, jobs, and schedules.
Let's migrate the elt module to a component.
Move definitions to defs
We'll start by moving the top-level elt module into defs/elt:
mv my_existing_project/elt/* my_existing_project/defs/elt
Now that our definitions are in the defs directory, we can update the root definitions.py file to no longer explicitly load the elt module's Definitions:
from pathlib import Path
from my_existing_project.analytics import assets as analytics_assets
from my_existing_project.analytics.jobs import (
regenerate_analytics_hourly_schedule,
regenerate_analytics_job,
)
import dagster as dg
defs = dg.Definitions.merge(
dg.Definitions(
assets=dg.load_assets_from_modules([analytics_assets]),
jobs=[regenerate_analytics_job],
schedules=[regenerate_analytics_hourly_schedule],
),
dg.load_from_defs_folder(project_root=Path(__file__).parent.parent),
)
Our project structure now looks like this:
tree
.
├── my_existing_project
│ ├── __init__.py
│ ├── analytics
│ │ ├── __init__.py
│ │ ├── assets.py
│ │ └── jobs.py
│ ├── definitions.py
│ └── defs
│ ├── __init__.py
│ └── elt
│ ├── __init__.py
│ ├── assets.py
│ └── jobs.py
├── pyproject.toml
├── README.md
└── uv.lock
5 directories, 12 files
The load_defs command in our definitions.py file will automatically load any definitions found within the defs module. This means that all of our definitions are now automatically loaded, with no need to import them up into any top-level organization scheme.
We can repeat the same process for our other modules.
Fully migrated project structure
Once each of our definitions modules are migrated, our project is left with a standardized structure:
tree
.
├── my_existing_project
│ ├── __init__.py
│ ├── definitions.py
│ └── defs
│ ├── __init__.py
│ ├── analytics
│ │ ├── __init__.py
│ │ ├── assets.py
│ │ └── jobs.py
│ └── elt
│ ├── __init__.py
│ ├── assets.py
│ └── jobs.py
├── pyproject.toml
├── README.md
└── uv.lock
5 directories, 12 files
Our project root now only constructs definitions from the defs module:
from pathlib import Path
import dagster as dg
defs = dg.load_from_defs_folder(project_root=Path(__file__).parent.parent)
We can run dg list defs to confirm that all of our definitions are being loaded correctly:
dg list defs
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Section ┃ Definitions ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Assets │ ┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Group ┃ Deps ┃ Kinds ┃ Description ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━┩ │
│ │ │ customers_table │ default │ │ │ │ │
│ │ ├───────────────── ───┼─────────┼──────┼───────┼─────────────┤ │
│ │ │ my_analytics_asset │ default │ │ │ │ │
│ │ ├────────────────────┼─────────┼──────┼───────┼─────────────┤ │
│ │ │ orders_table │ default │ │ │ │ │
│ │ ├────────────────────┼─────────┼──────┼───────┼─────────────┤ │
│ │ │ products_table │ default │ │ │ │ │
│ │ └────────────────────┴─────────┴──────┴───────┴─────────────┘ │
│ Jobs │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Description ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩ │
│ │ │ regenerate_analytics_job │ │ │
│ │ ├──────────────────────────┼─────────────┤ │
│ │ │ sync_tables_job │ │ │
│ │ └──────────────────────────┴─────────────┘ │
│ Schedules │ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ │
│ │ ┃ Key ┃ Cron ┃ │
│ │ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │
│ │ │ regenerate_analytics_job_schedule │ 0 * * * * │ │
│ │ ├───────────────────────────────────┼───────────┤ │
│ │ │ sync_tables_job_schedule │ 0 0 * * * │ │
│ │ └───────────────────────────────────┴───────────┘ │
└───────────┴───────────────────────────────────────────────────────────────┘