Setting up a deployment
Many users will want to manage multiple code locations within a single coherent
directory structure. dg facilitates this with the concept of a deployment
directory.
A deployment directory contains a root pyproject.toml containing
deployment-level settings and a code_locations directory containing one or
more code locations. A deployment does not define a Python environment by
default. Instead, Python environments are defined per code location.
To scaffold a new deployment, run:
dg deployment scaffold my-deployment
Creating a Dagster deployment at my-deployment.
Scaffolded files for Dagster project in my-deployment.
This will create a new directory my-deployment. Let's look at the structure:
cd my-deployment && tree
.
├── code_locations
└── pyproject.toml
2 directories, 1 file
Importantly, the pyproject.toml file contains an is_deployment setting
marking this directory as a deployment:
[tool.dg]
is_deployment = true
To add a code location to the deployment, run:
dg code-location scaffold code-location-1
Creating a Dagster code location at /.../my-deployment/code_locations/code-location-1.
Scaffolded files for Dagster project in /.../my-deployment/code_locations/code-location-1.
...
This will create a new directory code-location-1 within the code_locations.
It will also setup a new uv-managed Python environment for the code location. Let's have a look:
tree
.
├── code_locations
│ └── code-location-1
│ ├── code_location_1
│ │ ├── __init__.py
│ │ ├── components
│ │ ├── definitions.py
│ │ └── lib
│ │ └── __init__.py
│ ├── code_location_1_tests
│ │ └── __init__.py
│ ├── pyproject.toml
│ └── uv.lock
└── pyproject.toml
...
code-location-1 also contains a virtual environment directory .venv that is
not shown above. This environment is managed by uv and its contents are
specified in the uv.lock file.
The code-location-1 directory contains a pyproject.toml file that defines
it as a code location and component library:
...
[tool.dagster]
module_name = "code_location_1.definitions"
code_location_name = "code-location-1"
[tool.dg]
is_code_location = true
is_component_lib = true
...
Let's enter this directory and search for registered component types:
cd code_locations/code-location-1 && dg component-type list
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Component Type ┃ Summary ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dagster_components.definitions │ Wraps an arbitrary set of │
│ │ Dagster definitions. │
│ dagster_components.pipes_subprocess_script_collection │ Assets that wrap Python │
│ │ scripts executed with │
│ │ Dagster's │
│ │ PipesSubprocessClient. │
└───────────────────────────────────────────────────────┴────────────────────────────────┘
This is the default set of component types available in every new code
location. We can add to it by installing dagster-components[sling]:
uv add 'dagster-components[sling]'
Due to a bug in sling package metadata, if you are on a macOS machine with
Apple Silicon you may also need to run uv add sling_mac_arm64.
And now we have a new available component:
dg component-type list
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Component Type ┃ Summary ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dagster_components.definitions │ Wraps an arbitrary set of │
│ │ Dagster definitions. │
│ dagster_components.pipes_subprocess_script_collection │ Assets that wrap Python │
│ │ scripts executed with │
│ │ Dagster's │
│ │ PipesSubprocessClient. │
│ dagster_components.sling_replication_collection │ Expose one or more Sling │
│ │ replications to Dagster as │
│ │ assets. │
└───────────────────────────────────────────────────────┴────────────────────────────────┘
As stated above, environments are scoped per code location. dg commands will
only use the environment of code-location-1 when we are inside the
code-location-1 directory.
Let's create another code location to demonstrate this:
cd ../.. && dg code-location scaffold code-location-2
Creating a Dagster code location at /.../my-deployment/code_locations/code-location-2.
Scaffolded files for Dagster project in /.../my-deployment/code_locations/code-location-2.
...
Now we have two code locations. We can list them with:
dg code-location list
code-location-1
code-location-2
And finally, let's check the available component types in code-location-2:
cd code_locations/code-location-2 && dg component-type list
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Component Type ┃ Summary ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ dagster_components.definitions │ Wraps an arbitrary set of │
│ │ Dagster definitions. │
│ dagster_components.pipes_subprocess_script_collection │ Assets that wrap Python │
│ │ scripts executed with │
│ │ Dagster's │
│ │ PipesSubprocessClient. │
└───────────────────────────────────────────────────────┴────────────────────────────────┘
As you can see, we are back to only the default list of component types. This
is because we are now using the environment of code-location-2, in which we
have not installed dagster-components[sling].
For a final step, let's load up our two code locations with dg dev. dg dev
will automatically recognize the code locations in your deployment and launch
them in their respective environments. Let's run dg dev back in the
deployment root directory and load up the Dagster UI in your browser:
cd ../.. && dg dev
