dagster - How to start the grpc server when definitions are in a module

Question

My project structure looks like this:

• /definitions (for all dagster python definitions)
  • __init__.py
  • repositories.py
  • /exchangerates
    • pipelines.py
    • ...
  • ...
• workspace.yaml

I've tried running the grpc server using various methods, especially the following (started in project root):

• dagster api grpc -h 0.0.0.0 -p 4000 -f definitions/repositories.py
• dagster api grpc -h 0.0.0.0 -p 4000 -m definitions
• dagster api grpc -h 0.0.0.0 -p 4000 -m definitions.repositories

The first command yields the following error:

dagster.core.errors.DagsterImportError: Encountered ImportError: attempted relative import with no known parent package while importing module repositories from file C:UsersKlausPycharmProjectsdagsterexchangeratesdefinitions epositories.py. Consider using the module-based options -m for CLI-based targets or the python_package workspace.yaml target.

The second and third command yield the following error:

(stacktrace comes before this)
ModuleNotFoundError: No module named 'definitions'

How can this be solved?

EDIT: I've uploaded the current version of the example I'm working on to GitHub: https://github.com/kstadler/dagster-exchangerates

EDIT2: Reflected changes in directory structure

question from:https://stackoverflow.com/questions/65860584/how-to-start-the-grpc-server-when-definitions-are-in-a-module

Answer 1

sorry about the trouble - there are a couple of options here to get your server running.

To get it working with the '-f' option, the relative imports need to be replaced with absolute imports. That would look like this:

-from .pipelines import exchangerates_pipline
-from .partitions import year_partition_set
+from definitions.pipelines import exchangerates_pipline
+from definitions.partitions import year_partition_set

(This is the same error you would get if you tried to run python definitions/repositories.py directly).

I'm still digging into why exactly the third '-m' option isn't working the way I'd expect it to. Curiously the following command seems to work for me which should be close to identical:

python -m dagster.grpc -h 0.0.0.0 -p 4000 -m definitions.repositories

Incidentally, your example contains a workspace.yaml that should cause dagit and other Dagster processes to automatically start up the gRPC server for you for that module - so depending on your goal you may not need to run dagster api grpc yourself at all.