Multiple Dagster Instances on same machine?

[SgtGilly]

Hello. Are you able to have more than one dagster instance running on the same machine? Could two developers, using seperate user accounts and working out of their respective home directories each deploy their own instance and not step on eachother?

And if so, what is a correct way to configure the dagster.yaml to avoid collisions?

[prha]

You can have more than one dagster instance running on the same machine, if the DAGSTER_HOME directories are separate, each with their own dagster.yaml file. If you are using the default sqlite-based storage, having them in different directories will ensure separation.

You will have to use different ports when running the webservers, to ensure that they do not collide.

[SgtGilly]

So for sqlite-based storage, I have specified /var/log/dagster/user_1 as the directory in the first dagster.yaml and /var/log/dagster/user_2 in the other dagster.yaml but I am getting an OperationalError: database is locked sqlite error when trying to execute a job in each dagster instance at the same time. Do I need them to be in entirely different directories?

[prha]

Just so that I'm clear, you have two different DAGSTER_HOME env var values for each instance, which point to different directories that each contain their own dagster.yaml. The dagster.yaml configuration is identical, except that the base_dir value provided to the storage -> sqlite config points to two different directories? That sounds like it should work. Can you confirm that sqlite files are getting created in each of the user directories?

[SgtGilly]

Yes you are correct in that it should work. I found that my issue is in the dagster.yaml, for storage I was doing

storage:
  sqlite:
    base_dir: /var/log/dagster/${USER}

But it looks like I cant do ${USER} in dagster.yaml. Is there a way to extract the value of env variable USER in this regard?
I could use

storage:
  sqlite:
    base_dir: 
       env: CUSTOM_USER_ENV_VAR

but I wanted to avoid making env variables and having to programatically update the dagster.yamls I create for each user

[prha]

I think that programmatically setting env vars is the best option you have.

You might also consider subclassing the sqlite storage instance that reads the env var into Python to set the base path, and pointing all yaml files to this new storage class.