Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to use a remote PostgreSQL database not using the official postgres image. #371

Open
tessierp opened this issue Nov 15, 2022 · 5 comments
Open

How to use a remote PostgreSQL database not using the official postgres image. #371

tessierp opened this issue Nov 15, 2022 · 5 comments
Labels
documentation

Comments

@tessierp
Copy link

tessierp commented Nov 15, 2022 •
edited by alexmv
Loading

Hi,

Failed to start ZULIP, it fails to init the databaase, anyone knows how to fix this issue?

LOGs from Docker :

No additional puppet configuration executed for queue workers.
No additional puppet configuration executed for load balancer IPs.
Setting database configuration ...
Setting key "REMOTE_POSTGRES_HOST", type "string" in file "/etc/zulip/settings.py".
Setting key "REMOTE_POSTGRES_PORT", type "string" in file "/etc/zulip/settings.py".
Setting key "REMOTE_POSTGRES_SSLMODE", type "string" in file "/etc/zulip/settings.py".
Database configuration succeeded.
Setting Zulip secrets ...
Generating Zulip secrets ...
generate_secrets: No new secrets to generate.
Secrets generation succeeded.
Zulip secrets configuration succeeded.
Activating authentication backends ...
Setting key "AUTHENTICATION_BACKENDS", type "array" in file "/etc/zulip/settings.py".
Adding authentication backend "EmailAuthBackend".
Authentication backend activation succeeded.
Executing Zulip configuration ...
Setting key "EXTERNAL_HOST", type "string" in file "/etc/zulip/settings.py".
Setting key "MEMCACHED_LOCATION", type "string" in file "/etc/zulip/settings.py".
Setting key "RABBITMQ_HOST", type "string" in file "/etc/zulip/settings.py".
Setting key "RABBITMQ_USER", type "string" in file "/etc/zulip/settings.py".
Setting key "RATE_LIMITING", type "bool" in file "/etc/zulip/settings.py".
Setting key "REDIS_HOST", type "string" in file "/etc/zulip/settings.py".
Setting key "REDIS_PORT", type "integer" in file "/etc/zulip/settings.py".
Setting key "ZULIP_ADMINISTRATOR", type "string" in file "/etc/zulip/settings.py".
Zulip configuration succeeded.
Auto backup enabled.
=== End Initial Configuration Phase ===
=== Begin Bootstrap Phase ===
Waiting for database server to allow connections ...
Executing Zulip first start init ...
+++ readlink -f /home/zulip/deployments/current/scripts/setup/initialize-database
++ dirname /home/zulip/deployments/2022-08-24-23-56-24/scripts/setup/initialize-database
+ THIS_DIR=/home/zulip/deployments/2022-08-24-23-56-24/scripts/setup
+ cd /home/zulip/deployments/2022-08-24-23-56-24/scripts/setup/../..
+ ./manage.py checkconfig
+ ./manage.py migrate --noinput
Operations to perform:
  Apply all migrations: analytics, auth, confirmation, contenttypes, otp_static, otp_totp, sessions, social_django, two_factor, zerver
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
Traceback (most recent call last):
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
  File "/home/zulip/deployments/2022-08-24-23-56-24/zerver/lib/db.py", line 34, in execute
    wrapper_execute(self, super().execute, query, vars)
  File "/home/zulip/deployments/2022-08-24-23-56-24/zerver/lib/db.py", line 19, in wrapper_execute
    action(sql, params)
psycopg2.errors.ConfigFileError: could not open dictionary file "/usr/share/postgresql/14/tsearch_data/en_us.dict": No such file or directory
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "./manage.py", line 157, in <module>
    execute_from_command_line(sys.argv)
  File "./manage.py", line 122, in execute_from_command_line
    utility.execute()
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 105, in database_forwards
    self._run_sql(schema_editor, self.sql)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/migrations/operations/special.py", line 130, in _run_sql
    schema_editor.execute(statement, params=None)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/base/schema.py", line 145, in execute
    cursor.execute(sql, params)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/srv/zulip-venv-cache/e5128e116bae759c7692fa7c8bd0a48e9970619d/zulip-py3-venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
  File "/home/zulip/deployments/2022-08-24-23-56-24/zerver/lib/db.py", line 34, in execute
    wrapper_execute(self, super().execute, query, vars)
  File "/home/zulip/deployments/2022-08-24-23-56-24/zerver/lib/db.py", line 19, in wrapper_execute
    action(sql, params)
django.db.utils.InternalError: could not open dictionary file "/usr/share/postgresql/14/tsearch_data/en_us.dict": No such file or directory
  Applying zerver.0001_initial...Zulip first start database initi failed in "initialize-database" exit code 1. Exiting.`
@tessierp
Copy link
Author

Just want to update, I tried this yesterday with version 6.0 and still the same issue.

@kerray
Copy link

kerray commented Jan 20, 2023 •
edited
Loading

Wanted to try Zulip in Docker with my existing Postgres server and it fails on this.

It seems the Postgres server for Zulip used to need to have Zulip tsearch_extra extensions installed, but the extension itself is archived and deemed no longer necessary - zulip/tsearch_extras#7

SOLVED: Downloading the en_us.dict and en_us.affix files from https://github.com/postgrespro/hunspell_dicts/tree/master/hunspell_en_us and zulip_english.stop from zulip-server release, path zulip-server-6.0.tar\zulip-server-6.0\puppet\zulip\files\postgresql\zulip_english.stop and mounting them as binds inside the Postgres container at /usr/local/share/postgresql/tsearch_data/filename helped!

@stratosgear
Copy link

stratosgear commented Mar 28, 2023 •
edited
Loading

Hmmm, although I have correctly mounted the required files in place, I still get a:

  File "/home/zulip/deployments/2023-01-23-18-51-59/zerver/lib/db.py", line 34, in execute
    wrapper_execute(self, super().execute, query, vars)
  File "/home/zulip/deployments/2023-01-23-18-51-59/zerver/lib/db.py", line 19, in wrapper_execute
    action(sql, params)
django.db.utils.InternalError: could not open dictionary file "/usr/local/share/postgresql/tsearch_data/en_us.dict": No such file or directory
  Applying zerver.0001_initial...Zulip first start database initi failed in "initialize-database" exit code 1. Exiting.

I believe some script or process deletes them.... Or it access rights? I swear, the files are correctly mounted!

EDIT:

There is some "black magic" taking place at https://github.com/zulip/zulip/blob/main/puppet/zulip/manifests/postgresql_base.pp#L46 These appear to be symlinked from another location (/var/cache/postgresql/dicts in case of Debian, line 20) that they appear to be empty on my docker Container. And this is where I run out of time investigating further... :(

@stratosgear
Copy link

Not sure how this works... :(

Found a POSTGRESQL_MISSING_DICTIONARIES parameter but passing it as SETTING_POSTGRESQL_MISSING_DICTIONARIES = "True|False" in the environment variables of the zulip container, does not seem to make any difference regardless if set to True or False.

Also if zulip/tsearch_extras#7 is correct and tsearch_extras is not required, why are we still not able to use a plain external DB...?

@timabbott
Copy link
Member

You can use a normal external database, but you should ideally add dictionary files for it. Unfortunately, postgres doesn't support providing a dictionary file over the SQL protocol; for unknown reasons, it must be a file on disk on the database server host. I have no idea why they designed it that way.

https://zulip.readthedocs.io/en/latest/production/postgresql.html#cloud-provider-managed-postgresql-e-g-amazon-rds documents this detail.

That said, if you're using Docker postgres container, you might as well use the one that Zulip distributes with the relevant files already installed.

#371 (comment) has what I believe are correct instructions if you're using another postgres database server; maybe we can usefully document that; I'll leave this issue open for that work. Note that you need the dictionary files on the postgres server, not on the zulip container.

I expect SETTING_POSTGRESQL_MISSING_DICTIONARIES should work for folks who need that; we're happy to help anyone having trouble with that debug in chat.zulip.org.

@timabbott timabbott changed the title Failed to start ZULIP How to use a remote PostgreSQL database not using the official postgres image. Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kerray @stratosgear @timabbott @tessierp