sqlalchemy-redshift seems dormant and does not seem to be compatible with current environments despite being recommended by dvc

[DaveParr]

import-db docs reference sqlalchemy-redshift which seems to be an unsupported/dormant/soon to be archived package.

I also believe due to version constraints it's not possible to install the dependency in 'current' projects since sqlalchemy went 2.0.0:

poetry add sqlalchemy-redshift
Using version ^0.8.14 for sqlalchemy-redshift

Updating dependencies
Resolving dependencies... (0.0s)

Because no versions of sqlalchemy-redshift match >0.8.14,<0.9.0
 and sqlalchemy-redshift (0.8.14) depends on SQLAlchemy (>=0.9.2,<2.0.0), sqlalchemy-redshift (>=0.8.14,<0.9.0) requires SQLAlchemy (>=0.9.2,<2.0.0).
So, because cable-forecasting-v2 depends on both sqlalchemy (^2.0.38) and sqlalchemy-redshift (^0.8.14), version solving failed.

While a project might be able to use a prior sqlalchemy version right now, it appears that would be a significant practical constraint as sqlalchemy-redshift doesn't seem to aim to release a new update?

[shcheklein]

@DaveParr do you know about the alternatives? can you recommend an update to the docs?

[DaveParr]

@shcheklein I'm afraid I don't.

From sqlalchemy/sqlalchemy#11950 it seems that sqlalchemy are not interested in plugging this hole themselves, and also have highlighted that as redshift is 'proprietary' it makes any non-aws backed project a significantly larger lift.

I don't work for aws but I am guessing that they saw that this functionally overlaps a subset of the capabilities of awswrangler https://aws-sdk-pandas.readthedocs.io/en/3.11.0/stubs/awswrangler.redshift.connect.html and so that's where the focus has gone? However, as DVC seems to be sqlalchemy based and awswrangler might not be I don't think this is a drop in replacement.

For our current codebase I expect we will work around this using a dvc stage which calls a python command that does the execution against redshift using aws wrangler during it's execution. I realise it's slightly less ergonomic and we miss out on a few nice to haves but it will fulfil our needs today.

[shcheklein]

@skshetry should we remove it from docs or mention that it is deprecated?

[skshetry]

DVC is compatible with SQLAlchemy v1 as well, so that still works.

However, the PyPI packages mentioned on the page are third-party packages, and we can't guarantee that they will always work — they are only recommendations.

[shcheklein]

okay, let's close this for now. I think users can see and decide on their own.

[DaveParr]

Totally understand this decision, thanks for working it through with me.

Last thing, in case it's useful. In poking awswrangler I came across this: https://github.com/aws/amazon-redshift-python-driver which has this issue: aws/amazon-redshift-python-driver#253

Very unsure where this exists in the stack, I have no prior experience with it, but thought it might have some relevance to working this through for people more in the db connections weeds than me.

Eitherway, thanks for the work, DVC is providing us a tonne of benefit even outside this 👍🏻