Support configuring a destination for large result sets

[edgarrmondragon]

BigQuery queries have a limited response size¹ so syncs may fail when a large response is generated.

The python-bigquery-sqlalchemy² library supports passing a destination query parameter so the fix for this probably involves adding a new setting (e.g. destination_table) and passing that to the SQLAlchemy URL construction in

tap-bigquery/tap_bigquery/connector.py

Lines 45 to 47 in 0e37a0f

	def get_sqlalchemy_url(self, config: dict) -> str:
	"""Concatenate a SQLAlchemy URL for use in connecting to the source."""
	return f"bigquery://{config['project_id']}"

The string in question is a fully qualified table, e.g. different-project.different-dataset.table.

Footnotes

1. https://cloud.google.com/bigquery/docs/writing-results ↩

2. https://github.com/googleapis/python-bigquery-sqlalchemy?tab=readme-ov-file#connection-string-parameters ↩

[AlejandroUPC]

Hello @edgarrmondragon I will happily work on this I am just a bit lost and I do not really understand the issue, how would destination solve this? Also tagging @pnadolny13 to understand why it was decided to go w/ sqlalchemy instead of using standard google.cloud bigquery client?

[edgarrmondragon]

Hello @edgarrmondragon I will happily work on this I am just a bit lost and I do not really understand the issue

@AlejandroUPC You might wanna read through https://cloud.google.com/bigquery/docs/writing-results#large-results.

Have you run into this issue?

[ReubenFrankel]

Would #24 cover this?