Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] CdmChangeFeedSource attempts to stream data from multiple locations #98

Open
george-zubrienko opened this issue Aug 15, 2024 · 0 comments
Labels
code/bug Something isn't working

Comments

@george-zubrienko
Copy link
Contributor

Description

It seems sometimes in Dynamics data lake (Synapse) export the export location might change. However, due to the fact that we search for table using this code:

var tableBlobs = this.source.blobStorage.ListBlobsAsEnumerable(this.tablesPath).Where(blob =>
    blob.Name.Split("/")[^1].StartsWith($"{this.source.entityName.ToUpper()}_") &&
    blob.Name.EndsWith(".csv")).ToList();

and that filenames are not changed, we can hit two different paths in this case - old one with schemaA and new one with schemaB. Thus, stream will run normally and the abort with data type mismatch error when all rows from schemaA are exhausted

Steps to reproduce the issue

  1. Create two different paths leading to the table with the same name, but different schemas
  2. Run the source
  3. Observe failure after rows from first path are exhausted

Describe the results you expected

Either a hard failure with an ERROR level message, or a warning and automatic selection of files from a newer path (preferred)

System information

No response

@george-zubrienko george-zubrienko added the code/bug Something isn't working label Aug 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
code/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant