Apply hints for nested tables

[steinitzu]

Description

Draft of nested table hints implementation:

apply_hints(path=['a', 'b', 'c'], columns=...)

Is working so far but there are some bugs and tests needed.

Related Issues

• Resolves Simplify schema modification of child tables #1647

Additional Context

[netlify]

✅ Deploy Preview for dlt-hub-docs canceled.

Name	Link
🔨 Latest commit	74f5a1b
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/67e2fe9b7ef0c600081d929b

[rudolfix]

Please see my suggestion how to deal with naming convention. Docs requirements are in the ticket.

[zilto]

The current implementation adds the tables to the schema (as tested), but it doesn't affect how the data is loaded.

For example, the hints will appear in

pipeline.default_schema.tables.keys()
# ignoring the dlt tables
# 'nested_data', 'override_child_outer1', 'override_child_outer1_innerfoo','nested_data__outer1', 'nested_data__outer1__innerfoo'

Whereas the normalizer row counts show no ingested data for the tables

pipeline.last_trace.last_normalize_info.row_counts
# 'nested_data': 2, 'nested_data__outer1': 2, 'nested_data__outer1__innerfoo': 2

I believe changes need to be made to Extractor._write_to_dynamic_table() and _get_dynamic_table_name() to push data to the right table. (Extractor._write_to_static_table() should rely on the explicitly provided table name).

The extractor would need to hold some mapping, but it could be more appropriate to move the logic to dlt.common.normalizers.json.helpers or to a Schema method?

[rudolfix]

Relational normalizer follows its logic of creating nested tables and column names. it comes only from the data. there's no mechanism to rename those, except the root table name which the user must set.

dlt is data first, not schema first. it is counterintuitive if you chose to start your work with schema, not data.

I assume that in example you are giving, you used a custom table name for nested table. If this is not the case ping me on slack. maybe there's a bug somewhere

in the ticket above, there's a note:

You still may allow users to specify table_name on the nested hint. If you do so, you'll need to modify the normalizer so it maps paths to those names. IMO this is for another ticket and bigger overhaul of the schema
prevent following to be set on nested table:
parent_table_name: TTableHintTemplate[str] = None,
incremental: TIncrementalConfig = None,

so I'd say we block setting table name on nested hints (also parent name and incremental do make sense)

[zilto]

We need a good example (in Examples). my take is: let's go for something advanced:
here's a test test_merge_on_keys_in_schema: this test models the nested json into a set of tables with natural primary keys (not dlt generated ones). it uses some tricks to modify the nested tables. now we should be able to use our nested hints to just decorate resource properly. you are also free to change this test first and run it!
the goal of the example is to show how you can custom model your nested tables.

Working on the requested documentation (#1647), I encounter many edge cases of the current implementation. My understanding is that our goal is for these two snippets to be equivalent

Current test

From tests/load/pipeline/test_merge_disposition.py::test_merge_on_keys_in_schema

@dlt.source(schema=schema)
def ethereum(slice_: slice = None):
    @dlt.resource(
        table_name="blocks",
        write_disposition={"disposition": "merge", "strategy": merge_strategy},
    )
    def data():
        with open(
            "tests/normalize/cases/ethereum.blocks.9c1d9b504ea240a482b007788d5cd61c_2.json",
            "r",
            encoding="utf-8",
        ) as f:
            yield json.load(f) if slice_ is None else json.load(f)[slice_]

    # also modify the child tables (not nested)
    schema_ = dlt.current.source_schema()
    blocks__transactions = schema_.tables["blocks__transactions"]
    blocks__transactions["write_disposition"] = "merge"
    blocks__transactions["x-merge-strategy"] = merge_strategy  # type: ignore[typeddict-unknown-key]
    blocks__transactions["table_format"] = destination_config.table_format

    blocks__transactions__logs = schema_.tables["blocks__transactions__logs"]
    blocks__transactions__logs["write_disposition"] = "merge"
    blocks__transactions__logs["x-merge-strategy"] = merge_strategy  # type: ignore[typeddict-unknown-key]
    blocks__transactions__logs["table_format"] = destination_config.table_format

    return data

Target API

@dlt.source(schema=schema)
def ethereum(slice_: slice = None):
    @dlt.resource(
        table_name="blocks",
        write_disposition={"disposition": "merge", "strategy": "delete-insert"}
    )
    def data():
        with open(
            "./ethereum.blocks.9c1d9b504ea240a482b007788d5cd61c_2.json",
            "r",
            encoding="utf-8",
        ) as f:
            yield json.load(f) if slice_ is None else json.load(f)[slice_]

      
    data.apply_hints(
        path=["transactions"],
        write_disposition="merge",
        additional_table_hints={"x-merge-strategy": "delete-insert"},
        table_format=None,
    )

    data.apply_hints(
        path=["transactions", "logs"],
        write_disposition="merge",
        additional_table_hints={"x-merge-strategy": "delete-insert"},
        table_format=None,
    )
    
    return data

Problem

Currently, inspecting the source via ethereum().schema gives different results. The Current Test shows the hints on the blocks__transactions and the blocks__transactions__logs tables. It is not the case for the Target API.

When in the lifecycle should the hints be assigned? After .apply_hints()? After source creation? After first pipeline run?

Setting parents

  File "/home/tjean/projects/dlthub/dlt/dlt/extract/extract.py", line 353, in _extract_single_source
    extractors[item_format].write_items(
  File "/home/tjean/projects/dlthub/dlt/dlt/extract/extractors.py", line 140, in write_items
    self._write_to_static_table(resource, table_name, items, meta)
  File "/home/tjean/projects/dlthub/dlt/dlt/extract/extractors.py", line 223, in _write_to_static_table
    items = self._compute_and_update_table(resource, table_name, items, meta)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tjean/projects/dlthub/dlt/dlt/extract/extractors.py", line 263, in _compute_and_update_table
    diff_table = utils.diff_table(self.schema.name, existing_table, computed_table)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tjean/projects/dlthub/dlt/dlt/common/schema/utils.py", line 461, in diff_table
    ensure_compatible_tables(schema_name, tab_a, tab_b, ensure_columns=False)
  File "/home/tjean/projects/dlthub/dlt/dlt/common/schema/utils.py", line 529, in ensure_compatible_tables
    raise TablePropertiesConflictException(
dlt.common.schema.exceptions.TablePropertiesConflictException: In schema: ethereum: Cannot merge partial tables into table `blocks__transactions` due to property `parent` with different values: "None" != "blocks"

This PR explicitly sets parent when .apply_hints(path=...) is used. This differs from default behavior. How should we address this?

odd bug

When debugging the call to ensure_compatible_tables(), the values magically change before and after the comparison operator (see comments).

def ensure_compatible_tables(
    schema_name: str, tab_a: TTableSchema, tab_b: TPartialTableSchema, ensure_columns: bool = True
) -> None:
    """Ensures that `tab_a` and `tab_b` can be merged without conflicts. Conflicts are detected when

    - tables have different names
    - nested tables have different parents
    - tables have any column with incompatible types

    Note: all the identifiers must be already normalized
    """
    if tab_a["name"] != tab_b["name"]:
        raise TablePropertiesConflictException(
            schema_name, tab_a["name"], "name", tab_a["name"], tab_b["name"]
        )
    table_name = tab_a["name"]
    # check if table properties can be merged
    # breakpoint()
    # tab_a.get("parent"): None
    # tab_b.get("parent"): None 
    if tab_a.get("parent") != tab_b.get("parent"):
        # breakpoint()
        # tab_a.get("parent"): None
        # tab_b.get("parent"): "blocks"
        raise TablePropertiesConflictException(
            schema_name, table_name, "parent", tab_a.get("parent"), tab_b.get("parent")
        )

    if not ensure_columns:
        return

[rudolfix]

Working on the requested documentation (#1647), I encounter many edge cases of the current implementation. My understanding is that our goal is for these two snippets to be equivalent

right!

When in the lifecycle should the hints be assigned? After .apply_hints()? After source creation? After first pipeline run?

1. the schema property of source is the input schema that got passed in decorator. no hints are applied to it
2. "Current test" does a hack and modifies it directly. That's why you see it, but it is not how it should work
3. To get a schema with applied hints you must call source.discover_schema(). You'll get a schema clone with all hints applied to it. Note: now I see that compute_table_chain is not used there. please fix it if you want to see all your tables in schema clone.
4. Extract phase uses source input schema, applies all hints to it and then merges changes into existing pipeline schema (from the previous runs). During extract - this all happens on clones, which get comitted to storage when extract method exists. TLDR;> you can get pipeline.default_schema after extract or full run and you should see all the changes
5. please commit code that raises this weird bug. I'll take look. The "current test" is passing now.

this is pretty deep PR. we'll take time to iron this out. I still have some issues with user interface (ie. @dlt.resource should accept a list of nested resources so users do not need to do many apply_hints). we'll cover it at the end

[zilto]

I reviewed all files. I believe there are some typos and dev comments to remove.

I'm very excited about this feature. Awesome work!

[zilto]

sidenote: Reading through the codebase "evolve-columns-once": True is set after the table sees data for the first time. Maybe the name "columns-evolved-once" would better communicate that.

[rudolfix]

that was a good review thx! stuff fixed

[zilto]

All reviews were addressed. Looks good to me.

Just fix failing tests. Seems related to comparing lists and item ordering. Could use sets instead

[rudolfix]

All reviews were addressed. Looks good to me.
Just fix failing tests. Seems related to comparing lists and item ordering. Could use sets instead

ah yeah I know exactly what happens here. this must pass. it checks the inferred column order...