Skip to content

Commit

Permalink
[batch-inserts] Toy for generating batch insert events (dagster-io#20592
Browse files Browse the repository at this point in the history
)

## Summary & Motivation

Toy definitions for generating larged numbers of batch insert events
from a single-run backfill.

## How I Tested These Changes

Manual execution locally.
  • Loading branch information
smackesey authored Mar 25, 2024
1 parent 52c7830 commit de7fc9b
Showing 1 changed file with 37 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
"""An asset with a large number of partitions (hourly over 1 years = 8760 partitions) and a
single-run backfill policy. When backfilled, will generate a large number of `store_event` or
`store_event_batch` calls, depending on if batching is enabled. Use a DummyIOManager to avoid
unnecessary writes.
"""

from dagster import (
AssetExecutionContext,
BackfillPolicy,
Definitions,
HourlyPartitionsDefinition,
IOManager,
asset,
)

partitions_def = HourlyPartitionsDefinition(
start_date="2023-01-01-00:00", end_date="2024-01-01-00:00"
)


@asset(partitions_def=partitions_def, backfill_policy=BackfillPolicy.single_run())
def foo(context: AssetExecutionContext):
return {k: 1 for k in context.partition_keys}


class DummyIOManager(IOManager):
def load_input(self, context, obj):
return 1

def handle_output(self, context, obj):
pass


defs = Definitions(
assets=[foo],
resources={"io_manager": DummyIOManager()},
)

0 comments on commit de7fc9b

Please sign in to comment.