Skip to content

Commit

Permalink
Update DESIGN.md
Browse files Browse the repository at this point in the history
Small spelling fix
  • Loading branch information
computerscienceiscool authored Dec 6, 2024
1 parent b7638ed commit 009715c
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion doc/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ This section describes how end-to-end transactions that account for Kafka partit

On startup and after rebalance, each kafka-delta-ingest process must check its assigned partitions and re-seek the consumer when necessary. Each process must maintain an internal map of assigned partitions and current offsets. Kafka’s group management protocol guarantees that individual consumers within a group will be assigned a mutually exclusive set of partitions at any given time.

Upon startup or partition assignment (in case of rebalnce), to identify the last offset written to Delta Lake for each assigned partition, the kafka-delta-ingest process must locate the last txn action in the delta log for each of its assigned partitions and re-seek the consumer based on the txn.version attribute of the delta log.
Upon startup or partition assignment (in case of rebalance), to identify the last offset written to Delta Lake for each assigned partition, the kafka-delta-ingest process must locate the last txn action in the delta log for each of its assigned partitions and re-seek the consumer based on the txn.version attribute of the delta log.

When performing a write, each process must commit the last offset of each partition within a txn action contained in the delta log entry.

Expand Down

0 comments on commit 009715c

Please sign in to comment.