Skip to content

Commit f43d872

Browse files
Merge pull request #1724 from redis/DOC-5438-rdi-checkpoint-at-least-once
DOC-5348 RDI: info about at-least-once delivery and checkpointing
2 parents 97794d2 + 9eb4379 commit f43d872

File tree

1 file changed

+25
-2
lines changed

1 file changed

+25
-2
lines changed

content/integrate/redis-data-integration/architecture.md

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,8 @@ in sequence:
4242
currently uses an open source collector called
4343
[Debezium](https://debezium.io/) for this step.
4444

45-
1. The collector records the captured changes using Redis streams
45+
1. The collector records the captured changes using
46+
[Redis streams]({{< relref "/develop/data-types/streams" >}})
4647
in the RDI database.
4748

4849
1. A *stream processor* reads data from the streams and applies
@@ -68,6 +69,28 @@ RDI automatically enters a second phase called *change streaming*, where
6869
changes in the data are captured as they happen. Changes are usually
6970
added to the target within a few seconds after capture.
7071

72+
## At-least-once delivery guarantee
73+
74+
RDI guarantees *at-least-once delivery* to the target. This means that
75+
a given change will never be lost, but it might be added to the target
76+
more than once. Apart from a slight performance overhead, adding a
77+
change multiple times is harmless because the multiple writes
78+
are [*idempotent*](https://en.wikipedia.org/wiki/Idempotence) (that is
79+
to say that all writes after the first one make no change to the
80+
overall state).
81+
82+
## Checkpointing
83+
84+
RDI uses Redis streams to store the sequence of change events
85+
captured from the source. The events are then retrieved in order
86+
from the streams, processed, and written to the target. The stream
87+
processor uses a *checkpoint* mechanism to keep track of the last
88+
event in the sequence that it has successfully processed and stored. If the processor fails
89+
for any reason, it can restart from the last checkpoint and
90+
re-process any events that might not have been written to the target.
91+
This ensures that all changes are eventually recorded, even in the
92+
face of failures.
93+
7194
## Backpressure mechanism
7295

7396
Sometimes, data records can get added to the streams faster than RDI can
@@ -85,7 +108,7 @@ an error, just an informative message to note that RDI has applied
85108
the backpressure mechanism.
86109
{{</note>}}
87110

88-
### Supported sources
111+
## Supported sources
89112

90113
RDI supports the following database sources using [Debezium Server](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) connectors:
91114

0 commit comments

Comments
 (0)