Kafka Consumer Worker

[hekike]

What?

• Kafka Consumer as Go Worker instead of Kafka Connect
• Separate server and worker commands
• Horizontally scalable consumers via Kafka's consumer rebalancing
• Validating events against meters for better deadlettering

Why?

Motivation in order of importance:

• Increase deduplication consistency by deduping in consumer
• Reduce error blast radius when Redis is down by deduping after ingestion
• Minimize the number of correct events sent to dead-letter when a single event is malformed in the batch through validating against meters
• Enable autoscaling via Kafka consumer rebalancing
• Being able to implement more advanced dead-letter logic, like sinking it to a separate clickhouse table per namespace

Tradeoff:

• We are reimplementing some of Kafka Connect

Breaking Change

Operational breaking change. Replacing Kafka Connect with the sink-worker process.
There is no breaking change for the APIs.