Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc:kafka consumer got partition resolved ts fallback when ticdc and kafka got network partition #12096

Open
AkiraXie opened this issue Mar 14, 2025 · 1 comment · May be fixed by #12112
Open

cdc:kafka consumer got partition resolved ts fallback when ticdc and kafka got network partition #12096

AkiraXie opened this issue Mar 14, 2025 · 1 comment · May be fixed by #12112
Assignees
@3AceShowHand
Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug.

Comments

@AkiraXie
Copy link

What did you do?

  1. open kafka changefeed
  2. inject network partition chaos between ticdc and kafka

What did you expect to see?

no error

What did you see instead?

[2025/03/13 16:17:35.372 +08:00] [PANIC] [writer.go:103] ["partition resolved ts fallback"] [partition=1] [newWatermark=456615706631340446] [offset=8418122] [watermark=456615720538341892] [watermarkOffset=8416182] [stack="main.(*partitionProgress).updateWatermark\n\tgithub.com/pingcap/tiflow/cmd/kafka-consumer/writer.go:103\nmain.(*writer).WriteMessage\n\tgithub.com/pingcap/tiflow/cmd/kafka-consumer/writer.go:392\nmain.(*consumer).Consume\n\tgithub.com/pingcap/tiflow/cmd/kafka-consumer/consumer.go:139\nmain.main.func2\n\tgithub.com/pingcap/tiflow/cmd/kafka-consumer/main.go:98"]

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):
master

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):
master

(paste TiCDC version here)
@AkiraXie AkiraXie added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels Mar 14, 2025
@AkiraXie
Copy link
Author

/severity major

@AkiraXie AkiraXie changed the title cdc:kafka consumer got partition resolved ts fallback when kafka and ticdc got network partition cdc:kafka consumer got partition resolved ts fallback when ticdc and kafka got network partition Mar 14, 2025
@3AceShowHand 3AceShowHand self-assigned this Mar 14, 2025
@3AceShowHand 3AceShowHand mentioned this issue Mar 17, 2025
Open
@3AceShowHand 3AceShowHand linked a pull request Mar 26, 2025 that will close this issue
Open
@3AceShowHand 3AceShowHand added affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. and removed may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 may-affects-8.5 labels Mar 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@3AceShowHand 3AceShowHand

Labels
affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
2 participants
@3AceShowHand @AkiraXie