Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] *: GC API Refactorying #8989

Draft
wants to merge 46 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
71844d3
update kvproto
MyonKeminta Jan 7, 2025
54c235a
Add RWLockGroup
MyonKeminta Jan 9, 2025
e4ddb3e
changes to interfaces
MyonKeminta Jan 10, 2025
8f52875
(stash) wrap if-then-else transaction of etcd and add tests to the wr…
MyonKeminta Jan 16, 2025
77f7b9f
Finish the test
MyonKeminta Jan 17, 2025
9364ab6
Implement the low level txn interface for memkv and leveldb
MyonKeminta Jan 20, 2025
7f34146
expose etcd's transaction as LowLevelTxn interface to kv.Base, simula…
MyonKeminta Jan 20, 2025
413cc45
Fix lint and add comments
MyonKeminta Jan 21, 2025
29a8470
fix errors.toml
MyonKeminta Jan 21, 2025
ef4bf64
Merge commit '7f3414606' into m/gc-api-refactor
MyonKeminta Jan 22, 2025
ae9dd68
Merge branch 'm/low-level-txn-interface' into m/gc-api-refactor
MyonKeminta Jan 22, 2025
f143c33
Reimplement GCStateStorage
MyonKeminta Feb 6, 2025
a76ce69
Add partial implementation in GCStateManager
MyonKeminta Feb 13, 2025
a93a181
fix typo
MyonKeminta Feb 13, 2025
3cf8b04
add compatibility interface to GCStateManager
MyonKeminta Feb 13, 2025
1edfa44
Remove implementation to memory kv and leveldb kv
MyonKeminta Feb 17, 2025
a927abf
Renaming
MyonKeminta Feb 18, 2025
d6bc730
Adjust comments
MyonKeminta Feb 18, 2025
e9cf2a1
Remove unnecessary changes to mem_kv.go; adjust comments
MyonKeminta Feb 18, 2025
2f1887b
Merge branch 'master' of https://github.com/tikv/pd into m/low-level-…
MyonKeminta Feb 18, 2025
4fa3b5d
Fix lint
MyonKeminta Feb 18, 2025
6274387
Merge branch 'm/low-level-txn-interface' into m/gc-api-refactor
MyonKeminta Feb 18, 2025
4f868de
Remove useless code; address comments
MyonKeminta Feb 18, 2025
13b3106
Remove mentions to etcd in the new raw transaction interface
MyonKeminta Feb 18, 2025
c10d4b9
update kvproto
MyonKeminta Feb 18, 2025
e72f4ab
compatible with deletion behavior of deprecated UpdateServiceGCSafePoint
MyonKeminta Feb 18, 2025
b7f7d12
Merge branch 'master' into m/low-level-txn-interface
ti-chi-bot[bot] Feb 19, 2025
7d391a5
stash
MyonKeminta Feb 19, 2025
eb2e61b
Merge branch 'm/low-level-txn-interface' into m/gc-api-refactor
MyonKeminta Feb 19, 2025
217104e
Adapt interface changes
MyonKeminta Feb 19, 2025
4f0b073
Update some tests
MyonKeminta Feb 19, 2025
19b916c
stash
MyonKeminta Feb 19, 2025
427ad8c
restructuring and adding tests
MyonKeminta Feb 20, 2025
f3ae93e
Add tests
MyonKeminta Feb 24, 2025
75c8c28
Cover deleting GC barriers
MyonKeminta Feb 24, 2025
b0b64f2
Add tests which is not able to run for now (needs other unmerged chagnes
MyonKeminta Feb 24, 2025
08eaa27
Adjust behavior of the keyspaceID field
MyonKeminta Feb 25, 2025
d7b5580
some adjustments
MyonKeminta Feb 27, 2025
3c8a8b5
Merge commit 'b7f7d12' into m/gc-api-refactor
MyonKeminta Feb 27, 2025
cdf005d
Merge commit '1b81d80' into m/gc-api-refactor
MyonKeminta Feb 27, 2025
fca806d
Merge branch 'master' of https://github.com/tikv/pd into m/gc-api-ref…
MyonKeminta Feb 27, 2025
ff877a6
Adapt to the keypath change and fix tests
MyonKeminta Feb 27, 2025
58d49a2
stash
MyonKeminta Feb 28, 2025
ed5efbf
Fix and add mroe tests to CompatibleUpdateServiceGCSafePoint
MyonKeminta Mar 3, 2025
ed52c48
Add tests
MyonKeminta Mar 4, 2025
e027343
Add tests
MyonKeminta Mar 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions client/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,5 @@ require (
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

replace github.com/pingcap/kvproto => github.com/MyonKeminta/kvproto v0.0.0-20250227065230-984e1b47ae67
4 changes: 2 additions & 2 deletions client/go.sum
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/MyonKeminta/kvproto v0.0.0-20250227065230-984e1b47ae67 h1:PSK05hcHzHbfcOg8E3hYSCNi/tfgW7w5aXM9WRdA+ow=
github.com/MyonKeminta/kvproto v0.0.0-20250227065230-984e1b47ae67/go.mod h1:rXxWk2UnwfUhLXha1jxRWPADw9eMZGWEWCg92Tgmb/8=
github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
github.com/benbjohnson/clock v1.3.0 h1:ip6w0uFQkncKQ979AypyG0ER7mqUSBdKLOgAle/AT8A=
github.com/benbjohnson/clock v1.3.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
Expand Down Expand Up @@ -49,8 +51,6 @@ github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c h1:xpW9bvK+HuuTm
github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c/go.mod h1:X2r9ueLEUZgtx2cIogM0v4Zj5uvvzhuuiu7Pn8HzMPg=
github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86 h1:tdMsjOqUR7YXHoBitzdebTvOjs/swniBTOLy5XiMtuE=
github.com/pingcap/failpoint v0.0.0-20240528011301-b51a646c7c86/go.mod h1:exzhVYca3WRtd6gclGNErRWb1qEgff3LYta0LvRmON4=
github.com/pingcap/kvproto v0.0.0-20250224053625-b6a98c6bf02d h1:52qhTQG8G8V/pHo/w7F4d2Tw98KMk2C+gAe3U8SWRAg=
github.com/pingcap/kvproto v0.0.0-20250224053625-b6a98c6bf02d/go.mod h1:rXxWk2UnwfUhLXha1jxRWPADw9eMZGWEWCg92Tgmb/8=
github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3 h1:HR/ylkkLmGdSSDaD8IDP+SZrdhV1Kibl9KrHxJ9eciw=
github.com/pingcap/log v1.1.1-0.20221110025148-ca232912c9f3/go.mod h1:DWQW5jICDR7UJh4HtxXSM20Churx4CQL0fwL/SoOSA4=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
Expand Down
5 changes: 5 additions & 0 deletions errors.toml
Original file line number Diff line number Diff line change
Expand Up @@ -451,6 +451,11 @@ error = '''
failed to convert a path to absolute path
'''

["PD:gc:ErrGCOnInvalidKeyspace"]
error = '''
trying to manage GC in keyspace %v where keyspace level GC is not enabled
'''

["PD:gin:ErrBindJSON"]
error = '''
bind JSON error
Expand Down
2 changes: 2 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -221,3 +221,5 @@ require (
moul.io/zapgorm2 v1.1.0 // indirect
sigs.k8s.io/yaml v1.4.0 // indirect
)

replace github.com/pingcap/kvproto => github.com/MyonKeminta/kvproto v0.0.0-20250227065230-984e1b47ae67
1,717 changes: 1,708 additions & 9 deletions go.sum

Large diffs are not rendered by default.

11 changes: 11 additions & 0 deletions pkg/errs/errno.go
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ var (
ErrGetSourceStore = errors.Normalize("failed to get the source store", errors.RFCCodeText("PD:common:ErrGetSourceStore"))
ErrGetTargetStore = errors.Normalize("failed to get the target store", errors.RFCCodeText("PD:common:ErrGetTargetStore"))
ErrIncorrectSystemTime = errors.Normalize("incorrect system time", errors.RFCCodeText("PD:common:ErrIncorrectSystemTime"))
ErrInvalidArgument = errors.Normalize("invalid argument for %s: %v", errors.RFCCodeText("PD:common:ErrInvalidArgument"))
)

// tso errors
Expand Down Expand Up @@ -548,3 +549,13 @@ var (
ErrNotFoundSchedulingPrimary = errors.Normalize("cannot find scheduling primary", errors.RFCCodeText("PD:mcs:ErrNotFoundSchedulingPrimary"))
ErrSchedulingServer = errors.Normalize("scheduling server meets %v", errors.RFCCodeText("PD:mcs:ErrSchedulingServer"))
)

// GC errors
var (
ErrGCOnInvalidKeyspace = errors.Normalize("trying to manage GC in keyspace %v where keyspace level GC is not enabled", errors.RFCCodeText("PD:gc:ErrGCOnInvalidKeyspace"))
ErrDecreasingGCSafePoint = errors.Normalize("trying to update GC safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingGCSafePoint"))
ErrGCSafePointExceedsTxnSafePoint = errors.Normalize("trying to update GC safe point to a too large value that exceeds the txn safe point, current value: %v, given: %v, current txn safe point: %v", errors.RFCCodeText("PD:gc:ErrGCSafePointExceedsTxnSafePoint"))
ErrDecreasingTxnSafePoint = errors.Normalize("trying to update txn safe point to a smaller value, current value: %v, given: %v", errors.RFCCodeText("PD:gc:ErrDecreasingTxnSafePoint"))
ErrGCBarrierTSBehindTxnSafePoint = errors.Normalize("trying to set a GC barrier on ts %d which is already behind the txn safe point %d", errors.RFCCodeText("PD:gc:ErrGCBarrierTSBehindTxnSafePoint"))
ErrReservedGCBarrierID = errors.Normalize("trying to set a GC barrier with a barrier ID that is reserved: %v", errors.RFCCodeText("PD:gc:ErrReservedGCBarrierID"))
)
Loading