IndexedDB Event Store #4617

ospfranco · 2025-02-04T09:16:22Z

This PR is WIP.

This PR is meant to add a EventCache store for the the matrix-sdk-indexeddb crate of the SDK meant to be used in WASM contexts. Similar to the sqlite implementation is supposed to provide a disk persisted implementation for storing Events in LinkedChunks.

It follows the general structure of matrix-sdk-indexeddb/src/crypto_store and matrix-sdk-indexeddb/src/state_store, a new module matrix-sdk-indexeddb/src/event_cache_store has been added.

There are still many TODOs and questions in the codebase to which I would appreciate some guidance from the SDK maintainers.

Public API changes documented in changelogs (optional)

Signed-off-by: Oscar Franco

crates/matrix-sdk-indexeddb/src/event_cache_store/idb_operations.rs

ospfranco · 2025-02-04T09:17:46Z

crates/matrix-sdk-indexeddb/src/event_cache_store/indexeddb_serializer.rs

+
+/// Handles the functionality of serializing and encrypting data for the
+/// indexeddb store.
+pub struct IndexeddbSerializer {


Copy and paste from crypto_store not sure if a different serializer is needed or they can be merged and re-used by both modules.

Did we end up having to change this much, or is it a straight copy? If straight copy, is it possible to pull it into a shared mod?

crates/matrix-sdk-indexeddb/src/event_cache_store/migrations.rs

ospfranco · 2025-02-04T09:19:17Z

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

+    migration_conflict_strategy: MigrationConflictStrategy,
+}
+
+impl IndexeddbEventCacheStoreBuilder {


Follows the same pattern as state_store, not sure if this is wanted or not.

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

ospfranco · 2025-02-04T09:20:14Z

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

+        Ok(())
+    }
+
+    async fn try_take_leased_lock(


Copy and paste from crypto_store. I could not find any info on what this "leased_lock" is.

This is related to the cross-process lock, the doc is missing a high-level overview I think… Tell me if you need more info.

I just copied the same from the crypto_store, if you think it is fine as it is, then no need. It should just work, right?

Possible candidate to try to put into a shared mod as well.

ospfranco · 2025-02-04T09:20:29Z

crates/matrix-sdk/src/client/builder/mod.rs

    };

+    // let store_config = {


Will be removed later

codecov · 2025-02-04T17:15:26Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.73%. Comparing base (3f71d9a) to head (c686f90).
Report is 173 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4617      +/-   ##
==========================================
+ Coverage   85.71%   85.73%   +0.01%     
==========================================
  Files         291      291              
  Lines       33323    33322       -1     
==========================================
+ Hits        28563    28568       +5     
+ Misses       4760     4754       -6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Hywan · 2025-02-11T09:12:11Z

Be aware that #4632 is likely to create a non-negligible conflict with your patches. The new methods to load chunk from the event cache are going to be simpler and will allow to incrementally load the chunks.

ospfranco · 2025-02-11T09:34:20Z

Be aware that #4632 is likely to create a non-negligible conflict with your patches. The new methods to load chunk from the event cache are going to be simpler and will allow to incrementally load the chunks.

Ah, interesting... well... maybe it's worth to put this a bit on hold until that is merged. If the trait will be easier to implement, then it's worth it

Hywan · 2025-02-11T09:35:45Z

The trait will gain 2 new methods: load_last_chunk and load_previous_chunk, and the method used to load the entire LinkedChunk is for testing-purposes only. So yeah, it's a bit simpler but it needs the whole package :-/.

zzorba · 2025-02-13T13:59:43Z

crates/matrix-sdk-indexeddb/src/event_cache_store/error.rs

+    fn from(e: IndexeddbEventCacheStoreError) -> Self {
+        match e {
+            IndexeddbEventCacheStoreError::Json(e) => StoreError::Json(e),
+            IndexeddbEventCacheStoreError::StoreError(e) => e,


Does this compile, I'm surprised it doesn't need a wrapper for StoreError here?

zzorba · 2025-02-13T14:00:03Z

crates/matrix-sdk-indexeddb/src/event_cache_store/error.rs

+                EventCacheStoreError::Backend(Box::new(e))
+            }
+            IndexeddbEventCacheStoreError::Encryption(e) => EventCacheStoreError::Encryption(e),
+            _ => EventCacheStoreError::backend(e),


backend => Backend, right?

zzorba · 2025-02-13T14:29:24Z

crates/matrix-sdk-indexeddb/src/event_cache_store/error.rs

+        match frm {
+            IndexeddbEventCacheStoreError::Json(e) => CryptoStoreError::Serialization(e),
+            IndexeddbEventCacheStoreError::CryptoStoreError(e) => e,
+            _ => CryptoStoreError::backend(frm),


s/backend/Backend/

zzorba · 2025-02-13T14:29:47Z

crates/matrix-sdk-indexeddb/src/event_cache_store/error.rs

+    fn from(frm: IndexeddbEventCacheStoreError) -> CryptoStoreError {
+        match frm {
+            IndexeddbEventCacheStoreError::Json(e) => CryptoStoreError::Serialization(e),
+            IndexeddbEventCacheStoreError::CryptoStoreError(e) => e,


I would think this needs a wrapper as well, right?

zzorba · 2025-02-13T14:46:30Z

crates/matrix-sdk-indexeddb/src/event_cache_store/indexeddb_serializer.rs

+
+/// Handles the functionality of serializing and encrypting data for the
+/// indexeddb store.
+pub struct IndexeddbSerializer {


Did we end up having to change this much, or is it a straight copy? If straight copy, is it possible to pull it into a shared mod?

zzorba · 2025-02-13T15:01:01Z

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

+    /// Return all the raw components of a linked chunk, so the caller may
+    /// reconstruct the linked chunk later.
+    async fn reload_linked_chunk(&self, _room_id: &RoomId) -> Result<Vec<RawChunk<Event, Gap>>> {
+        Ok(vec![])


todo!, or this is just for tests?

zzorba · 2025-02-13T15:01:29Z

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

+    /// This will empty and remove all the linked chunks stored previously,
+    /// using the above [`Self::handle_linked_chunk_updates`] methods.
+    async fn clear_all_rooms_chunks(&self) -> Result<()> {
+        Ok(())


todo!, this should just be clearing the tables right?

zzorba · 2025-02-13T15:02:03Z

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

+        _content: Vec<u8>,
+        _ignore_policy: IgnoreMediaRetentionPolicy,
+    ) -> Result<()> {
+        Ok(())


Media content seems much simpler than the chunk system, is that correct?

zzorba · 2025-02-13T15:02:56Z

crates/matrix-sdk-indexeddb/src/event_cache_store/mod.rs

+        Ok(())
+    }
+
+    async fn try_take_leased_lock(


Possible candidate to try to put into a shared mod as well.

zzorba · 2025-02-13T15:05:22Z

crates/matrix-sdk/src/client/builder/mod.rs

-    let store_config = {
-        tracing::warn!("The IndexedDB backend does not implement an event cache store, falling back to the in-memory event cache store…");
-        store_config.event_cache_store(matrix_sdk_base::event_cache::store::MemoryStore::new())
+        let event_cache_store = matrix_sdk_indexeddb::open_event_cache_store(name, None).await?;


Seems like sqlite uses the same passphrase for both stores, should we do the same here instead of passing None?

ospfranco added 11 commits January 30, 2025 12:49

Modify the matrix-sdk and add feature flag for the event cache store

6955428

Create the basic struct to initialize an event_cache_store

b8bfd3e

Almost working instantiation of event cache store

c2761d5

Fix deps on feature indexeddb

3c6ca6a

Add license text at top

7eb8946

Copy try_take_leased_lock from crypto_store impl

64e68ba

Add missing methods from EventCacheTrait

1328003

Fix a bunch of errors, move error to it's own file

eebe00e

Fix final errors around creating the event cache store

e49bda6

First insertion into indexeddb

1b1029d

Merge branch 'main' into oscar/indexeddb-event-cache-store

c317bd8

ospfranco commented Feb 4, 2025

View reviewed changes

Finish implementing insert_chunk

0f8ebf1

ospfranco added 15 commits February 4, 2025 19:06

Comment unused code, finish implementing insert_chunk

ee40ddd

Get rid of name param

73713ac

Insert GAP chunk

d3b6f87

Tried to implement insert gap, but missing encode_value on serializer

8528522

Implement insert gap

7622be4

remove chunk deletes from LINKED_CHUNKS store, missing related entities

01cb5ba

Fix some compilation errors

8f2b427

PushItems

722bac2

PushItems

4ca221f

RemoteItem

0e05ac8

Half implementation of DetachLastItems

28647b6

Remove deletion

eee0bad

Add placeholder for deletion

0b0e15e

Change previous implementations to use serializer

993283c

Change previous implementations to use serializer

d755cc0

Do not interate through get_all_with_key

abee3ab

ospfranco added 4 commits February 13, 2025 09:25

Fix encoded room id as ref

06b0395

finish DetachLastItems

ae25b35

PushItems now uses TimeEventForCache struct

4dcf22a

Implement Update::Clear

610860b

zzorba reviewed Feb 13, 2025

View reviewed changes

ospfranco added 6 commits February 17, 2025 15:00

Fixes on initialization of event-cache store

ebda277

WIP, fixes for creating serializable objects

1af016f

More fixes on serialization and deserialization

6d8b405

Working retrieving of events

4179750

Update key handling

6c7b729

Fix ranges

c686f90

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IndexedDB Event Store #4617

IndexedDB Event Store #4617

ospfranco commented Feb 4, 2025

ospfranco Feb 4, 2025

zzorba Feb 13, 2025

ospfranco Feb 4, 2025

ospfranco Feb 4, 2025

Hywan Feb 5, 2025 •

edited

Loading

ospfranco Feb 5, 2025

zzorba Feb 13, 2025

ospfranco Feb 4, 2025

codecov bot commented Feb 4, 2025 •

edited

Loading

Hywan commented Feb 11, 2025

ospfranco commented Feb 11, 2025

Hywan commented Feb 11, 2025

zzorba Feb 13, 2025

zzorba Feb 13, 2025

zzorba Feb 13, 2025

zzorba Feb 13, 2025

zzorba Feb 13, 2025

zzorba Feb 13, 2025

zzorba Feb 13, 2025

zzorba Feb 13, 2025

zzorba Feb 13, 2025

zzorba Feb 13, 2025

IndexedDB Event Store #4617

Are you sure you want to change the base?

IndexedDB Event Store #4617

Conversation

ospfranco commented Feb 4, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hywan Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 4, 2025 • edited Loading

Codecov Report

Hywan commented Feb 11, 2025

ospfranco commented Feb 11, 2025

Hywan commented Feb 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Hywan Feb 5, 2025 •

edited

Loading

codecov bot commented Feb 4, 2025 •

edited

Loading