Investigate usage of SQLite database

[JurgenR]

Currently the Python shelve module is used for storing shares and transfers but this ties the cache to the implementation, if a change needs to be made to the Transfer, SharedItem, SharedDirectory objects then some migration code needs to be written.

Using an SQLite database would be cleaner, but comes with its own set of problems. If done right this could also remove the need to store all shared files and transfer in memory.

There's a case to be made to split cache into cache and storage, with flexibility to mix different types.

Assume a collection of 5 million shared files as a target (this seems roughly the max amount of shared files seen in rooms by users). Transfers are a bit hard to quantify.

Speed

When connected to the distributed network the client will receive many search requests. Because shared items are currently stored in memory this is very fast, using an SQLite database would slow this down significantly. There are libraries available to be able to make SQL queries asynchronously and querying the local shares is already done as part of a task. The queries don't actually need to be super fast like implemented today, but they also cannot be slow (eg.: 3s+). If a query on millions of records takes 250ms than this could be considered fine.

More worrying is the insertion and deletion of shared items during scanning, insertion of millions of records can be very slow.

Migration

If changes need to be made to the database there needs to be some easy way to do migrations.

The changes could include some migration hooks, try finding a library that does versioning.