Missing API and documentation to serialize/deserialize an In-Memory Storage instance

[vitaly-krugl]

I expect to be able to serialize/deserialize In-Memory Storage instance in order to persist the state and restore from state

There should be a supported mechanism to serialize/deserialize In-Memory Storage instance

Missing API and documentation to serialize/deserialize In-Memory Storage instance

Your Environment

• limits version: limits 4.2
• Operating system: OS X, Linux

[alisaifee]

This was never an expectation of the memory storage implementation which is only meant for testing or development purposes. If you need to maintain the state of the rate limit storage beyond the lifetime of the application, perhaps consider using any of the external storage implementations.

[vitaly-krugl]

Hi @alisaifee, thank you for the feedback. Unfortunately, the external options are not available in my environment.

However, the ability to serialize and deserialize the In-Memory storage (if it were possible) would work great for my use case.

[alisaifee]

The main problem from an implementation point of view with providing a serialization/de-serialization interface to the memory storage is that it relies on threading.RLock which in itself isn't serializable.

As an initial step that might make your use-case possible, I'll try to make some changes that would allow for example:

import limits
import pickle

storage = limits.storage.MemoryStorage()
l1 = limits.strategies.MovingWindowRateLimiter(storage)
assert l1.hit(limits.parse("1/hour"))
pickled = pickle.dumps(storage)

unpickled_storage = pickle.loads(pickled)
l2 = limits.strategies.MovingWindowRateLimiter(unpickled_storage)
assert not l2.hit(limits.parse("1/hour"))

[alisaifee]

The example shared above is now possible in 4.3 both for limits.storage.MemoryStorage and limits.aio.storage.MemoryStorage.

[vitaly-krugl]

Thank you @alisaifee. What are the limitations of this pickle support?

[alisaifee]

Thank you @alisaifee. What are the limitations of this pickle support?

If your objective is to maintain the state between application restarts, I don't foresee any limitations. If there are entries in the storage that should be expired while the application is restarting, they'll get cleared on restart as well. Is there any thing else that you're concerned about?

[vitaly-krugl]

Thank you @alisaifee! No other concerns at this time.