Framework to support multiple workers and a Redis implementation #183
Conversation
1. Added a SocketManager class to abstract sockets from the storage used. 2. Started implementing default/local version, which should keep things the same and Redis version as a reference distributed implementation. 3. Abstracted out some of the objects used internally by the sockets like Queues and session dictionaries, so that they can be distributed too.
…t's held by all context managers and released at exiting the last. - Heartbeats synced using Redis pubsub. - Integrated the manager and the session locking into virtsocket and the handler. - Optimized bulk reading from queues.
- Reworked the mixins to use the new code - The RedisMapping used for session uses 'pickle' now (so we can add more complex objects to the sessions) - Refactored the django_chat example to work with the new changes - 'transports' was not passed to the server in sgunicorn, so it seems it was never used
Refactoring.
- Convert 'None' to None when reading from a Redis queue. - Bugfixes
…etManager and converted the socket state syncing to use the events. Reduced the scope of socket locking to ignore simple reading of messages or heartbeats. Fixed to work wit the XHR polling transport. Some initial work on the mixing to work with the new code.
Extracted namespace creation in a method add_namespace to match remote_namespace.
Lua scripts for atomic bucket operations. Added a process '_orphan_cleaner' to detect and clean left-over socket data. Improved cleanup of socket data from Redis.
…l being detected.
Warning in the Readme
|
Hello and thank you for your great work, this is exactly what I was looking for. |
|
Yes, we use this branch for our project, so I'll be fixing any reported bugs or issues. Don't have much time to work on documenting it better now, or writing a demo, but if you have trouble getting it to work I'd be happy to help. |
|
Ditto, thanks for this! From poking around on the branch, is setting these environment variables using raw_env all that's necessary in a gunicorn environment? SOCKET_MANAGER_CLASS=socketio.contrib.redis.socket_manager.RedisSocketManager |
|
That's pretty much all you need. I put a sample gunicorn config file at: |
|
Hello, i made a small contribution to this fork. The Pull request in on ryesoft:master |
|
Hello, @dragozov . Thank you for your work on this issue. |
|
@uppavel I don't think so. The gunicorn workers are like separate python processes. They might not even be on the same machine if you use multiple servers. |
|
pinging people here, is there anyone who would like to undertake the analysis and merging of a bunch of PRs open on this repo ? I don't have time to do it any time soon! Who will step up ? |
Hey,
I've been coding on a fork to support multiple workers and it's getting in shape, so you may want to check it out.
My primary objective was to get it to work for our current project, so there are significant changes, but I've tried to keep it clean and the default, non-Redis implementation should be backwards compatible with the current logic. I even fixed one of the sample projects (django_chat) to use my version.
Let me know what you think.
Best,
Plamen Dragozov