(WIP) Kafka Messaging Service #12
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces a Nix flake configuration and custom overlays for pyobjc-core and pystray packages. The changes include: - Add flake.nix with configuration for aarch64-darwin system - Create flake.lock file with pinned dependencies - Add custom overlay for pyobjc-core in overlays/pyobjc-core/ - Include build.nix and default.nix for pyobjc-core - Add custom overlay for pystray in overlays/pystray/ - Include default.nix for pystray NOTE: The Nix flake is currently not working due to a missing dependency. The `pyobjc-framework-quartz` package is not yet packaged, which prevents the successful building of the development environment. Error message: pyobjc-framework-quartz not installed These changes are a work in progress towards allowing reproducible builds and development environments for the project, specifically targeting macOS ARM64 architecture. The custom overlays address compatibility issues and provide necessary modifications for building pyobjc-core and pystray on the target system. Further work is needed to package the missing dependency or find an alternative solution.
This commit resolves the issue with building Python packages in the Nix flake. - Modified flake.nix to remove pyobjc-core custom package - Updated overlays/pystray/default.nix to fetch and use the wheel from PyPI - Added overlays/pystray/build.nix for building the pystray package The primary fix was to install pystray from the pre-built wheel on PyPI instead of building it from source. This approach bypasses the need for the missing pyobjc-framework-quartz dependency and allows the flake to build successfully. These changes enable the Nix flake to correctly build the required Python packages, resolving the previous error and allowing the development environment to be created as intended.
Add shellHook to make the 'topos' command available in the default development shell by adding the app's bin directory to PATH.
This commit addresses the problem where spaCy couldn't find the 'en_core_web_sm' model. Changes include: - Remove 'active_spacy_model' from config.yaml - Add 'en_core_web_sm' as a direct dependency in pyproject.toml - Update token_classifiers.py to import and load the model directly Note: This fix may generate a warning about potential version incompatibility between the installed model (3.8.0) and the current spaCy version (3.7.2). Future updates should align these versions to ensure full compatibility.
- Add 'topos' package for direct execution - Create 'topos' dev shell with auto-start - Update README with new Nix usage instructions - Remove outdated Spacy model setup steps
This commit addresses regressions from previous changes and improves configuration management: - Reintroduce ability to change active spaCy model using `topos set --spacy trf` - Move config.yaml to user's system config directory - Add toposSetupHook in flake.nix to initialize config file if not present - Update utilities to use TOPOS_CONFIG_PATH environment variable - Modify OntologicalFeatureDetection and token_classifiers to load spaCy model from config - Remove direct spaCy model download from spacy_loader.py - Update get_config_path utility function These changes restore the flexibility of spaCy model selection and improve the overall configuration management of the application.
- Add PostgreSQL as a package dependency in development shells - Create a PostgreSQL setup hook that: - Initializes the database directory - Starts the PostgreSQL server - Creates necessary tables (conversation_cache, entities, relations) - Sets up user roles and permissions - Load environment variables from .env_dev file - Incorporate PostgreSQL setup in both default and topos development shells These changes streamline the development process by automatically setting up and configuring PostgreSQL when entering the Nix development environment, ensuring a consistent database setup across all developers.
- Ensure PGDATA directory exists before initialization - Use pg_isready to wait for PostgreSQL to start properly - Create the database if it doesn't exist - Remove redundant user creation (assuming user already exists) - Improve error handling in SQL execution - Adjust log file location and server stop/start commands - Clean up environment variable loading These changes ensure a more robust and reliable PostgreSQL setup process when entering the Nix development environment.
- Add kafkaSetupHook to initialize and start Kafka in Kraft mode - Configure Kafka server properties for a single-node setup - Initialize Kafka storage and start the Kafka server - Create a 'chat_topic' for initial testing Note: This is a work in progress and may require further refinement and testing. Future commits will focus on integrating this setup with the existing development environment and ensuring proper error handling and cleanup.
…cture - Replace flake-utils with flake-parts for improved flake composition - Restructure flake.nix for better organization and readability - Rename 'myapp' to 'toposPoetryEnv' for clarity - Separate Poetry environment and application building - Introduce a common 'overrides' configuration for Poetry packages - Update package definitions and development shells - Adjust PATH exports in shell environments - Streamline the default package to use the topos script directly These changes aim to enhance the maintainability and flexibility of our Nix setup while preserving existing functionality. The use of flake-parts allows for a more modular and extensible configuration.
…ce management - Add process-compose-flake and services-flake as new inputs - Implement a new process-compose configuration for managing services - Integrate Ollama LLM service with configurable models - Refactor PostgreSQL setup to use services-flake - Add a new topos-service.nix for defining the Topos service - Update flake.nix to use the new service configurations - Modify .env_dev to use 127.0.0.1 instead of localhost for POSTGRES_HOST - Simplify devShells configuration - Remove manual setup hooks for PostgreSQL and Kafka These changes aim to provide a more robust and manageable service configuration, leveraging process-compose for better service orchestration and dependency management.
- Replace manual Kafka setup with services-flake's apache-kafka service - Add Zookeeper service as a dependency for Kafka - Introduce kafkaPreStartup script to create the chat_topic - Update PostgreSQL service name from "pg1" to "pg" for consistency - Refine service dependencies: - Kafka now depends on Zookeeper - Topos depends on both PostgreSQL and Kafka - Remove redundant kafkaSetupHook - Adjust Kafka settings for simplified configuration - Include JVM options for Kafka to set heap size These changes streamline the service setup process and ensure proper initialization order, improving the overall reliability and maintainability of the development environment.
G-structure
force-pushed
the
luc/kafka
branch
from
28a68ea to
1bb7faa
Compare
- Update project description to "topos-cli" - Reorganize overlays and services into a dedicated 'nix' directory - Simplify service configurations: - Remove commented-out Kafka settings - Standardize service naming (e.g., "ollama" instead of "ollama1") - Update TOPOS_CONFIG_PATH to use ~/.topos instead of ~/.config/topos - Remove unused tkinter overlay - Adjust data directory paths for consistency - Clean up comments and whitespace These changes improve the maintainability of our Nix configuration and provide a cleaner structure for future development.
- README.md: - Add instructions for running Topos and its dependencies using `nix run` - Reorganize setup instructions into "Production" and "Development" sections - Remove outdated instructions for setting up Spacy models - Remove requirement for python-tk - Update development shell instructions - justfile: - Update 'run' recipe to use `nix run .` instead of `topos run` These changes simplify the setup process for new users and developers, leveraging Nix to manage dependencies and provide a consistent environment. The updated documentation reflects our shift towards a more Nix-centric workflow, making it easier for contributors to get started with the project.
G-structure
force-pushed
the
luc/kafka
branch
from
1bb7faa to
6039b90
Compare
- Replace "phi3" with "dolphin-llama3" in the Ollama service configuration
- Add new tables in PostgreSQL for groups, users, and user-group associations - Implement GroupManagementService and GroupManagerSQLite for managing users and groups - Add MissedMessageManager and MissedMessageService for handling missed messages - Update utilities with new helper functions The following files were copied from Jhonny's kafka-chat-server branch (commit: d4179da): - topos/services/messages/group_management_service.py - topos/services/messages/group_manager.py - topos/services/messages/missed_message_manager.py - topos/services/messages/missed_message_service.py These additions lay the groundwork for implementing group-based messaging and ensuring users can retrieve messages they missed while offline.
- Rename GroupManagerSQLite to GroupManagerPostgres - Update GroupManagementService to use PostgreSQL connection parameters - Refactor database operations in GroupManagerPostgres to use psycopg2 - Remove in-memory caching in favor of direct database queries - Update MissedMessageService to use the new GroupManagementService - Improve timestamp handling in get_missed_messages These changes align the group management system with our PostgreSQL database setup and improve overall consistency and reliability of the messaging system. ``` This commit message summarizes the main changes, focusing on the migration from SQLite to PostgreSQL and the associated refactoring of the group management and missed message services. It also highlights the removal of in-memory caching and improvements in timestamp handling.
This commit introduces a Kafka-based chat system and adds group management functionality: - Add KafkaManager class for handling Kafka producer and consumer operations - Implement WebSocket endpoint for real-time chat communication - Integrate Kafka message broadcasting with WebSocket connections - Add API endpoints for: - Retrieving missed messages - Creating chat groups - Joining existing chat groups - Update FastAPI app to use CORS middleware and lifespan management - Implement GroupManagementService and MissedMessageService integration These changes establish the foundation for a scalable, real-time chat system with group management capabilities, leveraging Kafka for message persistence and distribution.
G-structure
force-pushed
the
luc/kafka
branch
from
6039b90 to
1b5372b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.