From 3353bd96bc50d5a54358df5371c7491cbbc5982d Mon Sep 17 00:00:00 2001 From: Yen-Ju Wu Date: Tue, 30 Jan 2024 23:58:00 -0500 Subject: [PATCH 1/2] update design_doc --- doc/design_doc.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/doc/design_doc.md b/doc/design_doc.md index 599f1d2..1c64b61 100644 --- a/doc/design_doc.md +++ b/doc/design_doc.md @@ -15,16 +15,15 @@ * The catalog service ensures data consistency and correctness by adhering to the Iceberg Catalog REST interface. * Data durability mechanisms will be implemented to prevent data loss during restarts. * Performance: - * Optimized data retrieval and storage strategies to minimize latency in metadata access. - * Efficient indexing mechanisms to speed up metadata search operations. - * Caching strategies to reduce redundant metadata queries and enhance overall performance. + * Optimization on data retrieval and storage strategies to minimize latency in metadata access. + * Efficient indexing mechanisms, such as Bloom filters, enhance query performance to speed up metadata search operations. + * Partitioning strategies facilitate data pruning and improve query execution performance. * Engineering Complexity / Maintainability: - * The microservices architecture simplifies maintenance by encapsulating each building block. + * Centralized metadata management achieved by separating data and metadata, reducing complexity and facilitating consistent metadata handling. * Code modularity and clear interfaces facilitate easier updates and improvements. * Testing: * Comprehensive testing plans cover correctness through unit tests and performance through long-running regression tests. Unit tests focus on individual components of the catalog service, while regression tests evaluate system-wide performance and stability. * Other Implementations: - * Considered alternative implementations, including a monolithic catalog structure and alternative REST specifications. * Chose the Iceberg Catalog REST interface due to its industry adoption, standardization, and compatibility with various systems. ## Testing Plan From d9a51296d0ec70d7d671cdd045c782c0608f05bc Mon Sep 17 00:00:00 2001 From: Yen-Ju Wu Date: Wed, 31 Jan 2024 00:01:43 -0500 Subject: [PATCH 2/2] update design_doc --- doc/design_doc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/design_doc.md b/doc/design_doc.md index 1c64b61..7424bb1 100644 --- a/doc/design_doc.md +++ b/doc/design_doc.md @@ -16,7 +16,7 @@ * Data durability mechanisms will be implemented to prevent data loss during restarts. * Performance: * Optimization on data retrieval and storage strategies to minimize latency in metadata access. - * Efficient indexing mechanisms, such as Bloom filters, enhance query performance to speed up metadata search operations. + * Efficient indexing mechanisms, such as Bloom filters, enhance query performance. * Partitioning strategies facilitate data pruning and improve query execution performance. * Engineering Complexity / Maintainability: * Centralized metadata management achieved by separating data and metadata, reducing complexity and facilitating consistent metadata handling.