Refactor BaseEvaluation with Minor Project Structure Change, Rate Limiting, and pyproject.toml Migration

[jcourson8]

• Refactored BaseEvaluation class:

  • Moved utility functions to a new file: evaluation_utils.py.
  • Simplified the run method by breaking down its logic into several internal helper functions.
  • Note: I am not as familiar with using SyntheticEvaluation class and have not checked for breaking changes after refactoring.

• Implemented RateLimiter:

  • Added a RateLimiter class in src/chunking_evaluation/utils.py to regulate the embedding request flow.
  • Integrated optional rate-limiting into the BaseEvaluation _add_documents_to_collection helper function.

• Introduced tqdm for Progress Tracking:

  • Added a tqdm progress bar option to the BaseEvaluation class for better visualization of long-running processes.
  • Users can enable this feature by setting use_tqdm=True in the run method:

    results = evaluation.run(chunker, default_ef, use_tqdm=True, rate_limiter=rate_limiter)

• Migrated from setup.py to pyproject.toml:

  • Updated packaging configuration by moving to pyproject.toml, following modern Python packaging standards.
  • Note: Please test that pip install via git works on your end without issue.

• Google Colab Notebook Management:

  • Added a notebook folder to the project, which houses Jupyter Notebooks (.ipynb) from Google Colab.
  • Updated the README.md to provide a direct link for opening these notebooks in Google Colab
  • Makes it so we can programmatically update it within the repo.

Please test on your end and let me know if you have questions.

[jcourson8]

Oo, I also changed the structure of the results. It now is a dict {"scores": scores, "stats": stats} so you can access the stats via results.get('stats').

The README is updated but here is the new flow for evaluation:

from chunking_evaluation import BaseChunker, GeneralEvaluation
from chromadb.utils import embedding_functions

# Define a custom chunking class
class CustomChunker(BaseChunker):
    def split_text(self, text):
        # Custom chunking logic
        return [text[i:i+1200] for i in range(0, len(text), 1200)]

# Instantiate the custom chunker and evaluation
chunker = CustomChunker()
evaluation = GeneralEvaluation()

# Choose embedding function
default_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key="OPENAI_API_KEY",
    model_name="text-embedding-3-large"
)

# Create a RateLimiter instance
rate_limiter = RateLimiter(
    # Set your rate limits as needed (this is OpenAI's tier 1 rate limit)
    max_tokens_per_minute=1_000_000, 
    max_requests_per_minute=3_000,
)

# Evaluate the chunker
results = evaluation.run(chunker, default_ef, rate_limiter) # set use_tqdm=True to see progress bar

print(results.get('stats'))
# {'iou_mean': 0.17715979570301696, 'iou_std': 0.10619791407460026, 
# 'recall_mean': 0.8091207841640163, 'recall_std': 0.3792297991952294}

[jcourson8]

Also, there is a bit of a hack in the RateLimiter where I decrease the tokens per minute by 20% because I could not resolve an bug with OpenAI embedding endpoint. I outlined it here... https://community.openai.com/t/discrepancy-between-tiktoken-token-count-and-openai-embeddings-api-token-count-exceeding-tpm-limit-in-tier-2-account/959298.

[jcourson8]

Removed the hack in RateLimiter by implementing batching on top of the TPM and RPM rate limiting.