DRIVERS-2917 - Standardized Performance Testing of ODMs and Integrations

[NoahStapp]

Purpose

This is the first draft of the Django implementation of the MongoDB ODM Benchmarking suite. It contains a small suite of benchmark tests designed to measure django-mongodb-backend performance across data sizes and structures for what we expect to be common user operations.

This is NOT intended to be a comprehensive test suite for every operation, only the most common and widely applicable. This is also not intended to be Django specific: each of these tests must be implementable across all of our ODMs, up to the features supported by each library.

Structure

The benchmarks are contained within a separate Django application within tests/performance/perftest. This application exists only to execute the benchmarking suite.

Running Locally

After starting a local MongoDB server, the tests can be run by running python manage.py test from the tests/performance directory. The full suite run locally is expected to take approximately 10 minutes. For faster benchmarks, pass FASTBENCH=1 as an environment variable totests/performance/perftest/tests.py.

Review Items

• Do the Django operations being performed conform to our expectations of how django-mongodb-backend should be used?
• Are any operations missing that we expect to be frequently performed by users?
• Is the scope and breadth of the benchmarks sufficient for coverage without being cumbersome to implement across potentially dozens of libraries, first and third-party?
• Are the data sizes and model structures being used representative of real-world situations?