[Bug] Memory leak occurring when using lmdb.open

[tczhangzhi]

Affected Operating Systems

Linux

Affected py-lmdb Version

1.4.1

py-lmdb Installation Method

pip install lmdb

Using bundled or distribution-provided LMDB library?

Bundled

Distribution name and LMDB library version

(0, 9, 29)

Machine "free -m" output

              total        used        free      shared  buff/cache   available
Mem:         385609       55278       22059        5864      308271      322580
Swap:             0           0           0

Other important machine info

Not related.

Describe Your Problem

There's a memory leak occurring when using lmdb.open. The memory usage keeps increasing until all the programs have finished running. Here is a reproduction case:

import numpy as np
import lmdb
import os
import psutil
import shutil

def load_array():
    """ Store and load a numpy array using LMDB. """
    # with lmdb.open('lmdb_data', map_size=1000000000) as env:

    #     with env.begin(write=False) as txn:
    #         # Retrieve the bytes and convert back to numpy array
    #         data_bytes = txn.get(b'array')
    #         retrieved_data = np.frombuffer(data_bytes, dtype=np.float64).reshape(100, 100)
    env = lmdb.open('lmdb_data', map_size=1000000000) # 1GB size

    with env.begin(write=False) as txn:
        # Retrieve the bytes and convert back to numpy array
        data_bytes = txn.get(b'array')
        retrieved_data = np.frombuffer(data_bytes, dtype=np.float64).reshape(100, 100)

    env.close()
    # del env
    return retrieved_data

def store_array():
    # Generate a random numpy array
    array_size = (100, 100) # Example size
    array = np.random.rand(*array_size)

    # Set up the LMDB environment
    env = lmdb.open('lmdb_data', map_size=1000000000) # 1GB size

    with env.begin(write=True) as txn:
        # Convert the numpy array to bytes and store it
        txn.put(b'array', array.tobytes())

print(lmdb.__version__)
print(lmdb.version())

store_array()

# Monitor memory usage
process = psutil.Process(os.getpid())
initial_memory = process.memory_info().rss

# Repeat the process 1000 times
for _ in range(1000):
    load_array()
    current_memory = process.memory_info().rss
    print(current_memory)
    if current_memory > initial_memory * 1.01: # 10% increase threshold
        print("Potential memory leak detected.")
        break

# # Cleanup
# if os.path.exists('lmdb_data'):
#     shutil.rmtree('lmdb_data')

# print("Task completed without detecting memory leak.")

Errors/exceptions Encountered

Potential memory leak detected.

Describe What You Expected To Happen

Task completed without detecting memory leak.

Describe What Happened Instead

Potential memory leak detected.