Log files

Original notes on log file format

LevelDB stores data in a log-file that can be persisted. The filename extension for these files are .log and there can only be one active log-file at a time. A link to the current log file is stored in the manifest. Manifest files themselves also use the log records structure.

Blocks and records

A log file contain x number of blocks with a fixed size (32KB) and optionally ends with a trailer (padding with zero bytes). The actual user-content is sliced into framgents (records) and fitted in one or several of these blocks. To facilitate this, each record has a type. The following types exist for records

FULL == 1
FIRST == 2 
MIDDLE == 3
LAST == 4

User data

The user data structured span multiple records (fragments). For actual log files (not manifest) the user content is organzied as batches of operations in the following format

batch :=
    sequence number: uint64
    operations count: uint32
    operations*
operation :=
    record type: uint8 // One of PUT, DELETE
    key lenght: varint64
    key: uint8[key lenght]
    data lenght: varint64 // Only if PUT
    data: uint8[data lenght] // Only if PUT

MiNET LevelDB will cache user data in a dictionary with operation as the value.

Updating data

All batches of operations on the log file recieve a new sequence number guaranteed to be higher than any previous sequence numbers. This ensures that operations on the storage will not interfere with ongoing transactions. In theory this could enable the database to work concurrently and still maintain integrity.

Searching data

The database will scan the cached operations grouped by sequence number ordered by sequence number, in descending order. The original LevelDB use skip lists to store the datastructure, however MiNET LevelDB use a simple dictionary with the original user key as the key for the dictionary since performance has been good enough so far. The memtable will only contain the last version of a record (it will overwrite earlier memory versions).

Immutable memtable and compacting memtables

When the current log file exceed the allowed size (2MB default in LevelDB) a background process will create a new current memtable, store the old memtable as an immutable memtable, and eventually write the immutable memtable to a new level 9 table file. The format of this level 0 table follow normal table files. After it is written, it generate a new version for the database, and remove the immutable memtable.