Automatic offset tracking for stream queues

[viktorerlingsson]

WHAT is this pull request doing?

Adds broker tracking of consumer offsets in streams if no x-stream-offset is provided by the consumer. Does not track if the consumer tag is generated by the broker.

✅ When to run cleanup_consumer_offsets?
✅ IndexError when trying to cleanup if msg_size => segment_size

HOW can this pull request be tested?

Run specs

[carlhoerberg]

What happens with consumer_offset_capacity is reached? MFile doesn't auto expand.

[viktorerlingsson]

What happens with consumer_offset_capacity is reached? MFile doesn't auto expand.

Fixed here

[carlhoerberg]

Need to think about replication of the consumer offset file.

[viktorerlingsson]

@carlhoerberg Performance seems pretty similar with both solutions, maybe even slightly quicker with append-only (I increased file size to Config.instance.segment_size to make sure we don't have to constantly GC). So I think we can go with that solution 👍

[carlhoerberg]

Cool! Havent read all code but looks like the file is still expanded? Is that nessecary? Shouldnt a GC be issued instead? The file size should be a minimum segment size yes, but larger if there are many consumers we are tracking for, so calc the size of the file after GC and the capacity should be a multiple of that, say 1000-10000 larger or so, so that the offset of each consumer could be updated that many times without causing a new GC.

[viktorerlingsson]

Cool! Havent read all code but looks like the file is still expanded? Is that nessecary? Shouldnt a GC be issued instead? The file size should be a minimum segment size yes, but larger if there are many consumers we are tracking for, so calc the size of the file after GC and the capacity should be a multiple of that, say 1000-10000 larger or so, so that the offset of each consumer could be updated that many times without causing a new GC.

Yeah, it will try to compact the file first (only keep the latest offset for each ctag). If the file is still full after that (i.e. there was nothing to compact) it will be expanded. Probably safer to set the capacity to a multiple of tracked ctags as you say though. Will look at that tomorrow!

        cleanup_consumer_offsets if consumer_offset_file_full?(consumer_tag)
        expand_consumer_offset_file if consumer_offset_file_full?(consumer_tag)

[viktorerlingsson]

Cool! Havent read all code but looks like the file is still expanded? Is that nessecary? Shouldnt a GC be issued instead? The file size should be a minimum segment size yes, but larger if there are many consumers we are tracking for, so calc the size of the file after GC and the capacity should be a multiple of that, say 1000-10000 larger or so, so that the offset of each consumer could be updated that many times without causing a new GC.

@carlhoerberg Added this. Do we need to set some form of max size for the offsets file? capacity could in theory be very large if there are many consumers (and/or if ctags are long).

[viktorerlingsson]

@carlhoerberg added some replication for the consumer offsets file. Not really sure what the best approach is for when we replace the file during compaction though. Just replace_file after we've done the replace? Or do we also want to stream writing to a temp file and then replace the current (and then deleting temp file)? We don't have support for renaming files in replication now I guess

[carlhoerberg]

Ah, well spotted, can implement replace file on the client side with write-to-tempfile/rename in another PR

[carlhoerberg]

Should revive this