Memory consumption issue while streaming.

[wuarmin]

Hello @jeremyevans ,

I observe a weird issue while using #stream in one of my apps. Following code (which runs in a sidekiq worker) allocates lots of memory and retains it afterwards, so it's a heavy memory leak.

@relation.stream.each do |row| # relation is a ROM::Relation[:sql] (under the hood "sequel_pg", "~> 1.15")
  # do nothing with row, but the memory consumption increases
end

#paged_each (cursor-strategy) does not suffer from this "memory leak"

@relation.paged_each do |row| # relation is a ROM::Relation[:sql] (under the hood "sequel_pg", "~> 1.15")
  # memory consumption is stable
end

What could be the reason?

Thanks

[wuarmin]

I believe the problem lies not at Sequel. I think it's the mapping to a ROM::Struct, which happens at the each after the stream. WDYT?

[jeremyevans]

That's likely, since Sequel by default does not retain the records in memory for each. I have never used ROM, so I couldn't tell you for sure. You could try reproducing the issue with stream.each using Sequel without ROM. That should help determine where the problem is.

[wuarmin]

I will create an issue at romrb. This behavior can cause a hard time for users as it takes time to figure out the reasons.