v10 #425
v10 #425
Conversation
|
@timgit Can you put a couple of words on the motivation behind this major version? Which issues does this solve? I'm saying this with the utmost understanding of what it means to release something as open source - This is your project, and I don't expect you to justify every decision. Still, I'm just curious about the reasoning, and I want to hear about any improvements that should cause us to consider upgrading :) |
|
The primary motivation is stability and scalability. In my and others' experience, once a single queue exceeded 1-2mil created jobs in a backlog state, postgres was unable to quickly locate the next record for locking via SKIP LOCKED. This is mentioned as a limitation in the linked article in the readme. This produced high memory and CPU utilization issues on the server, which would then impact all other databases. The problem with pg-boss v9's design is that once any queue reaches this failure state, all queues, including internal maintenance queues, were affected since all jobs shared a single table. The design goal was to mitigate a problematic queue by isolating its storage into a dedicated table via partitioning. This doesn't necessarily remove this limitation of SKIP LOCKED, but it should in theory be less of a catastrophic failure if and when it occurs in the future. Over the years, there were several issues opened around queue concurrency, which added several functions and configuration which made the API more complicated and difficult to understand ( After a couple of failed attempts at building a reasonable migration plan into the new table structure, I decided it was best to release v10 without a schema migration. This wasn't my first choice, but I think it still was the cleanest way to make it predictable what to expect between these versions. |
No description provided.