feat: Support passing context to the pagination class initialization scope

[edgarrmondragon]

Feature scope

Taps (catalog, state, stream maps, etc.)

Description

Some pagination schemes may required access to the sync context object.

The RESTStream.get_new_paginator method is already being called within RESTStream.request_records, which has access to the context object, so it'd be rather easy to update the signature of the former to support passing it.

sdk/singer_sdk/streams/rest.py

Line 354 in 47290d0

paginator = self.get_new_paginator()

[edgarrmondragon]

To accomplish this in a backwards compatible way we'd need to do something like

try:
    paginator = self.get_new_paginator(context=context)
except TypeError:
    paginator = self.get_new_paginator()

An even better way to accomplish the purpose of this feature request would be to make the context accessible within any scope within the stream class:

class Stream:
  @final
  def get_context():
    return self.__context

  def get_new_paginator(self):
    context = self.get_context()
    start_dt = self.get_starting_datetime(context)
    return MyPaginator(start_dt)

[stale]

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

[edgarrmondragon]

Still relevant

[kinghuang]

I've just run into this myself. I'm implementing a parent-child stream, and need the context from the parent in order to create the paginator in the child.

[edgarrmondragon]

I've just run into this myself. I'm implementing a parent-child stream, and need the context from the parent in order to create the paginator in the child.

PRs are welcome for this feature!

[edgarrmondragon]

Small enough and I found the time to address this: #2529