Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add complete data ref for basic data payload #25

Open
MSeal opened this issue Feb 11, 2019 · 1 comment
Open

Add complete data ref for basic data payload #25

MSeal opened this issue Feb 11, 2019 · 1 comment
Labels
enhancement New feature or request
Milestone

Comments

@MSeal
Copy link
Member

MSeal commented Feb 11, 2019

We need an end-to-end working example for gluing a data reference and recalling it. A binder which enables this will will dictate the API improvements to enable the functionality.

@MSeal MSeal added this to the beta-complete milestone Feb 11, 2019
@MSeal MSeal added the enhancement New feature or request label Feb 11, 2019
@MSeal
Copy link
Member Author

MSeal commented Jul 7, 2020

For reference: #37 was a stab at achieving this change in combination with other changes. While that PR was too complex and large, we want a similar extension of the glue api contract to support remote referenced scraps. This should enable table references and blob store references that can be recalled programmatically.

Some concerns to hash out more:

  • Does the notebook author need complete reference paths? Likely yes, but custom reference endpoints that get registered could support default prefixes. e.g. ref=s3://bucket/prefix/filename.ext vs ref=filestore://filename.ext
  • Do we reuse papermill's IO registry? Unclear but would like to not have the large dependency chain
  • How should the kwargs for glue change? Contract changes here need to be carefully applied to avoid confusion and usability concerns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant