Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow direct-to-bucket uploads for file fields #22

Closed
thclark opened this issue Oct 6, 2022 · 1 comment
Closed

Allow direct-to-bucket uploads for file fields #22

thclark opened this issue Oct 6, 2022 · 1 comment
Assignees
Labels
backend Related to the back end feature A new feature of the app

Comments

@thclark
Copy link
Contributor

thclark commented Oct 6, 2022

Feature request

Use Case

To populate file fields, we may wish to upload files that are bigger than the 32mb limit imposed by Cloud Run (which is pretty sensibly chosen).

To circumvent that, django would have to be deployed with an agsi server (typically daphne) and accept cleartext http2 connections per the Cloud Run requirements. It's not clear whether that's yet technically possible with daphne, but we can follow this thread with interest and will ultimately recommend deployment with http2 enabled.

In the meantime (and still afterward, because regardless of http2 compatibility, it doesn't make sense to be passing large files through your servers), we need to support a pattern in which files can be uploaded direct to bucket without choking your django server.

Proposed solution

What we need is:

  • a way to provide the client with a pre-signed upload url when rendering ADD or CHANGE actions in the admin or other views
  • client-side JS that executes the upload using that url, rather than uploading the file with the form data
  • a widget that works in both wagtail and django-admin to implement this
  • a storage class and/or model FileField that allows creation of files in this way

Some hints

  • The django-direct-cloud-upload library has already achieved this, so strong inspiration should be taken from there, and we can probably reuse some code under their BSD3 license. However, it doesn't work with a 'normal' file field out of the box (so it's not a drop-in replacement, depending on how you later use the model or forms). There are also a number of setup steps that I think we can ease by providing a model field directly.
@thclark thclark self-assigned this Nov 10, 2022
@thclark thclark added backend Related to the back end feature A new feature of the app labels Nov 10, 2022
@thclark thclark moved this to Priority 3 (High) in Octue Board Nov 10, 2022
@thclark thclark assigned cortadocodes and unassigned thclark Feb 3, 2023
thclark added a commit that referenced this issue May 4, 2023
…ucket-uploads-for-file-fields

Feature/issue #22 allow direct to bucket uploads for file fields
thclark added a commit that referenced this issue May 4, 2023
…ucket-uploads-for-file-fields

Feature/issue #22 allow direct to bucket uploads for file fields
@thclark
Copy link
Contributor Author

thclark commented Jun 26, 2023

YAY! Done

@thclark thclark closed this as completed Jun 26, 2023
@github-project-automation github-project-automation bot moved this from Priority 3 (High) to Done in Octue Board Jun 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Related to the back end feature A new feature of the app
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants