feat(gatsby-source-s3): download non-image files

[NelisDrost]

I'd like to be able to retrieve files other than images from S3, and access them through GraphQL.

Currently, localfile nodes are only generated in GraphQL for image files, and not e.g.: mdx, json or other data/text files.

I'm specifically interested in mdx files (which I assume could then be ingested by gatsby-transformer-mdx).

Is there a design reason for only creating local file nodes for images? If there are drawbacks to handling all files, a config option for other file extensions to accept seems like a clean solution, e.g.:
{ resolve: 'gatsby-source-s3', options: { aws: { ... }, buckets: [...], file_extensions: ['.mdx', '.json'], },

[moonmeister]

@robinmetral may know the answer. If there's no particular reason I'd be happy to accept a PR.

[sebhewelt]

I was trying to understand why it doesn't process my markdown files. I believe gatsby-source-s3-images would be a more accurate name for this source plugin, rather than gatsby-source-s3 😅 Looking forward to that enhancement 👍👏

[moonmeister]

@sebhewelt You interested in contributing such an enhancement?

[robinmetral]

Hey folks, I'm super sorry for the extremely late reply here, I had ignored notifications on this repo for some reason 🙈

This was previously requested for .md files but never implemented. Here's the ref on the archived repo: gatsby-uc/gatsby-source-s3#505 (comment)

tl;dr there is no specific design reason weighing against this. The only challenge is that the plugin was built specifically to download image files for processing by sharp, not for any other purpose—so I'm not sure what exactly needs to change beyond removing this isImage check:

plugins/packages/gatsby-source-s3/src/gatsby-node.ts

Line 123 in 60fecbc

if (node.internal.type === "S3Object" && node.Key && isImage(node.Key)) {

For example it's possible that the schema needs to be different, depending on what you'd like to do with your files. This would probably require some fiddling/testing but can probably be done 🙂

I unfortunately don't have any time to implement this (and I haven't worked with Gatsby for some time, so would probably not be the best person to do it). But consider this a green light for a contribution 🟢