[docs/revamp] proof-of-concept for synchronizing API docs with Docusaurus

[cmpadden]

Summary & Motivation

Sphinx supports builders to create outputs in various formats: html, latex, etc. However, a third party extension sphinx_markdown_builder is required to generate markdown files for your Python project.

This proof-of-concept generates markdown files, and then copies them to the docs/sphinx directory of our Docusaurus project. Here is an example of how it is rendered:

Note that Docusaurus uses an MDX parser by default, which causes problems with the markdown files output by Sphinx. For example < and > would be interpreted as HTML tags. Docusaurus also supports the CommonMark format and can automatically detect if the file is mdx or md by setting config.markdown.format to detect (reference).

  markdown: {
    mermaid: true,
    format: 'detect',
  },

Whether this is an ideal way to synchronize our docs is tbd...

How I Tested These Changes

cd sphinx
make sync-docusaurus

cd ../docs-next
pnpm start

[netlify]

❌ Deploy Preview for dagsterapidocs failed.

Name	Link
🔨 Latest commit	b3b066b
🔍 Latest deploy log	https://app.netlify.com/sites/dagsterapidocs/deploys/66bb50e9ca254e0008236639

[github-actions]

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-k1vh8q2sh-elementl.vercel.app
https://colton-docs-revamp-sphinx-poc.dagster.dagster-docs.io

Direct link to changed pages:

[PedramNavid]

I don't think we should commit the converted docs, this should be part of the build process otherwise we'll end up with merge conflicts down the line.

The sphinx extension looks fairly straightforward, and I bet we can override it as well if we need to in order to enable other components. Could be interesting.

[PedramNavid]

What's this?

[cmpadden]

Weird! I'm not sure.

[cmpadden]

I don't think we should commit the converted docs, this should be part of the build process otherwise we'll end up with merge conflicts down the line.

The sphinx extension looks fairly straightforward, and I bet we can override it as well if we need to in order to enable other components. Could be interesting.

Removed the generated files. Note that I had to remove the sidebar link for the time being, as when the files don't exist it throws errors.

If we decide to, we could still have it built locally so that vale can run, and then gitignore it. This way there is a tighter feedback loop than build -> ci -> vale fails on generated docs.