Skip to content

Commit

Permalink
docs: update Sphinx configuration and add new documentation
Browse files Browse the repository at this point in the history 
This updates the sphinx conf.py file so that Dagster packages no longer
need to be manually added to the paths. It also adds a few helper
extensions, mainly `sphinx_copybutton` to allow you to copy a code
snippet when output is rendered as html.
  • Loading branch information
@PedramNavid
PedramNavid committed Aug 20, 2024
1 parent f49d389 commit 896a88d
Show file tree
Hide file tree
Showing 4 changed files with 111 additions and 67 deletions.
5 changes: 4 additions & 1 deletion docs/Makefile
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help
.PHONY: help vale

help:
@egrep -h '\s##\s' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m %-30s\033[0m %s\n", $$1, $$2}'
Expand Down Expand Up @@ -27,3 +27,6 @@ mdx-format: ## Format mdx files
cd next; yarn mdx-format

mdx-full-format: docs_ruff mdx-format

vale:
vale content sphinx
19 changes: 19 additions & 0 deletions docs/sphinx/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Dagster API Documentation

Check failure on line 1 in docs/sphinx/README.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Dagster.section-heading-sentence-case] 'Dagster API Documentation' should be in sentence case

Raw Output:
{"message": "[Dagster.section-heading-sentence-case] 'Dagster API Documentation' should be in sentence case", "location": {"path": "docs/sphinx/README.md", "range": {"start": {"line": 1, "column": 3}}}, "severity": "ERROR"}

This directory contains the Sphinx documentation for the Dagster API.
The core API documentation is created in `index.rst` and the files in `sections/api/apidocs`.

There are currently two workflows for generating the API documentation:

- The existing workflow uses tox to generate JSON output that is read by our NextJS

Check warning on line 8 in docs/sphinx/README.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Google.Contractions] Use 'that's' instead of 'that is'.

Raw Output:
{"message": "[Google.Contractions] Use 'that's' instead of 'that is'.", "location": {"path": "docs/sphinx/README.md", "range": {"start": {"line": 8, "column": 58}}}, "severity": "INFO"}

Check warning on line 8 in docs/sphinx/README.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Google.We] Try to avoid using first-person plural like 'our'.

Raw Output:
{"message": "[Google.We] Try to avoid using first-person plural like 'our'.", "location": {"path": "docs/sphinx/README.md", "range": {"start": {"line": 8, "column": 74}}}, "severity": "WARNING"}
site. Running tox or `make json` will generate JSON output
- The new workflow uses Sphinx to generate HTML output. You can run `make html`
to generate HTML output, and view the static site by opening `build/html/index.html`.

## Development

Run `make install` to install the necessary dependencies for building the documentation.

You can run `make watch` to watch for changes to the API documentation and automatically rebuild the documentation.
Note: some changes might not be reflected immediately, for example style-sheets.

Check warning on line 18 in docs/sphinx/README.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Dagster.eol-whitespace] Remove whitespace characters from the end of the line.

Raw Output:
{"message": "[Dagster.eol-whitespace] Remove whitespace characters from the end of the line.", "location": {"path": "docs/sphinx/README.md", "range": {"start": {"line": 18, "column": 81}}}, "severity": "WARNING"}
You can run `make clean` and `make html` to force a full rebuild.
137 changes: 71 additions & 66 deletions docs/sphinx/conf.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,73 +13,35 @@
import os
import sys

paths = [
### dagster packages
"../../python_modules/automation",
"../../python_modules/dagster",
"../../python_modules/dagster-pipes",
"../../python_modules/dagster-graphql",
"../../python_modules/dagit",
"../../python_modules/dagster-webserver",
"../../python_modules/libraries/dagster-airbyte",
"../../python_modules/libraries/dagster-aws",
"../../python_modules/libraries/dagster-azure",
"../../python_modules/libraries/dagster-celery",
"../../python_modules/libraries/dagster-celery-docker",
"../../python_modules/libraries/dagster-census",
"../../python_modules/libraries/dagster-dask",
"../../python_modules/libraries/dagster-datadog",
"../../python_modules/libraries/dagster-datahub",
"../../python_modules/libraries/dagster-docker",
"../../python_modules/libraries/dagster-embedded-elt",
"../../python_modules/libraries/dagster-embedded-elt/sling",
"../../python_modules/libraries/dagster-embedded-elt/dlt",
"../../python_modules/libraries/dagster-fivetran",
"../../python_modules/libraries/dagster-github",
"../../python_modules/libraries/dagster-k8s",
"../../python_modules/libraries/dagster-looker",
"../../python_modules/libraries/dagster-managed-elements",
"../../python_modules/libraries/dagster-mlflow",
"../../python_modules/libraries/dagster-msteams",
"../../python_modules/libraries/dagster-mysql",
"../../python_modules/libraries/dagster-openai",
"../../python_modules/libraries/dagster-pagerduty",
"../../python_modules/libraries/dagster-pandas",
"../../python_modules/libraries/dagster-pandera",
"../../python_modules/libraries/dagster-papertrail",
"../../python_modules/libraries/dagster-postgres",
"../../python_modules/libraries/dagster-prometheus",
"../../python_modules/libraries/dagster-shell",
"../../python_modules/libraries/dagster-slack",
"../../python_modules/libraries/dagster-snowflake",
"../../python_modules/libraries/dagster-snowflake-pandas",
"../../python_modules/libraries/dagster-snowflake-pyspark",
"../../python_modules/libraries/dagster-spark",
"../../python_modules/libraries/dagster-ssh",
"../../python_modules/libraries/dagster-twilio",
"../../python_modules/libraries/dagstermill",
"../../python_modules/libraries/dagster-celery-k8s",
"../../python_modules/libraries/dagster-dbt",
"../../python_modules/libraries/dagster-ge",
"../../python_modules/libraries/dagster-gcp",
"../../python_modules/libraries/dagster-gcp-pandas",
"../../python_modules/libraries/dagster-gcp-pyspark",
"../../python_modules/libraries/dagster-polars",
"../../python_modules/libraries/dagster-pyspark",
"../../python_modules/libraries/dagster-databricks",
"../../python_modules/libraries/dagster-duckdb",
"../../python_modules/libraries/dagster-duckdb-pandas",
"../../python_modules/libraries/dagster-duckdb-polars",
"../../python_modules/libraries/dagster-duckdb-pyspark",
"../../python_modules/libraries/dagster-wandb",
"../../python_modules/libraries/dagster-deltalake",
"../../python_modules/libraries/dagster-deltalake-pandas",
### autodoc_dagster extension
"./_ext",
]

ignored_folders = ["dagster-test"]
base_path = "../../python_modules"
python_modules_path = os.path.abspath(base_path)
libraries_path = os.path.abspath(os.path.join(base_path, "libraries"))

paths = []

# Add python_modules folders
for folder in os.listdir(python_modules_path):
folder_path = os.path.join(python_modules_path, folder)
if os.path.isdir(folder_path) and not any(
ignored in folder_path for ignored in ignored_folders
):
paths.append(folder_path)

# Add libraries folders
for folder in os.listdir(libraries_path):
folder_path = os.path.join(libraries_path, folder)
if os.path.isdir(folder_path) and not any(
ignored in folder_path for ignored in ignored_folders
):
paths.append(folder_path)

# Add the _ext folder
paths.append(os.path.abspath("./_ext"))

# Insert paths into sys.path
for path in paths:
sys.path.insert(0, os.path.abspath(path))
sys.path.insert(0, path)

# -- Project information -----------------------------------------------------

Expand Down Expand Up @@ -108,6 +70,10 @@
"dagster_sphinx",
# Renders a collapsible HTML component. Used by autodoc_dagster.
"sphinx_toolbox.collapse",
# Adds a copy button to code blocks
"sphinx_copybutton",
# Displays the duration of the build process
"sphinx.ext.duration",
]

# -- Extension configuration -------------------------------------------------
Expand All @@ -125,6 +91,11 @@
# marked `@public` will _not_ be included in the docs!
autodoc_default_options = {"members": True, "undoc-members": True}

# Determines the order in which members (e.g., methods, attributes) are documented
# within a class or module. "groupwise" groups members by type (e.g., methods together,
# attributes together) before sorting alphabetically within each group.
autodoc_member_order = "groupwise"

# List of all packages that should be mocked when autodoc is running. Autodoc is going to import
# dagster packages, which in turn import various third-party packages. The vast majority of those
# packages are not actually needed to build the docs, but autodoc will nonetheless choke if it can't
Expand Down Expand Up @@ -170,8 +141,13 @@
"wandb",
]

# Disable type annotations in the generated documentation.
autodoc_typehints = "none"

# Preserve default values in function signatures when generating documentation.
# This allows readers to see the original default arguments as defined in the source code.
autodoc_preserve_defaults = True

# From https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html#confval-autosectionlabel_prefix_document
# "Prefix each section label with the name of the document it is in, followed by a colon. For
# example, index:Introduction for a section called Introduction that appears in document index.rst.
Expand All @@ -180,3 +156,32 @@

# Only support Google-style docstrings
napoleon_numpy_docstring = False

viewcode_line_numbers = True

# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

html_theme = "furo"
html_static_path = ["_static"]
html_css_files = ["css/style.css"]

# Turo theme options
# See: https://pradyunsg.me/furo/customisation/colors/
html_theme_options = {
"light_css_variables": {
"color-brand-primary": "#4F43DD",
"color-brand-content": "#7C4DFF",
"color-api-name": "var(--color-blurple)",
"color-api-pre-name": "var(--color-blurple)",
"font-stack": "Geist, sans-serif",
"font-stack--monospace": "Geist Mono, monospace",
"font-stack--headings": "Geist, sans-serif",
},
"dark_css_variables": {
"color-api-name": "var(--color-blurple)",
"color-api-pre-name": "var(--color-blurple)",
},
"light_logo": "images/logo-light.svg",
"dark_logo": "images/logo-dark.svg",
}
17 changes: 17 additions & 0 deletions docs/sphinx/requirements.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
sphinx>=7

sphinx-autobuild
# For generating CLI docs
sphinx_click

sphinx_toolbox
sphinxcontrib-serializinghtml

# Generate opengraph data
sphinxext-opengraph

# Copy code button
sphinx-copybutton

# Furo theme
furo

0 comments on commit 896a88d

Please sign in to comment.