Add documentation with MkDocs

.github/workflows/docs.yml

@@ -0,0 +1,52 @@
+name: Deploy docs
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - 'master'
+  pull_request:
+permissions:
+  contents: write
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v4
+
+      - name: Configure Git Credentials
+        run: |
+          git config user.name github-actions[bot]
+          git config user.email 41898282+github-actions[bot]@users.noreply.github.com
+        if: (github.event_name != 'pull_request')
+
+      - name: Set up Python 3.9
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.9'
+          cache: 'pip'
+          cache-dependency-path: |
+            setup.py
+            requirements-docs.txt
+
+      - name: Save time for cache for mkdocs
+        run: echo "cache_id=$(date --utc '+%V')" >> $GITHUB_ENV
+
+      - name: Caching
+        uses: actions/cache@v4
+        with:
+          key: mkdocs-material-${{ env.cache_id }}
+          path: .cache
+          restore-keys: |
+            mkdocs-material-
+
+      - name: Install Dependencies
+        run: pip install -r requirements-docs.txt
+
+      - name: Deploy to GitHub Pages
+        run: mkdocs gh-deploy --force
+        if: (github.event_name != 'pull_request')
+
+      - name: Build docs to check for errors
+        run: mkdocs build
+        if: (github.event_name == 'pull_request')

docs/api_docs/python/tfma-constants.md

@@ -0,0 +1,3 @@
+# TFMA Constants
+
+::: tensorflow_model_analysis.constants

docs/api_docs/python/tfma-contrib.md

@@ -0,0 +1,3 @@
+# TFMA Constants
+
+::: tensorflow_model_analysis.contrib

docs/api_docs/python/tfma-evaluators.md

@@ -0,0 +1,5 @@
+
+# TFMA Evaluators
+
+::: tensorflow_model_analysis.evaluators
+

docs/api_docs/python/tfma-experimental.md

@@ -0,0 +1,5 @@
+
+# TFMA Experimental
+
+::: tensorflow_model_analysis.experimental
+

docs/api_docs/python/tfma-extractors.md

@@ -0,0 +1,5 @@
+
+# TFMA Extractors
+
+::: tensorflow_model_analysis.extractors
+

docs/api_docs/python/tfma-metrics.md

@@ -0,0 +1,5 @@
+
+# TFMA Metrics
+
+::: tensorflow_model_analysis.metrics
+

docs/api_docs/python/tfma-post_export_metrics.md

@@ -0,0 +1,5 @@
+
+# TFMA Post_Export_Metrics
+
+::: tensorflow_model_analysis.post_export_metrics
+

docs/api_docs/python/tfma-sdk.md

@@ -0,0 +1,5 @@
+
+# TFMA SDK
+
+::: tensorflow_model_analysis.sdk
+

docs/api_docs/python/tfma-types.md

@@ -0,0 +1,5 @@
+
+# TFMA Types
+
+::: tensorflow_model_analysis.types
+

docs/api_docs/python/tfma-utils.md

@@ -0,0 +1,5 @@
+
+# TFMA Utils
+
+::: tensorflow_model_analysis.utils
+

docs/api_docs/python/tfma-validators.md

@@ -0,0 +1,5 @@
+
+# TFMA Validators
+
+::: tensorflow_model_analysis.validators
+

docs/api_docs/python/tfma-version.md

@@ -0,0 +1,5 @@
+
+# TFMA Version
+
+::: tensorflow_model_analysis.version
+

docs/api_docs/python/tfma-view.md

@@ -0,0 +1,5 @@
+
+# TFMA View
+
+::: tensorflow_model_analysis.view
+

docs/api_docs/python/tfma-writers.md

@@ -0,0 +1,5 @@
+
+# TFMA Writers
+
+::: tensorflow_model_analysis.writers
+

docs/api_docs/python/tfma.md

@@ -0,0 +1,3 @@
+# TFMA
+
+::: tensorflow_model_analysis

g3doc/faq.md → docs/faq.md

@@ -49,7 +49,7 @@ TFMA supports keras models, models based on generic TF2 signature APIs, as well
 TF estimator based models (although depending on the use case the estimator
 based models may require an `EvalSavedModel` to be used).
 
-See [get_started](get_started.md) guide for the full list of model types
+See [get started](get_started.md) guide for the full list of model types
 supported and any restrictions.
 
 ### How do I setup TFMA to work with a native keras based model?
@@ -61,7 +61,7 @@ assumptions:
     (this can be changed using `model_specs[0].signature_name`).
 *   Built in metrics from `model.compile(...)` should be evaluated (this can be
     disabled via `options.include_default_metric` within the
-    [tfma.EvalConfig](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/EvalConfig)).
+    [tfma.EvalConfig](../api_docs/python/tfma#tensorflow_model_analysis.EvalConfig)).
 
 ```python
 from google.protobuf import text_format
@@ -425,7 +425,7 @@ single label per example, not for multi-label.
 This is most likely caused because the metrics are configured for a binary
 classification problem, but the model is outputing probabilities for both of the
 classes instead of just one. This is common when
-[tensorflow's classification API](https://www.tensorflow.org/tfx/serving/signature_defs#classification_signaturedef)
+[tensorflow's classification API](https://tensorflow.github.io/serving/guide/signature_defs/#classification-signaturedef)
 is used. The solution is to choose the class that you would like the predictions
 to be based on and then binarize on that class. For example:
 

g3doc/get_started.md → docs/get_started.md

@@ -25,7 +25,7 @@ If you just want to jump in and get started, check out our
 notebook.
 
 This page can also be viewed from
-[tensorflow.org](https://www.tensorflow.org/tfx/model_analysis/get_started).
+[tensorflow.org](https://tensorflow.github.io/model-analysis/get_started).
 
 ## Model Types Supported
 
@@ -75,19 +75,19 @@ different model types.
 ## Setup
 
 Before running an evaluation, a small amount of setup is required. First, a
-[`tfma.EvalConfig`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/EvalConfig)
+[`tfma.EvalConfig`](../api_docs/python/tfma#tensorflow_model_analysis.EvalConfig)
 object must be defined that provides specifications for the model, metrics, and
 slices that are to be evaluated. Second a
-[`tfma.EvalSharedModel`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/default_eval_shared_model)
+[`tfma.EvalSharedModel`](../api_docs/python/tfma#tensorflow_model_analysis.default_eval_shared_model)
 needs to be created that points to the actual model (or models) to be used
 during the evaluation. Once these have been defined, evaluation is performed by
 calling
-[`tfma.run_model_analysis`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/run_model_analysis)
+[`tfma.run_model_analysis`](../api_docs/python/tfma#tensorflow_model_analysis.run_model_analysis)
 with an appropriate dataset. For more details, see the [setup](setup.md) guide.
 
 If running within a TFX pipeline, see the TFX
-[guide](https://www.tensorflow.org/tfx/guide) for how to configure TFMA to run
-as a TFX [Evaluator](https://www.tensorflow.org/tfx/guide/evaluator) component.
+[guide](https://tensorflow.github.io/tfx/guide) for how to configure TFMA to run
+as a TFX [Evaluator](https://tensorflow.github.io/tfx/guide/evaluator) component.
 
 ## Examples
 
@@ -296,4 +296,4 @@ frontend components included in TFMA. For example:
 *   [Visualizations](visualizations.md)
 *   [Architecture](architecture.md)
 *   [FAQ](faq.md)
-*   [API Reference](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma)
+*   [API Reference](../api_docs/python/tfma)

docs/index.md

@@ -0,0 +1,39 @@
+# Improving Model Quality With TensorFlow Model Analysis
+
+## Introduction
+
+As you tweak your model during development, you need to check whether your
+changes are improving your model. Just checking accuracy may not be enough. For
+example, if you have a classifier for a problem in which 95% of your instances
+are positive, you may be able to improve accuracy by simply always predicting
+positive, but you won't have a very robust classifier.
+
+## Overview
+
+The goal of TensorFlow Model Analysis is to provide a mechanism for model
+evaluation in TFX. TensorFlow Model Analysis allows you to perform model
+evaluations in the TFX pipeline, and view resultant metrics and plots in a
+Jupyter notebook. Specifically, it can provide:
+
+*   [Metrics](https://tensorflow.github.io/model-analysis/metrics) computed on entire training and holdout
+    dataset, as well as next-day evaluations
+*   Tracking metrics over time
+*   Model quality performance on different feature slices
+*   [Model validation](https://tensorflow.github.io/model-analysis/model_validations) for ensuring that
+    model's maintain consistent performance
+
+## Next Steps
+
+Try our [TFMA tutorial](https://tensorflow.github.io/tfx/tutorials/model_analysis/tfma_basic).
+
+Check out our [github](https://github.com/tensorflow/model-analysis) page for
+details on the supported
+[metrics and plots](https://tensorflow.github.io/model-analysis/metrics) and associated notebook
+[visualizations](https://tensorflow.github.io/model-analysis/visualizations).
+
+See the [installation](https://tensorflow.github.io/model-analysis/install) and
+[getting started](https://tensorflow.github.io/model-analysis/get_started) guides for information and
+examples on how to get [set up](https://tensorflow.github.io/model-analysis/setup) in a standalone
+pipeline. Recall that TFMA is also used within the [Evaluator](evaluator.md)
+component in TFX, so these resources will be useful for getting started in TFX
+as well.

docs/javascripts/mathjax.js

@@ -0,0 +1,19 @@
+window.MathJax = {
+  tex: {
+    inlineMath: [["\\(", "\\)"]],
+    displayMath: [["\\[", "\\]"]],
+    processEscapes: true,
+    processEnvironments: true
+  },
+  options: {
+    ignoreHtmlClass: ".*|",
+    processHtmlClass: "arithmatex"
+  }
+};
+
+document$.subscribe(() => { 
+  MathJax.startup.output.clearCache()
+  MathJax.typesetClear()
+  MathJax.texReset()
+  MathJax.typesetPromise()
+})

g3doc/metrics.md → docs/metrics.md

@@ -10,13 +10,13 @@ TFMA supports the following metrics and plots:
         are computed outside of the graph in beam using the metrics classes
         directly.
 *   Standard TFMA metrics and plots
-    ([`tfma.metrics.*`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/metrics))
+    ([`tfma.metrics.*`](../api_docs/python/tfma-metrics))
 
 *   Custom keras metrics (metrics derived from
     [`tf.keras.metrics.Metric`](https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Metric))
 
 *   Custom TFMA metrics (metrics derived from
-    [`tfma.metrics.Metric`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/metrics/Metric))
+    [`tfma.metrics.Metric`](../api_docs/python/tfma-metrics#tensorflow_model_analysis.metrics.Metric))
     using custom beam combiners or metrics derived from other metrics).
 
 NOTE: In TFMA, plots and metrics are both defined under the metrics library. By
@@ -38,10 +38,10 @@ classification, ranking, etc.
 ## Configuration
 
 There are two ways to configure metrics in TFMA: (1) using the
-[`tfma.MetricsSpec`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/api/MetricsSpec)
+[`tfma.MetricsSpec`](../api_docs/python/tfma#tensorflow_model_analysis.metricsSpec)
 or (2) by creating instances of `tf.keras.metrics.*` and/or `tfma.metrics.*`
 classes in python and using
-[`tfma.metrics.specs_from_metrics`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/api/metrics/specs_from_metrics)
+[`tfma.metrics.specs_from_metrics`](../api_docs/python/tfma-metrics#tensorflow_model_analysis.metrics.specs_from_metrics)
 to convert them to a list of `tfma.MetricsSpec`.
 
 The following sections describe example configurations for different types of
@@ -598,7 +598,7 @@ All the supported plots are stored in a single proto called
 ### EvalResult
 
 The return from an evaluation run is an
-[`tfma.EvalResult`](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/EvalResult).
+[`tfma.EvalResult`](../api_docs/python/tfma#tensorflow_model_analysis.EvalResult).
 This record contains `slicing_metrics` that encode the metric key as a
 multi-level dict where the levels correspond to output name, class ID, metric
 name, and metric value respectively. This is intended to be used for UI display
@@ -702,7 +702,7 @@ will be used by the combiner (see [architecture](architecture.md) for more info
 on what are extracts). All preprocessors will be executed sequentially in the
 order of the list. If the `preprocessors` is empty, then the combiner will be
 passed
-[StandardMetricInputs](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/metrics/StandardMetricInputs)
+[StandardMetricInputs](../api_docs/python/tfma-metrics#tensorflow_model_analysis.metrics.StandardMetricInputs)
 (standard metric inputs contains labels, predictions, and example_weights). The
 `combiner` is a `beam.CombineFn` that takes a tuple of (slice key, preprocessor
 output) as its input and outputs a tuple of (slice_key, metric results dict) as
@@ -713,7 +713,7 @@ Note that slicing happens between the `preprocessors` and `combiner`.
 Note that if a metric computation wants to make use of both the standard metric
 inputs, but augment it with a few of the features from the `features` extracts,
 then the special
-[FeaturePreprocessor](https://www.tensorflow.org/tfx/model_analysis/api_docs/python/tfma/metrics/FeaturePreprocessor)
+[FeaturePreprocessor](../api_docs/python/tfma-metrics#tensorflow_model_analysis.metrics.FeaturePreprocessor)
 can be used which will merge the requested features from multiple combiners into
 a single shared StandardMetricsInputs value that is passed to all the combiners
 (the combiners are responsible for reading the features they are interested in

docs/stylesheets/extra.css

@@ -0,0 +1,42 @@
+:root {
+  --md-primary-fg-color:        #FFA800;
+  --md-primary-fg-color--light: #CCCCCC;
+  --md-primary-fg-color--dark:  #425066;
+}
+
+.video-wrapper {
+  max-width: 240px;
+  display: flex;
+  flex-direction: row;
+}
+.video-wrapper > iframe {
+  width: 100%;
+  aspect-ratio: 16 / 9;
+}
+
+.buttons-wrapper {
+    flex-wrap: wrap;
+    gap: 1em;
+    display: flex;
+    /* flex-grow: 1; */
+    /* justify-content: center; */
+    /* align-content: center; */
+}
+
+.buttons-wrapper > a {
+    justify-content: center;
+    align-content: center;
+    flex-wrap: nowrap;
+    /* gap: 1em; */
+    align-items: center;
+    text-align: center;
+    flex: 1 1 30%;
+    display: flex;
+}
+
+.md-button > .buttons-content {
+    align-items: center;
+    justify-content: center;
+    display: flex;
+    gap: 1em;
+}