Implement Dataset and Corpus data loaders. (DistrictDataLabs#535)

This PR implements a significant change in the way yellowbrick handles datasets, moving them from data that can be downloaded and loaded using example code to prime time members of the library that can be loaded into pandas data frames and series or into well-structured numpy arrays with correct data types. 

We have completely overhauled dataset management using the yellowbrick-datasets repository as our data management tool. Data is still stored on S3 but contains .csv.gz and meta.json files for loading into pandas if it's installed or .npz files for loading into valid numpy arrays. New `Dataset` and `Corpus` manage access to the data, downloading it if it's not already on disk and providing access to the contents in the source directory. We maintain our security checking with sha256 hashes and a new manifest.json method. 

Fixes DistrictDataLabs#416

Author: bbengfort

.gitignore

@@ -61,6 +61,7 @@ docs/_build/
 # IDE/editor droppings
 *.swp
 *.swo
+.vscode/settings.json
 
 # OS droppings
 .DS_Store
@@ -120,6 +121,4 @@ fabric.properties
 
 # Data downloaded from Yellowbrick
 data/
-.vscode/settings.json
-
-yellowbrick/datasets/fixtures
+yellowbrick/datasets/fixtures

MANIFEST.in

@@ -4,11 +4,24 @@ include *.txt
 include *.yml
 include *.cfg
 include Makefile
-recursive-include docs *.rst
-recursive-include docs *.jpg
-recursive-include docs *.png
-recursive-include docs *.py
-recursive-include docs Makefile
-recursive-include tests *.py
-recursive-include examples *.py
-recursive-include examples *.ipynb
+include MANIFEST.in
+
+include examples/*.ipynb
+include examples/*.md
+
+graft docs
+prune docs/_build
+
+graft tests
+prune tests/fixtures
+prune tests/actual_images
+
+graft yellowbrick
+prune yellowbrick/datasets/fixtures
+
+global-exclude __pycache__
+global-exclude *.py[co]
+global-exclude .ipynb_checkpoints
+global-exclude .DS_Store
+global-exclude .env
+global-exclude .coverage.*

docs/api/datasets.rst

@@ -79,3 +79,12 @@ Unless otherwise specified, most of the examples currently use one or more of th
 - **mushroom**: suitable for classification/clustering
 - **occupancy**: suitable for classification
 - **spam**: suitable for binary classification
+
+
+API Reference
+-------------
+
+.. automodule:: yellowbrick.datasets.path
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/changelog.rst

@@ -10,6 +10,8 @@ Version 1.0
 * Contributors: Benjamin Bengfort, Rebecca Bilbro, Nathan Danielsen, Kristen McIntyre, Larry Gray, Prema Roman, John Healy, Sourav Singh, Francois Dion, Jerome Massot
 
 Major Changes:
+    - New datasets module that provide greater support for interacting with Yellowbrick example datasets including support for Pandas, npz, and text corpora.
+    - Management repository for Yellowbrick example data, yellowbrick-datasets.
     - Add support for matplotlib 3.0.1 or greater.
     - ``UMAPVisualizer`` as an alternative manifold to TSNE for corpus visualization that is fast enough to not require preprocessing PCA or SVD decomposition and preserves higher order similarities and distances.
 

examples/README.md

@@ -1,66 +1,43 @@
-# Yellowbrick Examples 
+# Yellowbrick Examples
 
 [![Visualizers](../docs/images/visualizers.png)](../docs/images/visualizers.png)
 
-Welcome to the yellowbrick examples directory! This directory contains a gallery of visualizers and their application to classification, regression, clustering, and other machine learning techniques with Scikit-Learn. Examples have been submitted both by the Yellowbrick team and also users like you! The result is a rich gallery of tools and techniques to equip your machine learning with visual diagnostics and visualizer workflows! 
+Welcome to the yellowbrick examples directory! This directory contains a gallery of visualizers and their application to classification, regression, clustering, and other machine learning techniques with scikit-learn. Examples have been submitted both by the Yellowbrick team and also users like you! The result is a rich gallery of tools and techniques to equip your machine learning with visual diagnostics and visualizer workflows!
 
-## Getting Started 
+## Getting Started
 
-The notebook to explore first is the `examples.ipynb` Jupyter notebook. This notebook contains the executable examples from the tutorial in the documentation. However, before you can successfully run this notebook, you must first download the sample datasets. To download the samples run the downloader script:
+The notebook to explore first is the `examples.ipynb` Jupyter notebook. This notebook contains the executable examples from the tutorial in the documentation. You can run the notebook as follows:
 
 ```
-$ python download.py 
+$ jupyter notebook examples.ipynb
 ```
 
-This should create a directory called `examples/data`, which in turn will contain CSV or text datasets. There are two primary problems that the download script may have: first, you may get the error `"The requests module is required to download data"`. To fix this problem:
+If you don't have jupyter installed, or other dependencies, you may have to `pip install` them.
 
-```
-$ pip install requests 
-```
-
-The second problem may be `"Download signature does not match hardcoded signature!"` This problem means that the file you're trying to download has changed. Either download a more recent version of Yellowbrick, or use the URLs in the `download.py` script to fetch the data manually. If there are any other problems, please notify us via [GitHub Issues](https://github.com/DistrictDataLabs/yellowbrick/issues). 
-
-Once the example data has been downloaded, you can run the examples notebook as follows:
-
-```
-$ jupyter notebook examples.ipynb 
-```
-
-If you don't have jupyter installed, or other dependencies, you may have to `pip install` them. 
-
-## Organization 
+## Organization
 
 The examples directory contains many notebooks, folders and files. At the top level you will see the following:
 
-- examples.ipynb: a notebook with executable versions of the tutorial visualizers 
-- download.py: a script to download the example data sets 
-- palettes.ipynb: a visualization of the Yellowbrick palettes 
-- data: a directory containing the example datasets. 
+- examples.ipynb: a notebook with executable versions of the tutorial visualizers
+- palettes.ipynb: a visualization of the Yellowbrick palettes
+- regression.ipynb: a notebook exploring the regression model visualizers.
 
-In addition to these files and directory, you will see many other directories, whose names are the GitHub usernames of their contributors. You can explore these user submitted examples or submit your own! 
+In addition to these files and directory, you will see many other directories, whose names are the GitHub usernames of their contributors. You can explore these user submitted examples or submit your own!
 
-### Contributing 
+### Contributing
 
 To contribute an example notebook of your own, perform the following steps:
 
-1. Fork the repository into your own account 
-2. Checkout the develop branch (see [contributing to Yellowbrick](http://www.scikit-yb.org/en/latest/about.html#contributing) for more. 
-3. Create a directory in the repo, `examples/username` where username is your GitHub username. 
-4. Create a notebook in that directory with your example. See [user testing](http://www.scikit-yb.org/en/latest/evaluation.html) for more. 
-5. Commit your changes back to your fork. 
-6. Submit a pull-request from your develop branch to the Yellowbrick develop branch. 
-7. Complete the code review steps with a Yellowbrick team member. 
-
-That's it -- thank you for contributing your example! 
-
-A couple of notes. First, please make sure that the Jupyter notebook you submit is "run" -- that is it has the output saved to the notebook and is viewable on GitHub (empty notebooks don't serve well as a gallery). Second, please do not commit datasets, but instead provide instructions for downloading the dataset. You can create a downloader utility similar to ours. 
-
-One great tip, is to create your PR right after you fork the repo; that way we can work with you on the changes you're making and communicate about how to have a very successful contribution! 
+1. Fork the repository into your own account
+2. Checkout the develop branch (see [contributing to Yellowbrick](http://www.scikit-yb.org/en/latest/about.html#contributing) for more.
+3. Create a directory in the repo, `examples/username` where username is your GitHub username.
+4. Create a notebook in that directory with your example. See [user testing](http://www.scikit-yb.org/en/latest/evaluation.html) for more.
+5. Commit your changes back to your fork.
+6. Submit a pull-request from your develop branch to the Yellowbrick develop branch.
+7. Complete the code review steps with a Yellowbrick team member.
 
-### User Examples 
+That's it -- thank you for contributing your example!
 
-In this section we want to thank our examples contributors, and describe their notebooks so that you can find an example similar to your application! 
+A couple of notes. First, please make sure that the Jupyter notebook you submit is "run" -- that is it has the output saved to the notebook and is viewable on GitHub (empty notebooks don't serve well as a gallery). Second, please do not commit datasets, but instead provide instructions for downloading the dataset. You can create a downloader utility similar to ours.
 
-- [bbengfort](https://github.com/bbengfort): visualizing text classification 
-- [rebeccabilbro](https://github.com/rebeccabilbro): visualizing book reviews data 
-- [nathan](https://github.com/ndanielsen/): visualizing the Iris dataset 
+One great tip, is to create your PR right after you fork the repo; that way we can work with you on the changes you're making and communicate about how to have a very successful contribution!

examples/download.py

@@ -1,5 +1,6 @@
 [metadata]
 description-file = DESCRIPTION.txt
+license_file = LICENSE.txt
 
 [wheel]
 universal = 1