Merge branch 'master' into feature/png_compression

.flake8

@@ -0,0 +1,3 @@
+[flake8]
+max-line-length = 88
+extend-ignore = E203, W503

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,48 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: "[BUG]"
+labels: ''
+assignees: ''
+
+---
+
+## 🐛🐛 Bug Report
+
+
+### ⚗️ Current Behavior
+A clear and concise description of the behavior.
+
+**Input Code**
+- REPL or Repo link if applicable:
+
+```python
+def function_wrong():
+    pass
+```
+
+**Expected behavior/code**
+A clear and concise description of what you expected to happen (or code).
+
+```python
+def function_right():
+    # Here is the fix
+    print("Ok)
+
+```
+
+### ⚙️ Environment
+
+- `Python` version(s):
+    - *good*: [e.g. 3.8]
+    - **better**: [3.8.6 - Clang 12.0.0 (clang-1200.0.32.27)]
+- `OS`: [e.g. Ubuntu 18.04, OSX 10.13.4, Windows 10]
+- `IDE`: [Vim, VS-Code, Pycharme]
+- `Packages`: [ `Tensorflow==2.1.2 - latest`]
+
+
+### 🧰 Possible Solution  (optional)
+<!--- Only if you have suggestions on a fix for the bug -->
+
+### 🖼 Additional context/Screenshots (optional)
+Add any other context about the problem here. If applicable, add screenshots to help explain.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,34 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: "[FEATURE]"
+labels: ''
+assignees: ''
+
+---
+
+## 🚨🚨 Feature Request
+
+- [ ] Related to an existing [Issue](../../../issues) 
+- [ ] A new implementation (Improvement, Extension) 
+
+### Is your feature request related to a problem?
+
+A clear and concise description of what the problem is. Ex. I have an issue when [...]
+
+### If your feature will improve `HUB`
+
+A clear and concise description of how it will help `HUB`. Please prefer references, if possible [...]
+
+
+### Description of the possible solution
+
+A clear and concise description of what you want to happen. Add any considered drawbacks.
+
+### An alternative solution to the problem can look like
+
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Teachability, Documentation, Adoption, Migration Strategy**
+If you can, explain how users will be able to use this and possibly write out a version the docs.
+Maybe a screenshot or design?

.github/PULL_REQUEST_TEMPLATE/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,26 @@
+## 🚀 🚀 Pull Request
+
+### All Submissions:
+
+- [ ] Have you followed the guidelines in our Contributing document?
+- [ ] Have you checked to ensure there aren't other open [Pull Requests](../../../pulls) for the same update/change?
+- [ ] Is your [Pull Requests](../../../pulls) linked to an [Issue](../../../issues)
+
+
+
+### New Feature Submissions:
+
+- [ ] Does your submission pass all `tests`?
+- [ ] Does your submission not lower the `coverage-rate`?
+- [ ] Have you lint your code locally prior to submission; including `black` style?
+- [ ] Have you describe your new feature in the `docs`?
+
+
+### Changes to Core Features:
+
+- [ ] Have you added an explanation of what your changes do and why you'd like us to include them?
+- [ ] Have you written new `tests` for your core changes, as applicable?
+- [ ] Have you successfully ran `tests` with your changes locally?
+- [ ] Have you keep the `coverage-rate` constant?
+- [ ] Have you keep the `coverage-rate` up?
+- [ ] Are the `docs` up to date?

CONTRIBUTING.md

@@ -19,6 +19,42 @@ To contribute a feature/fix:
 * Add a feature. For an extended feature please create an issue to discuss.
 
 
+## Get Started
+
+Ready to contribute? Here's how to set up `Hub` for local development.
+
+1. Fork the `Hub` repo on GitHub.
+2. Clone your fork locally:
+```
+    $ git clone [email protected]:your_name_here/hub.git
+```
+3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed,
+   this is how you set up your fork for local development:
+```
+    $ mkvirtualenv Hub
+    $ cd Hub/
+```
+4. Create a branch for local development:
+```
+    $ git checkout -b name-of-your-bugfix-or-feature
+```
+   Now you can make your changes locally.
+
+5. While hacking your changes, make sure to cover all your developments with the required
+   unit tests, and that none of the old tests fail as a consequence of your changes.
+   For this, make sure to run the tests suite and check the code coverage::
+```
+    $ pytest .      # Run the tests
+```
+6. Commit your changes and push your branch to GitHub::
+```
+    $ git add .
+    $ git commit -m "Your detailed description of your changes."
+    $ git push origin name-of-your-bugfix-or-feature
+```
+7. Submit a pull request through the GitHub website.
+
+
 ## Formatting and Linting
 Hub uses Black and Flake8 to ensure a consistent code format throughout the project.
 if you are using vscode then Replace `.vscode/settings.json` content with the following:

README.md

@@ -1,6 +1,6 @@
-<p align="center">
+ <p align="center">
     <br>
-    <img src="https://raw.githubusercontent.com/snarkai/Hub/master/docs/logo/hub_logo.png" width="50%"/>
+    <img src="https://raw.githubusercontent.com/activeloopai/Hub/master/docs/logo/hub_logo_explainer.png" width="50%"/>
     </br>
 </p>
 
@@ -19,59 +19,62 @@
         <img alt="tweet" src="https://img.shields.io/twitter/url/http/shields.io.svg?style=social"> </a>  
    </br> 
     <a href="https://join.slack.com/t/hubdb/shared_invite/zt-ivhsj8sz-GWv9c5FLBDVw8vn~sxRKqQ">
-  <img src="https://user-images.githubusercontent.com/13848158/97266254-9532b000-1841-11eb-8b06-ed73e99c2e5f.png" height="35" />
-    </a>
+  <img src="https://user-images.githubusercontent.com/13848158/97266254-9532b000-1841-11eb-8b06-ed73e99c2e5f.png" height="35" /> </a>
 
 ---
 
 </a>
 </p>
 
-
-Activeloop's Hacktoberfest is extended till Nov 27! Check out our [Hacktoberfest Project Dashboard](https://github.com/activeloopai/Hub/projects/2) to contribute! 
-
-
-<h3 align="center"> The Docker Hub for datasets. </h3>
-<h4 align="center"> Hub is the fastest way to access & manage datasets for PyTorch and TensorFlow, and build scalable data pipelines.</h4>
+<h3 align="center"> Introducing Data 2.0, powered by Hub. </br>The fastest way to access & manage datasets for PyTorch/TensorFlow, and build scalable data pipelines.</h3>
 
 ---
 
-### Contributors
+### What is Hub for?
 
-[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/0)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/0)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/1)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/1)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/2)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/2)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/3)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/3)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/4)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/4)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/5)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/5)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/6)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/6)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/7)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/7)
+Software 2.0 needs Data 2.0, and Hub delivers it. Most of the time Data Scientists/ML researchers work on data management and preprocessing instead of training models. With Hub, we are fixing this. We store your (even petabyte-scale) datasets as single numpy-like array on the cloud, so you can seamlessly access and work with it from any machine. Hub makes any data type (images, text files, audio, or video) stored in cloud usable as fast as if it were stored on premise. With same dataset view, your team can always be in sync. 
 
+Hub is being used by Waymo, Red Cross, World Resources Institute, Omdena, and others.
 
-### Problems with Current Workflows
+### Features 
 
-We realized that there are a few problems related with current workflow in deep learning data management through our experience of working with deep learning companies and researchers. Most of the time Data Scientists/ML researchers work on data management and preprocessing instead of doing modeling. Deep Learning often requires to work with large datasets. Those datasets can grow up to terabyte or even petabyte size.  It is hard to manage data, store, access, and version-control. It is time-consuming to download the data and link with the training or inference code. There is no easy way to access a chunk of it and possibly visualize. Wouldn’t it be more convenient to have large datasets stored & version-controlled as single numpy-like array on the cloud and have access to it from any machine at scale?
+* Store and retrieve large datasets with version-control
+* Collaborate as in Google Docs: Multiple data scientists working on the same data in sync with no interruptions
+* Access from multiple machines simultaneously
+* Integrate with your ML tools like Numpy, Dask, Ray, [PyTorch](https://docs.activeloop.ai/en/latest/integrations/pytorch.html), or [TensorFlow](https://docs.activeloop.ai/en/latest/integrations/tensorflow.html)
+* Deploy on Google Cloud, S3, Azure as well as Activeloop (by default - and for free!) 
+* Create arrays as big as you want. You can store images as big as 100k by 100k!
+* Keep shape of each sample dynamic. This way you can store small and big arrays as 1 array. 
+* [Visualize](http://app.activeloop.ai/?utm_source=github&utm_medium=repo&utm_campaign=readme) any slice of the data in a matter of seconds without redundant manipulations
 
 ## Getting Started
 
 ### Access public data. Fast
 
-We’ve talked the talk, now let’s walk through how it works:
+To load a public dataset, one needs to write dozens of lines of code and spend hours accessing and understanding the API, as well as downloading the data. With Hub, you only need 2 lines of code, and you **can get started working on your dataset in under 3 minutes**. 
+
 ```sh
 pip3 install hub
 ```
 
 You can access public datasets with a few lines of code.
 ```python
-import hub
+from hub import Dataset
 
-mnist = hub.load("mnist/mnist")
-mnist["data"][0:1000].compute()
+mnist = Dataset("activeloop/mnist")
+mnist["image"][0:1000].compute()
 ```
 
 ### Train a model
 
 Load the data and directly train your model using pytorch
 
 ```python
-import hub
+from hub import Dataset
 import torch
 
-mnist = hub.load("mnist/mnist")
-mnist = mnist.to_pytorch(lambda x: (x["data"], x["labels"]))
+mnist = Dataset("activeloop/mnist")
+mnist = mnist.to_pytorch(lambda x: (x["image"], x["label"]))
 
 train_loader = torch.utils.data.DataLoader(mnist, batch_size=1, num_workers=0)
 
@@ -81,7 +84,7 @@ for image, label in train_loader:
 
 ### Upload your dataset and access it from <ins>anywhere</ins> in 3 simple steps
 
-1. Register a free account at [Activeloop](http://app.activeloop.ai) and authenticate locally
+1. Register a free account at [Activeloop](http://app.activeloop.ai/?utm_source=github&utm_medium=repo&utm_campaign=readme) and authenticate locally
 ```sh
 hub register
 hub login
@@ -93,11 +96,13 @@ from hub import Dataset, schema
 import numpy as np
 
 ds = Dataset(
-    "username/basic",
-    schema={
+    "username/dataset_name",
+    shape = (4,),
+    mode = "w+",
+    schema = {
         "image": schema.Tensor((512, 512), dtype="float"),
         "label": schema.Tensor((512, 512), dtype="float"),
-    },
+    }
 )
 
 ds["image"][:] = np.zeros((4, 512, 512))
@@ -107,41 +112,46 @@ ds.commit()
 
 3. Access it from anywhere else in the world, on any device having a command line.
 ```python
-import hub
+from hub import Dataset
 
-ds = hub.load("username/basic")
+ds = Dataset("username/dataset_name")
 ```
-### Look at Hub in action on Google Colab
-- MNIST Classification with Hub and PyTorch  
-&nbsp;
-[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1LUeZG20A4X4WZX2AYHdI4F6InG6Jb51i?usp=sharing)
+Instead of `username/dataset_name` you could also use `./local/path/`, `s3://path` or `gcs://`
 
-For more advanced data pipelines like uploading large datasets or applying many transformations, please see [docs](http://docs.activeloop.ai).
+## Documentation
 
-## Things you can do with Hub
-* Store large datasets with version-control
-* Collaborate as in Google Docs: Multiple data scientists working on the same data in sync with no interruptions
-* Access from multiple machines simultaneously
-* Integration with your ML tools like Numpy, Dask, PyTorch, or TensorFlow.
-* Create arrays as big as you want
-* Take a quick look on your data without redundant manipulations/in a matter of seconds/etc.
+For more advanced data pipelines like uploading large datasets or applying many transformations, please read the [docs](http://docs.activeloop.ai/?utm_source=github&utm_medium=repo&utm_campaign=readme).
 
 ## Use Cases
-* **Aerial images**: [Satellite and drone imagery](https://activeloop.ai/usecase/intelinair)
+* **Satellite and drone imagery**: [Smarter farming with scalable aerial pipelines](https://activeloop.ai/usecase/intelinair?utm_source=github&utm_medium=repo&utm_campaign=readme), [Mapping Economic Well-being in India](https://towardsdatascience.com/faster-machine-learning-using-hub-by-activeloop-4ffb3420c005), [Fighting desert Locust in Kenya with Red Cross](https://omdena.com/projects/ai-desert-locust/)
 * **Medical Images**: Volumetric images such as MRI or Xray
 * **Self-Driving Cars**: [Radar, 3D LIDAR, Point Cloud, Semantic Segmentation, Video Objects](https://medium.com/snarkhub/extending-snark-hub-capabilities-to-handle-waymo-open-dataset-4dc7b7d8ab35)
 * **Retail**: Self-checkout datasets
 * **Media**: Images, Video, Audio storage
 
+## Community
+
+Join our [Slack community](https://join.slack.com/t/hubdb/shared_invite/zt-ivhsj8sz-GWv9c5FLBDVw8vn~sxRKqQ) to get help from Activeloop team and other users, as well as stay up-to-date on dataset management/preprocessing best practices.
+
+<img alt="tweet" src="https://img.shields.io/twitter/follow/activeloopai?label=stay%20in%20the%20Loop&style=social"> on Twitter.
+
+As always, thanks to our amazing contributors!     </br>
+
+[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/0)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/0)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/1)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/1)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/2)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/2)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/3)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/3)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/4)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/4)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/5)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/5)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/6)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/6)[![](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/images/7)](https://sourcerer.io/fame/davidbuniat/activeloopai/Hub/links/7)
+
+
 ## Examples
 Activeloop’s Hub format lets you achieve faster inference at a lower cost. Test out the datasets we’ve converted into Hub format - see for yourself!
 - [Waymo Open Dataset](https://medium.com/snarkhub/extending-snark-hub-capabilities-to-handle-waymo-open-dataset-4dc7b7d8ab35)
 - [Aptiv nuScenes](https://medium.com/snarkhub/snark-hub-is-hosting-nuscenes-dataset-for-autonomous-driving-1470ae3e1923)
 
 
-
-# Disclaimers
+## Disclaimers
 
 Similarly to other dataset management packages, `Hub` is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license.
 
 If you're a dataset owner and wish to update any part of it (description, citation, etc.), or do not want your dataset to be included in this library, please get in touch through a [GitHub issue](https://github.com/activeloopai/Hub/issues/new). Thanks for your contribution to the ML community!
+
+
+## Acknowledgement
+ This technology was inspired from our experience at Princeton University and would like to thank William Silversmith @SeungLab with his awesome [cloud-volume](https://github.com/seung-lab/cloud-volume) tool. We are heavy users of [Zarr](https://zarr.readthedocs.io/en/stable/) and would like to specially thank their community for building such a great fundamental block. 

docs/source/api.md

@@ -27,7 +27,14 @@
    :private-members:
    :special-members:
 ```
-
+### Sharded Dataset
+```eval_rst
+.. autoclass:: hub.api.sharded_datasetview.ShardedDatasetView
+   :members:
+   :no-undoc-members:
+   :private-members:
+   :special-members:
+```
 ## Pipelines
 ### Transform
 ```eval_rst
@@ -105,6 +112,11 @@
    :no-undoc-members:
    :private-members:
    :special-members:
+.. autoclass:: hub.schema.text.Text
+   :members:
+   :no-undoc-members:
+   :private-members:
+   :special-members:
 .. autoclass:: hub.schema.video.Video
    :members:
    :no-undoc-members: