Added Target SDK, Added Stream Maps for Taps

- Adds support for Target class and Sink classes
- Adds support for Stream Maps for Taps (meltano#63, !92)
- Adds target cookiecutter for new projects
- Documentation improvements via docstrings
- Bump min python version to 3.6.1

CHANGELOG.md

@@ -16,6 +16,11 @@ The next few lines form the template for unreleased changes.
 ### Fixed
 -->
 
+## Unreleased
+
+- Added Target SDK implementation for Singer targets (#96, !4)
+- Add stream map inline transform and filter capabilities (#63, !4, !92)
+
 ## v0.2.0
 
 ### Added

cookiecutter/tap-template/{{cookiecutter.tap_id}}/pyproject.toml

@@ -6,7 +6,7 @@ authors = ["{{ cookiecutter.admin_name }}"]
 license = "Apache 2.0"
 
 [tool.poetry.dependencies]
-python = "<3.9,>=3.6"
+python = "<3.9,>=3.6.1"
 requests = "^2.25.1"
 singer-sdk = "^0.2.0"
 

cookiecutter/target-template/README.md

@@ -0,0 +1,24 @@
+# Singer SDK Target Template
+
+To use this cookie cutter template:
+
+```bash
+pip3 install pipx
+pipx ensurepath
+# You may need to reopen your shell at this point
+pipx install cookiecutter
+```
+
+Initialize Cookiecutter template directly from Git:
+
+```bash
+cookiecutter https://gitlab.com/meltano/singer-sdk --directory="cookiecutter/target-template"
+```
+
+Or locally from an already-cloned `singer-sdk` repo:
+
+```bash
+cookiecutter ./singer-sdk/cookiecutter/target-template
+```
+
+See the [dev guide](../../docs/dev_guide.md).

cookiecutter/target-template/cookiecutter.json

@@ -0,0 +1,7 @@
+{
+    "destination_name": "MyDestinationName",
+    "admin_name": "FirstName LastName",
+    "target_id": "target-{{ cookiecutter.destination_name.lower() }}",
+    "library_name": "{{ cookiecutter.target_id.replace('-', '_') }}",
+    "serialization_method": ["Per record", "Per batch"]
+}

cookiecutter/target-template/cookiecutter.tests.yml

@@ -0,0 +1,7 @@
+tests:
+  - destination_name: TargetBatchSinkTest
+    target_id: target-batchsink-test
+    serialization_method: Per batch
+  - destination_name: TargetRecordSinkTest
+    target_id: target-recordsink-test
+    serialization_method: Per record

cookiecutter/target-template/{{cookiecutter.library_name}}/.gitignore

@@ -0,0 +1,132 @@
+# Secrets and internal config files
+.secrets/*
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/

cookiecutter/target-template/{{cookiecutter.library_name}}/.secrets/.gitignore

@@ -0,0 +1,6 @@
+# IMPORTANT! This folder is hidden from git - if you need to store config files or other secrets,
+# make sure those are never staged for commit into your git repo. You can store them here or another
+# secure location.
+
+*
+!.gitignore

cookiecutter/target-template/{{cookiecutter.library_name}}/README.md

@@ -0,0 +1,101 @@
+# {{ cookiecutter.target_id }}
+
+`{{ cookiecutter.target_id }}` is a Singer target for {{ cookiecutter.destination_name }}.
+
+Build with the [Meltano Target SDK](https://sdk.meltano.com).
+
+## Installation
+
+- [ ] `Developer TODO:` Update the below as needed to correctly describe the install procedure. For instance, if you do not have a PyPi repo, or if you want users to directly install from your git repo, you can modify this step as appropriate.
+
+```bash
+pipx install {{ cookiecutter.target_id }}
+```
+
+## Configuration
+
+### Accepted Config Options
+
+- [ ] `Developer TODO:` Provide a list of config options accepted by the target.
+
+A full list of supported settings and capabilities for this
+target is available by running:
+
+```bash
+{{ cookiecutter.target_id }} --about
+```
+
+### Source Authentication and Authorization
+
+- [ ] `Developer TODO:` If your target requires special access on the source system, or any special authentication requirements, provide those here.
+
+## Usage
+
+You can easily run `{{ cookiecutter.target_id }}` by itself or in a pipeline using [Meltano](www.meltano.com).
+
+### Executing the Target Directly
+
+```bash
+{{ cookiecutter.target_id }} --version
+{{ cookiecutter.target_id }} --help
+# Test using the "Carbon Intensity" sample:
+tap-carbon-intensity | {{ cookiecutter.target_id }} --config /path/to/{{ cookiecutter.target_id }}-config.json
+```
+
+## Developer Resources
+
+- [ ] `Developer TODO:` As a first step, scan the entire project for the text "`TODO:`" and complete any recommended steps, deleting the "TODO" references once completed.
+
+### Initialize your Development Environment
+
+```bash
+pipx install poetry
+poetry install
+```
+
+### Create and Run Tests
+
+Create tests within the `{{ cookiecutter.library_name }}/tests` subfolder and
+  then run:
+
+```bash
+poetry run pytest
+```
+
+You can also test the `{{cookiecutter.target_id}}` CLI interface directly using `poetry run`:
+
+```bash
+poetry run {{cookiecutter.target_id}} --help
+```
+
+### Testing with [Meltano](meltano.com)
+
+_**Note:** This target will work in any Singer environment and does not require Meltano.
+Examples here are for convenience and to streamline end-to-end orchestration scenarios._
+
+Your project comes with a custom `meltano.yml` project file already created. Open the `meltano.yml` and follow any _"TODO"_ items listed in
+the file.
+
+Next, install Meltano (if you haven't already) and any needed plugins:
+
+```bash
+# Install meltano
+pipx install meltano
+# Initialize meltano within this directory
+cd {{ cookiecutter.target_id }}
+meltano install
+```
+
+Now you can test and orchestrate using Meltano:
+
+```bash
+# Test invocation:
+meltano invoke {{ cookiecutter.target_id }} --version
+# OR run a test `elt` pipeline with the Carbon Intensity sample tap:
+meltano elt tap-carbon-intensity {{ cookiecutter.target_id }}
+```
+
+### Singer SDK Dev Guide
+
+See the [dev guide](../../docs/dev_guide.md) for more instructions on how to use the Singer SDK to 
+develop your own taps and targets.

cookiecutter/target-template/{{cookiecutter.library_name}}/meltano.yml

@@ -0,0 +1,24 @@
+version: 1
+send_anonymous_usage_stats: true
+project_id: {{cookiecutter.target_id}}
+plugins:
+  extractors:
+  loaders:
+  - name: {{cookiecutter.target_id}}
+    namespace: {{cookiecutter.library_name}}
+    # TODO: To test using Meltano, replace with absolute path
+    #       to the {{cookiecutter.target_id}}.sh script:
+    executable: ./{{cookiecutter.target_id}}.sh
+    capabilities:
+    - state
+    - catalog
+    - discover
+    config:
+      start_date: '2010-01-01T00:00:00Z'
+    settings:
+    # TODO: To configure using Meltano, declare settings and their types here:
+    - name: username
+    - name: password
+      kind: password
+    - name: start_date
+      value: '2010-01-01T00:00:00Z'

cookiecutter/target-template/{{cookiecutter.library_name}}/pyproject.toml

@@ -0,0 +1,22 @@
+[tool.poetry]
+name = "{{cookiecutter.target_id}}"
+version = "0.0.1"
+description = "`{{cookiecutter.target_id}}` is a Singer target for {{cookiecutter.destination_name}}, built with the Meltano SDK for Singer Targets."
+authors = ["{{ cookiecutter.admin_name }}"]
+license = "Apache 2.0"
+
+[tool.poetry.dependencies]
+python = "<3.9,>=3.6.1"
+requests = "^2.25.1"
+singer-sdk = "^0.2.1"
+
+[tool.poetry.dev-dependencies]
+pytest = "^6.1.2"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.poetry.scripts]
+# CLI declaration
+{{cookiecutter.target_id}} = '{{cookiecutter.library_name}}.tap:cli'