Skip to content

Commit

Permalink
Added custom streams based on provided sql query
Browse files Browse the repository at this point in the history
  • Loading branch information
jlloyd-widen committed Jan 12, 2024
1 parent 0a4cd47 commit aebf20f
Show file tree
Hide file tree
Showing 7 changed files with 1,271 additions and 578 deletions.
44 changes: 36 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
Built with the [Meltano Tap SDK](https://sdk.meltano.com) for Singer Taps.

The primary advantage of this version of `tap-mysql` is that it emphasizes and completes
the `LOG_BASED` replication method, whereas other variants have buggy or incomplete
the `LOG_BASED` replication method, whereas other variants have buggy or incomplete
implementations of such. Other advantages include inheriting the capabilities of a tap
built on the Meltano Tap SDK.

Expand Down Expand Up @@ -34,26 +34,52 @@ pipx install git+https://github.com/ORG_NAME/tap-mysql.git@main
### Accepted Config Options

<!--
Developer TODO: Provide a list of config options accepted by the tap.
This section can be created by copy-pasting the CLI output from:
```
tap-mysql --about --format=markdown
```
-->

| Setting | Required | Default | Description |
|:---------------------|:--------:|:-------:|:--------------------------------------------------------------------------------------------------------------------------------------------|
| host | True | None | The hostname of the MySQL instance. |
| port | False | 3306 | The port number of the MySQL instance. |
| user | True | None | The username |
| password | True | None | The password for the user |
| custom_streams | False | None | An array of customized streams to use. |
| stream_maps | False | None | Config object for stream maps capability. For more information check out [Stream Maps](https://sdk.meltano.com/en/latest/stream_maps.html). |
| stream_map_config | False | None | User-defined config values to be used within map expressions. |
| flattening_enabled | False | None | 'True' to enable schema flattening and automatically expand nested properties. |
| flattening_max_depth | False | None | The max depth to flatten schemas. |
| batch_config | False | None | |

A full list of supported settings and capabilities for this
tap is available by running:

```bash
tap-mysql --about
```

#### `custom_stream` configuration

Custom streams are defined in the `custom_streams` configuration option. This option is
an array of objects, each of which defines a custom stream. Each custom stream object
has the following properties:

| Property | Required | Default | Description |
|:-------------|:--------:|:-------:|:--------------------------------------------------------------------------------------------------------------------------------------------------------------|
| name | True | None | The name of the custom stream. |
| db_schemas | False | None | An array of schema names of the MySQL instance that is being queried. The same query will be run against each schema. |
| sql | True | None | The custom sql query to use for this stream. If provided, the string `{db_schema}` will be replaced with the schema name(s) from the `db_schemas` property.}` |
| primary_keys | False | None | The primary keys of the custom stream. |

### Configure using environment variables

This Singer tap will automatically import any environment variables within the working directory's
`.env` if the `--config=ENV` is provided, such that config values will be considered if a matching
This Singer tap will automatically import any environment variables within the working
directory's
`.env` if the `--config=ENV` is provided, such that config values will be considered if
a matching
environment variable is set either in the terminal context or in the `.env` file.

### Source Authentication and Authorization
Expand All @@ -64,7 +90,8 @@ Developer TODO: If your tap requires special access on the source system, or any

## Usage

You can easily run `tap-mysql` by itself or in a pipeline using [Meltano](https://meltano.com/).
You can easily run `tap-mysql` by itself or in a pipeline
using [Meltano](https://meltano.com/).

### Executing the Tap Directly

Expand All @@ -88,7 +115,7 @@ poetry install
### Create and Run Tests

Create tests within the `tests` subfolder and
then run:
then run:

```bash
poetry run pytest
Expand Down Expand Up @@ -132,5 +159,6 @@ meltano elt tap-mysql target-jsonl

### SDK Dev Guide

See the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more instructions on how to use the SDK to
See the [dev guide](https://sdk.meltano.com/en/latest/dev_guide.html) for more
instructions on how to use the SDK to
develop your own taps and targets.
16 changes: 16 additions & 0 deletions meltano.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,22 @@ plugins:
- name: user
- name: password
kind: password
- name: custom_streams
kind: array
config:
custom_streams:
- name: example_table_name
db_schemas:
- example1
- example2
sql: SELECT * FROM {db_schema}.filesondisk LIMIT 5
- name: example_table_name2
db_schemas:
- example1
sql: SELECT * FROM example1.filesondisk LIMIT 5
select:
- '*-example_table_name.*'
- 'example1-example_table_name2.*'
loaders:
- name: target-jsonl
variant: andyh1203
Expand Down
34 changes: 34 additions & 0 deletions plugins/loaders/target-jsonl--andyh1203.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"plugin_type": "loaders",
"name": "target-jsonl",
"namespace": "target_jsonl",
"variant": "andyh1203",
"label": "JSON Lines (JSONL)",
"docs": "https://hub.meltano.com/loaders/target-jsonl--andyh1203",
"repo": "https://github.com/andyh1203/target-jsonl",
"pip_url": "target-jsonl",
"description": "JSONL loader",
"logo_url": "https://hub.meltano.com/assets/logos/loaders/jsonl.png",
"settings": [
{
"name": "destination_path",
"kind": "string",
"value": "output",
"label": "Destination Path",
"description": "Sets the destination path the JSONL files are written to, relative\nto the project root.\n\nThe directory needs to exist already, it will not be created\nautomatically.\n\nTo write JSONL files to the project root, set an empty string (`\"\"`).\n"
},
{
"name": "do_timestamp_file",
"kind": "boolean",
"value": false,
"label": "Include Timestamp in File Names",
"description": "Specifies if the files should get timestamped.\n\nBy default, the resulting file will not have a timestamp in the file name (i.e. `exchange_rate.jsonl`).\n\nIf this option gets set to `true`, the resulting file will have a timestamp associated with it (i.e. `exchange_rate-{timestamp}.jsonl`).\n"
},
{
"name": "custom_name",
"kind": "string",
"label": "Custom File Name Override",
"description": "Specifies a custom name for the filename, instead of the stream name.\n\nThe file name will be `{custom_name}-{timestamp}.jsonl`, if `do_timestamp_file` is `true`.\nOtherwise the file name will be `{custom_name}.jsonl`.\n\nIf custom name is not provided, the stream name will be used.\n"
}
]
}
Loading

0 comments on commit aebf20f

Please sign in to comment.