Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to set base url for custom deployment #131

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add option to set base url for custom deployment #131

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,3 +136,6 @@ dmypy.json
.vscode/

Pipfile

# mac dir settings
.DS_Store
19 changes: 15 additions & 4 deletions cognite/transformations_cli/clients.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@

logger = logging.getLogger(name=None)


def get_client(obj: Dict, timeout: int = 60) -> CogniteClient:
client_id = obj.get("client_id")
client_secret = obj.get("client_secret")
Expand All @@ -18,11 +17,22 @@ def get_client(obj: Dict, timeout: int = 60) -> CogniteClient:
audience = obj.get("audience")
cdf_project_name = obj.get("cdf_project_name")
cluster = obj.get("cluster", "europe-west1-1")
base_url = f"https://{cluster}.cognitedata.com"
base_url = obj.get("base_url")
token_custom_args_str = obj.get("token_custom_args")
if base_url is None or len(base_url) == 0:
base_url = f"https://{cluster}.cognitedata.com"
if not audience:
scopes = scopes.strip().split(" ") if scopes else [f"https://{cluster}.cognitedata.com/.default"]
scopes = scopes.strip().split(" ") if scopes else [f"{base_url}/.default"]
try:
token_custom_args = {"audience": audience} if audience else {}
token_custom_args = {}
if token_custom_args_str:
for pair in token_custom_args_str.split(","):
key, value = pair.split("=")
# remove any whitespace around the key & convert to lower case
key = key.strip().lower()
# remove any whitespace around the value
value = value.strip()
token_custom_args[key] = value
client_config = ClientConfig(
base_url=base_url,
client_name="transformations_cli",
Expand All @@ -33,6 +43,7 @@ def get_client(obj: Dict, timeout: int = 60) -> CogniteClient:
client_secret=client_secret,
token_url=token_url,
scopes=scopes,
audience=audience,
**token_custom_args,
),
)
Expand Down
18 changes: 16 additions & 2 deletions cognite/transformations_cli/commands/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,15 @@

@click.group(context_settings={"help_option_names": ["-h", "--help"]})
@click.version_option(prog_name="transformations_cli", version=__version__)
@click.option(
"--base-url",
help="URL to CDF host (e.g. https://api.cognitedata.com). Provide this or make sure to set 'TRANSFORMATIONS_BASE_URL' environment variable",
envvar="TRANSFORMATIONS_BASE_URL",
)
@click.option(
"--cluster",
default="europe-west1-1",
help="The CDF cluster where Transformations is hosted (e.g. greenfield, europe-west1-1)",
help="The CDF cluster where Transformations is hosted (e.g. greenfield, europe-west1-1). Cluster setting is ignored if the base url is provided",
envvar="TRANSFORMATIONS_CLUSTER",
)
@click.option(
Expand All @@ -43,7 +48,7 @@
)
@click.option(
"--scopes",
help="Scopes to interact with transformations API, relevant for OAuth2 authentication method. 'TRANSFORMATIONS_SCOPES' environment variable can be used instead.",
help="Scopes to interact with transformations API, relevant for OAuth2 authentication method (Space Separated List). 'TRANSFORMATIONS_SCOPES' environment variable can be used instead.",
envvar="TRANSFORMATIONS_SCOPES",
)
@click.option(
Expand All @@ -56,9 +61,15 @@
help="Project to interact with transformations API, 'TRANSFORMATIONS_PROJECT' environment variable can be used instead. Required for OAuth2 and optional for api-keys.",
envvar="TRANSFORMATIONS_PROJECT",
)
@click.option(
"--token-custom-args",
help="Custom arguments passed along with the request (e.g. arg1=value1,arg2=value2), 'TRANSFORMATIONS_TOKEN_CUSTOM_ARGS' environment variable can be used instead.",
envvar="TRANSFORMATIONS_TOKEN_CUSTOM_ARGS",
)
@click.pass_context
def transformations_cli(
context: Context,
base_url: Optional[str] = None,
cluster: str = "europe-west1-1",
api_key: Optional[str] = None,
client_id: Optional[str] = None,
Expand All @@ -67,8 +78,10 @@ def transformations_cli(
scopes: Optional[str] = None,
audience: Optional[str] = None,
cdf_project_name: Optional[str] = None,
token_custom_args: Optional[str] = None,
) -> None:
context.obj = {
"base_url": base_url,
"cluster": cluster,
"api_key": api_key,
"client_id": client_id,
Expand All @@ -77,6 +90,7 @@ def transformations_cli(
"scopes": scopes,
"audience": audience,
"cdf_project_name": cdf_project_name,
"token_custom_args": token_custom_args,
}


Expand Down
45 changes: 23 additions & 22 deletions docs/source/quickstart.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ Usage
Authenticate with API keys
^^^^^^^^^^^^^^^^^^^^^^^^^^^
To use transformations-cli:
- The ``TRANSFORMATIONS_API_KEY`` environment variable must be set to a valid API key for a service account which has access to Transformations.
- ``TRANSFORMATIONS_PROJECT`` environment variable is optional for API key authentication, CDF project can be inferred from API key if skipped.
- The ``TRANSFORMATIONS_API_KEY`` environment variable must be set to a valid API key for a service account which has access to Transformations.
- ``TRANSFORMATIONS_PROJECT`` environment variable is optional for API key authentication, CDF project can be inferred from API key if skipped.

Authenticate with OIDC credentials
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand All @@ -33,8 +33,9 @@ When using OIDC, you need to set the environment variables:
- ``TRANSFORMATIONS_PROJECT``: Required
- ``TRANSFORMATIONS_SCOPES``: Transformations CLI assumes that this is optional, generally required to authenticate except for Aize project. Space separated for multiple scopes.
- ``TRANSFORMATIONS_AUDIENCE``: Optional
- ``TRANSFORMATIONS_TOKEN_CUSTOM_ARGS``: Optional custom arguments passed along with the request (e.g. arg1=value1,arg2=value2)

By default, transformations-cli runs against the main CDF cluster (europe-west1-1). To use a different cluster, specify the ``--cluster`` parameter or set the environment variable ``TRANSFORMATIONS_CLUSTER``. Note that this is a global parameter, which must be specified before the subcommand. For example:
By default, transformations-cli runs against the main CDF cluster (europe-west1-1). To use a different cluster, specify the ``--cluster`` parameter or set the environment variable ``TRANSFORMATIONS_CLUSTER``. If CDF is hosted on a different URL, specify the ``--base-url`` parameter or set the environment variable ``TRANSFORMATIONS_BASE_URL``. Specifying base URL will take precedence over cluster. Note that this is a global parameter, which must be specified before the subcommand. For example:

.. code-block:: bash

Expand All @@ -50,32 +51,32 @@ By default, transformations-cli runs against the main CDF cluster (europe-west1-
- Options
- Description
* - list
-
-
- ``--limit``, ``--interactive``, ``--data-set-id``, ``data-set-external-id``, ``destination-type``, ``conflict-mode``, ``--tag``
- List transformations
* - show
-
-
- ``--external-id``, ``--id``, ``--job-id``
- Show a transformation/job
* - jobs
-
-
- ``--external-id``, ``--id``, ``--limit``, ``--interactive``
- List jobs
* - delete
-
-
- ``--external-id``, ``--id``, ``--delete-schedule``
- Delete a transformation
* - query
- ``query``
- ``--source-limit``, ``--infer-schema-limit``, ``--limit``
- Run a query
* - run
-
-
- ``--external-id``, ``--id``, ``--watch``, ``--watch-only``, ``--time-out``
- Run a transformation
* - deploy
- ``path``, ``--debug``
-
-
- Deploy transformations

Help
Expand Down Expand Up @@ -188,8 +189,8 @@ It prints the transformation details in a tabular format, such as latest job's m

``transformations-cli jobs``
----------------------------
``transformations-cli jobs`` can list the latest jobs.
You can optionally provide the ``external_id`` or ``id`` of the transformations of which jobs you want to list.
``transformations-cli jobs`` can list the latest jobs.
You can optionally provide the ``external_id`` or ``id`` of the transformations of which jobs you want to list.
You can also provide ``--limit``, which defaults to 10. Use ``--limit=-1`` if you want to list all.

.. code-block:: bash
Expand All @@ -215,12 +216,12 @@ You can also provide ``--limit``, which defaults to 10. Use ``--limit=-1`` if yo
- No
- Limit for the job history. Use -1 to retrieve all results.
* - ``--id``
-
-
- No
- No
- List jobs by transformation ``id``. Either this or ``--external-id`` must be specified.
* - ``--external-id``
-
-
- No
- No
- List jobs by transformation ``external_id``. Either this or ``--id`` must be specified.
Expand All @@ -232,7 +233,7 @@ You can also provide ``--limit``, which defaults to 10. Use ``--limit=-1`` if yo

``transformations-cli delete``
------------------------------
``transformations-cli`` provides a delete subcommand, which can delete a transformation.
``transformations-cli`` provides a delete subcommand, which can delete a transformation.

At minimum, this command requires either an ``--id`` or ``--external-id`` to be specified:

Expand All @@ -258,12 +259,12 @@ You can also specify ``--delete-schedule`` flag to delete a scheduled transforma
- Required
- Description
* - ``--id``
-
-
- No
- No
- ``id`` of the transformation to be deleted. Either this or ``--external-id`` must be specified.
* - ``--external-id``
-
-
- No
- No
- ``external_id`` of the transformation to be deleted. Either this or ``--id`` must be specified.
Expand All @@ -281,7 +282,7 @@ transformations-cli also allows you to run queries.

transformations-cli query "select * from _cdf.assets limit 100"

This will print the schema and the results.
This will print the schema and the results.
The query command is intended for previewing your SQL queries, and is not designed for large data exports. For this reason, there are a few limits in place. Query command takes ``--infer-schema-limit``, ``--source-limit`` and ``--limit`` options. Default values are 100, 100 and 1000 in the corresponding order.


Expand Down Expand Up @@ -362,12 +363,12 @@ If you want to watch a job for completion without actually starting a transforma
- Required
- Description
* - ``--id``
-
-
- No
- No
- The ``id`` of the transformation to run. Either this or ``--external-id`` must be specified.
* - ``--external-id``
-
-
- No
- No
- The ``external_id`` of the transformation to run. Either thisor ``--id`` must be specified.
Expand Down Expand Up @@ -401,7 +402,7 @@ To deploy a set of transformations, use the deploy subcommand:

transformations-cli deploy <path>

The ``<path>`` argument should point to a directory containing YAML manifests.
The ``<path>`` argument should point to a directory containing YAML manifests.
This directory is scanned recursively for ``*.yml`` and ``*.yaml`` files, so you can organize your transformations into separate subdirectories.

.. list-table:: Deploy args
Expand All @@ -415,7 +416,7 @@ This directory is scanned recursively for ``*.yml`` and ``*.yaml`` files, so you
* - ``path``
- .
- Yes
- Root folder of transformation manifests.
- Root folder of transformation manifests.

.. list-table:: Debug options
:widths: 25 25 25 25 25
Expand Down Expand Up @@ -444,4 +445,4 @@ Important notes:
:language: YAML

.. literalinclude:: transformation_apikey.yaml
:language: YAML
:language: YAML