Skip to content

Latest commit

 

History

History
294 lines (233 loc) · 6.3 KB

migrationguide.md

File metadata and controls

294 lines (233 loc) · 6.3 KB
Raw

Migration from jetfire-cli@v2 to transformations-cli@main

Deploy Step Changes:

  1. Deploy step should use transformations-cli@main instead of jetfire-cli@v2:
# GitHub Workflow
- name: Transformations Deploy step
  uses: cognitedata/transformations-cli@main
  1. project-name input should be renamed as cdf-project-name:
# GitHub Workflow
with:
  path: <transformations_folder>
    ..
    cdf-project-name: extractor-bluefield-testing
    ...

3- Make sure scopes are space separated if you need multiple scopes:

# GitHub Workflow
with:
  path: <transformations_folder>
  ...
  scopes: scope1 scope2 scope3
  ...

4- You can reference values as environment values or actual values in manifests, make sure you provide env variables you need in your manifests.

# GitHub Workflow
env:
  TOKEN_URL: https://login.microsoftonline.com/someaad/oauth2/v2.0/token
  CLIENT_SECRET: ${{secrets.CLIENT_SECRET}}
  FIXED_TRANSFORMATION_NAME: example
# Transformation manifest file

name: ${FIXED_TRANSFORMATION_NAME} # read from environment variable
externalId: hello # value is used directly
authentication:
  clientSecret: ${CLIENT_SECRET} # read from environment variable
  clientId: randomclientid # value is used directly 
  tokenUrl: ${TOKEN_URL} # read from environment variable

Manifest Changes:

There are two options to achieve this: Set legacy flag on your existing manifest with no changes or start using the new format!

Set legacy flag in your legacy manifest:

Add legacy: true to your legacy transformation manifests.

# Legacy transformation manifest
legacy: true
name: hello
externalId: hello
...

Start using the new manifest format:

Although we provide backward compatibility with the legacy flag, we recommend using the new manifest format as:

  • it has an improved structure
  • new functionality will only be made available in the new format
  • it enables the use of environment variables

You can find the full documentation of the transformation manifest here.

The following sections show the requirements for migrating the different fields of the manifest:

Migrate name:

  • No changes required.

Migrate externalId:

  • No changes required.

Migrate destination:

  • assets, events, timeseries, datapoints, sequences, labels, relationships, files, raw are now case sensitive and should be lower case.
  • We now also support data_sets.
  • These resource types change formatting:
Legacy (Case insensitive) New (Case sensitive with underscores)

assethierarchy

asset_hierarchy

stringdatapoints

string_datapoints

Migrate action:

  • No changes required.

Migrate query:

In the old version, the query field expected the path to a SQL file. Now you can either provide a SQL file path or provide the query directly as a string.

Legacy New 
# Legacy transformation manifest
query: my_query.sql
 
#  Transformation manifest
query: "select 'asset1' as name"
#  Transformation manifest
query:
  file:
    my_query.sql

Migrate schedule:

  • No changes required.
  • is_paused option is supported in the new CLI. If you need to pause some schedules, you can use the following format:
#  Transformation manifest
schedule:
  interval: "* * * * *"
  is_paused: true

Migrate shared:

  • No changes required, default is changed to true.

Migrate ignoreNullFields:

  • No changes required.

Migrate apiKey:

  • apiKey values used to be treated as environment variables in jetfire-cli. Now you need to reference them as ${API_KEY} instead of API_KEY to treat them as environment variables.
Legacy New 
# Legacy transformation manifest
apiKey: API_KEY # as provided in deploy step
 
#  Transformation manifest
authentication:
  apiKey: ${API_KEY} # as provided in deploy step

Migrate authentication:

  • clientId and clientSecret were assumed to be environment variables while others are used as actual values in the old CLI manifest. ${MY_NAME} will be treated as the value for the environment variable named MY_NAME while MY_NAME will be used as the value itself in the new manifest. This leaves where to use environment variables to the user so it is really flexible.
  • Also, we added support for audience parameter to retrieve the token.
Legacy New 
# Legacy transformation manifest
# The following two is given as the name of an environment variable
clientId: COGNITE_CLIENT_ID
clientSecret: COGNITE_CLIENT_SECRET
# The following are explicit values, not environment variables
tokenUrl: https://my-idp.com/oauth2/token
scopes:
  - https://bluefield.cognitedata.com/.default
cdfProjectName: my-project
 
# Transformation manifest
clientId: ${CLIENT_ID}
clientSecret: ${CLIENT_SECRET}
tokenUrl: https://my-idp.com/oauth2/token
scopes: 
  - https://bluefield.cognitedata.com/.default
cdfProjectName: my-project
# audience: ""
  • apiKey configuration is moved under authentication.
Legacy New 
# Legacy ransformation manifest
apiKey: API_KEY # as provided in deploy step
# Legacy ransformation manifest
apiKey: 
  read: READ_API_KEY # as provided in deploy step
  write: WRITE_API_KEY # as provided in deploy step
 
# Transformation manifest
authentication:
  apiKey: ${API_KEY} # as provided in deploy step
# Transformation manifest
authentication:
  apiKey: 
    read: ${READ_API_KEY} # as provided in deploy step
    write: ${WRITE_API_KEY} # as provided in deploy step

Migrate notifications:

  • No changes required for jetfire-cli@v2.
  • If you are migrating from jetfire-cli@v1, make sure that you reflect notifications in your manifests. Missing notifications on existing transformations get deleted.