Ignore the "extend" field

[anna-geller]

Feature description

We wanted to add backend handling to ignore the property "extend" and all its sub-properties so that flows could be deployed directly from blueprints. However, this is not the case yet.

[MichaScant]

Hey, can I try work on this issue please?

[MichaScant]

Hi sorry can you clarify, do you want the property extend ignored always or add an option to ignore it?

Should the user have the option to add extend if it and any sub-properties are ignored?

[anna-geller]

always ignored - it contains extra metadata that shouldn't be parsed in the editor

[loicmathieu]

I think it would be better to be filtered from the UI than having ad-hoc handling inside the Flow.
If a blueprint is not just a flow but a more complex structure, we must extract the flow and create it instead of storing some piece of YAML that would never has any meaning.

This extend part will stay forever inside the flow YAML which is not a good idea.

I even think it's a bad idea to mix flow YAML and blueprint specific YAML inside the same root, if at some point we want to have an extend property inside the flow YAML it would not be possible without breaking existing blueprints.

So this extend inside blueprint at the root level is annoying, it would be better to have something like:

flow:
  id: flowId
  namespace: flowNamespace
  tasks: []
# all other blueprint information here
title: blueprint title
description: blueprint description

[anna-geller]

We discussed this a long time ago and we decided against it. Blueprints must be useable as is without nesting it inside of some parent flow source property.

Up to you if you prefer to filter the extent on the BE or FE side, the only requirements are:

1. Blueprints stay as defined in the blueprints repo with a top-level extend property
2. The extend property must be ignored (LMK your decision if you approve the BE implementation from PR fix(core): now ignoring the extend field in the backend #6540 or if we need to handle it only on the FE side) -- my feeling is that we need to do it on the backend side as otherwise you won't be able to sync such flow with extend via CI, Git Sync or Terraform

[loicmathieu]

You cannot sync a flow with extend as there is no such property.
As far as I know, you cannot sync via CI a blueprint to a flow, those are different objects.

So I'll make the change on the BE, a flow YAML must not have this extend, it only clutter it as, as I said, if we didn't filter it before, it will stay forever in the source.

[anna-geller]

Next steps: we discussed with Loic so he has all the context as to why this topic is important (we will be able to use many blueprints for QA directly) and Loic will coordinate the implementation with Florian and Brian

[Ben8t]

@brian-mulier-p assigned to you, let us know if things are not clear (we had a meeting 2 weeks ago, and Loic has all the context).

[brian-mulier-p]

We discussed with @loicmathieu, and the extend keyword could be leveraged for an extension management later on.
To tackle this we may need to rename extend to _ignore, ignoreProperties or additionalProperties instead. I have no strong opinion on this, @Ben8t / @anna-geller WDYT?

[Ben8t]

What about metadata instead? @anna-geller

Not a big fan of the ignore ones, they seems unrelated to the current props and semantic we start to have (task, triggers, namespace, etc. all of these relate somehow to an actual concept in Kestra). Thinking also about merging the description field here.

[anna-geller]

@Ben8t We need metadata keyword for this one next year

@brian-mulier-p Extension management within a flow? What do you mean? Let's not use ignore in the naming as we use .kestraignore in git plugin and once we add git sync from UI, this may be brought to the top level

Reminder of what this is intended to be used for -- extra info about a blueprint:

extend:
  title: Trigger multiple Airbyte Cloud syncs in parallel, then run a dbt job
  description: |
    This flow runs Airbyte Cloud syncs in parallel and then runs dbt Core's CLI
    commands. 

    It's recommended to configure the GCP service account and [Airbyte API
    token](https://portal.airbyte.com/apiKeys), referenced in the
    `pluginDefaults`, as a secret.
  tags:
    - Ingest
    - Git
    - dbt
    - BigQuery
    - Parallel
  ee: false
  demo: false
  meta_description: "This flow runs Airbyte Cloud syncs in parallel and then runs
    dbt Core's CLI commands. "

Extend is really best so far, but if you have some strong reasons why not to use it, go for one of those options:

1. blueprintInfo
2. blueprintMetadata
3. about
4. hidden (if you want that ignore direction)
5. extra