-
Notifications
You must be signed in to change notification settings - Fork 71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Conditionally required settings #1519
Comments
Initial proposal from office hours 2023-03-22 (
# ...
class TapSpotify(Tap):
# ...
# config_jsonschema as a callable to evaluate at runtime
config_jsonschema = lambda cls, config: th.PropertiesList(
th.Property(
"client_id",
th.StringType,
required=config.get("auth_flow") == "client_credentials",
description="App client ID",
),
th.Property(
"client_secret",
th.StringType,
required=config.get("auth_flow") == "client_credentials",
description="App client secret",
),
th.Property(
"refresh_token",
th.StringType,
required=config.get("flow") == "client_credentials",
description="Refresh token",
),
th.Property(
"username",
th.StringType,
required=config.get("flow") in ["password", None],
description="Username",
),
th.Property(
"password",
th.StringType,
required=config.get("flow") in ["password", None],
description="Password",
),
th.Property(
"oauth2_flow",
th.StringType,
required=False,
default="client_credentials"
description="OAuth2 flow",
),
).to_dict()
# ...
# ...
class PluginBase(metaclass=abc.ABCMeta):
# ...
config_jsonschema: dict | Callable[[type[PluginBase], Mapping[str, Any]], dict] = {}
# ...
# new method to resolve JSON schema base on defined type
@classmethod
def _resolve_config_jsonschema(
cls: type[PluginBase],
config: Mapping[str, Any] = None,
) -> dict[str, Any]:
config_jsonschema = cls.config_jsonschema
if not isinstance(config_jsonschema, dict):
config_jsonschema = config_jsonschema(cls, config or {})
return config_jsonschema
# ...
@classproperty
def _env_var_config(cls) -> dict[str, Any]:
"""Return any config specified in environment variables.
Variables must match the convention "<PLUGIN_NAME>_<SETTING_NAME>",
all uppercase with dashes converted to underscores.
Returns:
Dictionary of configuration parsed from the environment.
"""
plugin_env_prefix = f"{cls.name.upper().replace('-', '_')}_"
config_jsonschema = cls._resolve_config_jsonschema() # used here
cls.append_builtin_config(config_jsonschema)
return parse_environment_config(config_jsonschema, plugin_env_prefix)
# ...
def _validate_config(
self,
raise_errors: bool = True,
warnings_as_errors: bool = False,
) -> tuple[list[str], list[str]]:
"""Validate configuration input against the plugin configuration JSON schema.
Args:
raise_errors: Flag to throw an exception if any validation errors are found.
warnings_as_errors: Flag to throw an exception if any warnings were emitted.
Returns:
A tuple of configuration validation warnings and errors.
Raises:
ConfigValidationError: If raise_errors is True and validation fails.
"""
warnings: list[str] = []
errors: list[str] = []
log_fn = self.logger.info
config_jsonschema = self._resolve_config_jsonschema(self.config) # used here
if config_jsonschema:
self.append_builtin_config(config_jsonschema)
self.logger.debug(
f"Validating config using jsonschema: {config_jsonschema}",
)
validator = JSONSchemaValidator(config_jsonschema)
errors = [e.message for e in validator.iter_errors(self._config)]
if errors:
summary = (
f"Config validation failed: {'; '.join(errors)}\n"
f"JSONSchema was: {config_jsonschema}"
)
if raise_errors:
raise ConfigValidationError(summary)
log_fn = self.logger.warning
else:
summary = f"Config validation passed with {len(warnings)} warnings."
for warning in warnings:
summary += f"\n{warning}"
if warnings_as_errors and raise_errors and warnings:
raise ConfigValidationError(
f"One or more warnings ocurred during validation: {warnings}",
)
log_fn(summary)
return warnings, errors
# ... |
We could add a OneOf(
ObjectType(
Discriminator(name="flow", value="oauth"),
Property("client_id", StringType, secret=True, required=True),
Property("client_secret", StringType, secret=True, required=True),
),
ObjectType(
Discriminator(name="flow", value="apikey"),
Property("api_key", StringType, secret=True, required=True),
),
) Roughly equivalent to JSON Schema{
"oneOf": [
{
"type": "object",
"properties": {
"flow": {
"const": "oauth"
},
"client_id": {
"type": "string"
},
"client_secret": {
"type": "string"
}
},
"required": [
"flow",
"client_id",
"client_secret"
]
},
{
"type": "object",
"properties": {
"flow": {
"const": "apikey"
},
"key": {
"type": "string"
}
},
"required": [
"flow",
"key"
]
}
]
} I think this covers all the use cases described above. Wdyt? |
@edgarrmondragon - Yeah, this solution looks really good. Where does this statement need to exist in relationship to the PropertiesList(
Property("flow", StringType()),
OneOf(
ObjectType(
Discriminator(name="flow", value="oauth"),
Property("client_id", StringType, secret=True, required=True),
Property("client_secret", StringType, secret=True, required=True),
),
ObjectType(
Discriminator(name="flow", value="apikey"),
Property("api_key", StringType, secret=True, required=True),
),
)
) If I understand correctly, I think the above should work. I wonder if it's also possible to declare these below the properties list. PropertiesList(
Property("flow", StringType),
Property("client_id", StringType, secret=True),
Property("client_secret", StringType, secret=True),
Property("api_key", StringType, secret=True),
constraints=[
OneOf(
ObjectType(
Discriminator(name="flow", value="oauth"),
PropertyRef("client_id"),
PropertyRef("client_secret"),
),
ObjectType(
Discriminator(name="flow", value="apikey"),
PropertyRef("api_key"),
),
)
]
) This type of syntax is probably worse for the above use case, since there aren't any redundancies across declarations. But in a scenario where we wouldn't want to redeclare overlapped settings in multiple scenarios, it could be handy to declare the conditions after the property declarations. Wdyt? We can discuss again in office hours today too. 😄 |
@aaronsteers I couldn't join OH yesterday but I've just watched the recording and here's some notes.
|
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the |
I think the only missing piece here is better support in |
https://json-schema.org/understanding-json-schema/reference/conditionals would make for an elegant solution to this. |
Notes from office hours:
We want to be able to tell if the config is valid:
Reuben use cases:
The text was updated successfully, but these errors were encountered: