Add Pipeline and Step Comment Attributes #1557
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
This PR adds a
comment
attribute to thePipelineConfig
andStep
classes.What does this PR do?
Adds the ability for pipeline authors to document what a pipeline and/or step does for future maintainers (including themselves).
Relevant implementation details
For pipelines containing many steps, it becomes unclear what effect a pipeline or step has. To discern what a step does, for example, a pipeline maintainer must look at the extract_source attribute, navigate to the Python or SQL file, and then read through the file to determine what a step does.
Caveats/things to watch out for when reviewing:
Linked issues
Resolves #1468.
Functionality
databricks labs remorph ...
PipelineConfig
andStep
classes with a metadata attribute.Tests